> I wonder if such an adversarial game approach could be outfitted to not just well-defined games but to wholly generalizable improvements. e.g., could be used as a way to improve RLAIF potentially?

That's a good question!

Here's my (amateur) understanding of the landscape:

- RLHF: Given a mixture of unlabeled LLM responses, first gather human feedback on which response is preferred to mark them as Good or Bad. Use these annotations to train a Reward Model that attempts to model the preferences of humans on the input data. Then use this Reward Model for training the model with traditional RL techniques.

- RLAIF: Given good and bad examples of LLM responses, instead of using human feedback, use an off-the-shelf zero-shot LLM to annotate the data. Then, one can either train a traditional Reward Model using these auto-annotated samples, or else one can use the LLM to generate scores in real-time when training the models (a more "online" method of real-time scoring). In either case, each of these Reward methods can be used for training with RL.

- Adversarial Games: By limiting the scope of responses to situations where the preference of one answer vs. another can be computed with an algorithm (I.E., clearly-defined rules of a game), then we bypass the need to deal with a "fuzzy" Reward Model (whether built through traditional RLHF, or through RLAIF). The whole reason why RLAIF is a "thing" is because high-quality human-annotated data is difficult to acquire, so researchers attempt to approximate it with LLMs. But if we bypass that need and can clearly define the rules of a game, then we basically have an infinite source of high-quality annotated data -- although limited in scope to apply only to the context of a game.

If the rules of the game exist only within the boundaries of the game (such as Chess, or Go, or Starcraft), then the things learned may not generalize well outside of the game. But the expectation is that -- if the context of the game goes through the semantic language space (or through "coding space", in the context of training coding models) -- then the things that the LLM learns within the game will have general applicability in the general space.

So if I can understand your suggestion, to make a similar RLAIF-type improvement to adversarial training, then instead of using a clearly-defined game structure to define the game space, then we would use another LLM to act as the "arbiter" of the game -- perhaps by first defining the rules of a challenge, and then judging between the two competitors which response is better.

Instead of needing to write code to say "Player A wins" or "Player B wins", using an LLM to determine that would shortcut that.

That's an interesting idea, and I need to mull it over. My first thought is that -- I was trying to get away from "fuzzy" reward models and instead use something that is deterministically "perfect". But maybe the advantage of being able to move more quickly (and explore more complex game spaces) would outweigh it.

I need to think this through. There are some situations where I could really see your generalized approach working quite well (such as the proposed "Adversarial Gandalf" game -- using an LLM as the arbiter would probably work quite well), but there are others where using an outside tool (such as a compiler, in the case of the code-vulnerability challenges) would still be necessary.

I wasn't aware of the RLAIF paper before -- thank you for the link! You've given me a lot to think about, and I really appreciate the dialog!

The trouble is that those board games don't translate well to other domains. But if the game space can operate through the realm of language and semantics, then the hope is that we can tap into the adversarial growth curve, but for LLMs.

Up until now, everything that we've done has just been imitation learning (even RLHF is only a poor approximation "true" RL).

What a value proposition! ;-)

Why wouldn’t AI change it? Everyone is expecting that it will, and it’s already starting to happen, just visit Amazon. The biggest reasons are that low-effort AI produced works by lazy authors & publishers may drown out the great works and make the tiny percentage far tinier and much harder to find, which may prevent many great works from ever being “discovered” and recognized as great. The new ability for many people without skill to use AI produce works that compete with skilled manual creation is a huge disincentive for creators to spend their lives studying and honing their skills. I’d bet there’s a hollowing out of the arts already occurring in universities globally. My interaction with college students over the last couple of years has very suddenly and dramatically turned into discussions about AI and concern about whether there will even be jobs for the subjects they’re studying.

This is basically what many authors do anyway. They write something. Read it at a later time and then rewrite it.

I kno that the AIs broke shortly after - then the "offense" to the essentially rule 34 type shit - People used AI to make T Swift nude!! How could they - said no one. All that type stuff will happen and we may lose access.

Microsoft is never going back. Google is never going back. Amazon, X/Tesla, Facebook... do you understand?

Do you think their developers deal with a broken AI?? Haha, nah - there are reason some of the less clued in staff think their AIs are "awake" - I. house AI at Microsoft is many years ahead of copilot in its current and likely near foreseeable future state.

To be clear, the time to stop this has passed, we can still opt to reflect it but it will never go away.

I'm sorry, I don't buy it. The "it's too dangerous to release" line has turned out every single time to just be a marketing blurb to get people hyped for whatever it is that they haven't yet released but most assuredly will release. It's spouted either by researchers who are naturally overconfident in their own research field or by the executives of major corporations who would benefit immensely if prospective users and governments overestimated their tech's capabilities.

I used to think this way, then I got bored of hex editing in memory values of games I was playing to cheat.

Is there challenge in hunting down memory locations for important player data? Yes. But it is a different challenge than playing the actual game, and winning at one should not be confused with winning at the other.

For using other models it should be pretty straightforward to just modify the api functions to suit whatever model is being used -- would be fun to try out custom models! (Feel free to pull request the repo btw if you do modify such things)

An idea we had initially was actually to use an open-source model and fine-tune it using the DB of responses (including the hidden violation bot and refinement bot outputs) collected from people playing the game. That way the game could get better and better over time as more user data gets collected.

Disclaimer we actually did implement this via postgres and now have thousands of responses from players in case anyone wants to follow through on this idea.

It's someone's weekend project, there's absolutely no bar whatsoever.

The project is great imo, I might PR some stuff even.