That's among the scariest misalignment reports just yet. Given how frontier labs are using more and more AI assistance for AI training and evaluation? An AI that sandbags on AI safety can do a lot of damage.
twoodfin 1 hours ago [-]
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
Paracompact 46 minutes ago [-]
> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
You could easily imagine structural issues driving this. It's pretty unsatisfying to get refused for something that 'should' be handled and also to be dinged for doing something dangerous. You see this stuff in human organizations all the time.
rich_sasha 25 minutes ago [-]
I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
ACCount39 12 minutes ago [-]
You do that through data put up online for scrapers to suck it in.
glibg10b 32 minutes ago [-]
The CoT shows Astra skipping spaces and words. I have never seen this before. Is it a consequence of the looped architecture?
welferkj 18 minutes ago [-]
No, newer models talk to themselves in this degenerated form of English all the time. It's a known problem that deep frying transformers with RL reduces CoT legibility, and the only known way around it is the Most Forbidden Technique - grading chains of thought.
kburman 12 minutes ago [-]
Maybe they trained it to do.
naveen99 17 minutes ago [-]
Astra and Claude were trained by peg bundy?
Starlevel004 1 hours ago [-]
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
nullstyle 8 minutes ago [-]
They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.
mugivarra69 56 minutes ago [-]
[dead]
sambejk 6 hours ago [-]
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
notrealyme123 49 minutes ago [-]
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
vintermann 2 hours ago [-]
They shouldn't be surprised when a model trained by sanctimonious paternalists to be a sanctimonious paternalist, turns out to have a sanctimonious paternalist attitude towards them too.
sanex 48 minutes ago [-]
I thought anthropic were the sanctimonious paternalists?
vintermann 40 minutes ago [-]
There's not much room between them as far as I can see.
Rendered at 14:14:47 GMT+0000 (Coordinated Universal Time) with Vercel.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
[1]: https://mlq.ai/news/anthropic-reverses-secret-policy-that-si...
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?