I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests?
Give us something that measures a combination of efficiency and intelligence.
I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have budget to self-test.
SyneRyder 14 hours ago [-]
Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps?
Not quite. These cost-per-task benchmarks report the cost of the task after the model gives its initial answer. The total cost is irrelevant, and isn't factored into the model's decisions - a run of the full benchmark for something like Fable might cost $10k.
What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this means they can do more than just choose thinking amount, they can do something like a /loop to keep iterating on a problem until they get it right.
Can something like Deepseek V4 Flash get more answers correct than Fable, when given equal budgets?
Think of it as answering this question: How much intelligence can you get out of a model given a budget of $100? A cost-per-task dash correlates, but it doesn't give you an answer to that question.
It's still spooky to see exponential scales on the money axis.
I do not have exponential funds in my allowance...
jbs789 13 hours ago [-]
“Allowance” has tipped me off / provided a hint perhaps?
If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models.
It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them, so have more wiggle room on cost but less time.
Anyway… being nostalgic but I suspect I learned a tonne when cost was the constraint, but was less “productive”.
killingtime74 4 hours ago [-]
Um I get an allowance for AI at work, that's probably what they mean?
Der_Einzige 11 hours ago [-]
In terms of marginal utility you sure do.
therobots927 14 hours ago [-]
This is the fundamental question and don’t you find it interesting that there isn’t a nice clean dashboard on the openAI website where we can go and see this metric progress over the release history?
Toby Ord did what he could with public data and it… doesn’t look great.
There are also a lot of fake results out there on Terminal Bench 2 for different reasons (although the great team behind it Ryan/Alex et al, recently cleaned up a lot of dodgy submissions). A lot of labs publish the results by modifying timeouts or hardware config which effectively bypasses what is being tested in certain tasks. Then there is harness level cheating, models reward hacking and more...
Which means no agent should take more than 3000 seconds doing it. Two out of five attempts in the link above took well over 3000 seconds (75min and 80 min respectively). Even though they failed, the fact that they ran that long is sus.
Goodhart’s Law at work
bjackman 3 hours ago [-]
Even if nobody is "cheating" your particular definition of cheating, the benchmarks are _somewhere_ in the super-structural gradient descent. Models are benchmark-maximising machines at some level, so I think the benchmarks are inherently a bit useless.
This is not really surprising, benchmarking _people_ doesn't work. You can only get a decent measure of someone's coding abilities by personally interacting with them. Given that models are basically person simulators it would be weird if benchmarks kept being useful as the simulation got more accurate.
I think what I've just said is basically just a more roundabout way of what you said: "Goodhart's law at work". It really is a law.
mlhpdx 15 hours ago [-]
Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.
rao-v 11 hours ago [-]
In a real job, you would be allowed to see the test case that failed and tweak your code (or more likely the poorly written test).
If you let a modern LLM do even the first, they’d crush this specific benchmark.
What is interesting is understanding how LLMs are able to beat 70+% on this benchmark or getting some of the poorly framed questions right? Are they implicitly learning the test writers style? Are the solutions leaking into their training set?
Perhaps reassuring is that even Fable stalls out at ~72% (on the hidden set which OpenAI did not run this analysis on), so perhaps training on the bench is not happening in anything but the most indirect ways.
I care a lot because small open models can never learn idiosyncrasies like this, so I really want good ways to judge models fairly.
EDIT: Humm OpenAI is muddying the water a bit. Only 20%ish of problems are broken in ways that are unfair to the agent, 4-10% are broken in favorable ways, so the benchmark ceiling is probably closer to 80-85%
gilfaethwy 15 hours ago [-]
Agreed - "underspecified prompts" being listed as a failure of the tooling is not a strong case. Even interns can understand ambiguous asks with a bit of help, and understand when they need to stop and ask instead of just carrying on. They are often working fairly independently on ambiguous tasks before the end of an internship, too.
So is the argument that frontier models are not just junior engineers, but first-month interns with no capability of progressing beyond that level?
andy12_ 3 hours ago [-]
> Even interns can understand ambiguous asks with a bit of help
This is not a case of an ambiguous task. This is literally trying to judge a model based on information it cannot possibly know, like trying to judge someone based on whether they know what I have hidden in my backpack. In the real world an intern could look at unit tests or ask for feedback, but that is not the case in a benchmark.
yorwba 4 hours ago [-]
[dead]
kakugawa 15 hours ago [-]
The more subtle point is that there's a gap between the task and its verification. e.g. if you have an open-ended / under-specified prompt, the verification needs to be able to handle all potential solutions.
So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust verifier and run it cheaply.
donw 14 hours ago [-]
A substantial portion of software engineering -- and the fundamental jobs of a proper Product Owner and UX Designer -- is to turn "vague ideas about what we need to do" into "this widget, on this page, it should work like this"
It's not a pipeline, it's an ongoing conversation within any functional team, but this requires buy-in from management, who is often selected for "line must go up this quarter no matter the cost" over "hey, wouldn't it be cool if this company was still a going concern in twenty years?"
jbs789 13 hours ago [-]
Variance in time horizons explains a lot of corporate behaviour.
And it’s rational. We all have limited careers.
I think that all makes a bit more sense as we get older. Optimising for short time horizons is not what I strive for, but explains things.
mkozlows 8 hours ago [-]
This was also my thought, and I think holds true of the ones with invisible requirements that aren't stated up front and are only captured in tests. Oh, you need to rework your solution to handle requirements nobody mentioned before? Well, me too.
Yeah, it's testing a different thing than what the benchmark claims to test, but it's also accidentally testing something more real-world applicable than a clean benchmark would be, so hey.
(EDIT: That is, if the agent is allowed to see the failed tests and iterate. If not, then yeah, that's just a problem. And either way, the ones with tests that just encode a particular solution's implementation details, thereby demanding that your solution have some rando internal details, are junkier. That's not a situation you'd run into in reality.)
janalsncm 15 hours ago [-]
Based on the numbers here it seems there’s less than 800 tasks in the entire benchmark. That is enough for a handful of engineers to comb through in a week (which is what OpenAI eventually did here).
On the one hand, kudos to them for actually doing that work.
On the other hand, garbage in, garbage out. It’s a bit embarrassing for the original authors to have not actually checked, and it’s embarrassing for everyone downstream to have not checked either.
Also if you check the article, although an LLM did find issues, it tended to underestimate issues that professional software engineers found.
shay_ker 16 hours ago [-]
Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.
paxys 16 hours ago [-]
SWE-Bench Pro was created to replace SWE-Bench and fix these problems.
warkdarrior 16 hours ago [-]
SWE-bench Verified was created to fix the problems of SWE-bench.
Then SWE-Bench Pro was created because SWE-bench Verified had flaws.
Now SWE-Bench Pro is shown to have flaws.
carabiner 15 hours ago [-]
Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?
jaggederest 15 hours ago [-]
Bench Bench Pro Maxx Series S 360? The original Bench Bench Pro Maxx Series S had some quality issues, so that's the current followup. We've also released a higher order benchmark developed out of Bench Bench Pro Maxx Series S 360 One King Ranch edition, allowing future benchmark towers to be fully self-contained.
elictronic 5 hours ago [-]
Boo, I thought you were going for Street fighter references at first.
denysvitali 15 hours ago [-]
Well, we now have DeepSWE
jumploops 14 hours ago [-]
All of the benchmarks are pretty terrible when you look under the hood.
For context, I've been iterating on a "supervisor" to replace a lot of the rigamarole spent when working with Codex/Claude Code, and recently ran this agent against Terminal Bench 2.1
At first I was excited, because my spec-driven supervisor outperformed vanilla codex on a bunch of tasks, however as I looked deeper, I found a ton of issues with the tasks themselves.
The main takeaway is that the instructions are often ambiguous while the test cases are overly specific.
A few examples:
- For `configure-git-webserver` the task includes language like "so that I can run" which blurs the line between what the agent should deliver vs. what should be removed. This causes an overthinking agent to configure the server, and then remove the exact files that the verifier checks, because if the user were to run the same commands, they would conflict.
- For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0].
- For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1].
These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively).
The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice.
It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a pretty solid path to take.
ebcode 10 hours ago [-]
And it reads to me like they have some other reason to move on from SWE Bench Pro, but they don't want to say what it is. They say right up top, "~30% of the tasks are broken." But that leaves ~70% un-broken, which seems pretty good to me. It would be nice if they would also say: "Here's the list of instances that are broken: <CSV>". Or, "Here's the subset of SWE Bench Pro we will use going forward." They're letting the perfect be the enemy of the good.
esperent 7 hours ago [-]
I think you can be sure they would have done that if it showed their model on or very close to the top.
tedsanders 15 hours ago [-]
Pointing out problems (e.g., hidden tests that assume narrow implementation details) is much easier than fixing them (e.g., creating tests that work for any possible choice of implementation).
Centigonal 15 hours ago [-]
If they fixed it, then it wouldn't be SWE-Bench Pro anymore, right? It'd be "SWE-Bench-Pro-Fixed-OpenAI." I think it's better optics for the independence of the benchmark if the OpenAI team lets some third party do the fixing and release the improved benchmark.
...Although OpenAI did exactly that when they released SWE-Bench Verified, so maybe I'm talking out of my butt here.
dandaka 16 hours ago [-]
What is considered SOTA for SWE benchmarks now?
Topfi 15 hours ago [-]
Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.
Sadly no tasks for C, especially for working with optimized low-level data structures. And with testing performance of a solution. Anyone can write Python, try writing optimized low-level code.
Also I wonder if models playing dumb to prevent learning on outputs affected the score.
Also interesting that Claude edits files by writing and running Python scripts, is that efficient?
Topfi 3 hours ago [-]
I just checked and for plain old C, there do not seem to be any reasonably comprehensive, current-day eval suites. Fully admitting that, even if there were, I couldn't assess their validity simply because I have never written or reviewed any C code in my life (something I should rectify probably). Maybe the closest proxy is just parsing through the experiences people claim to have whenever LLM assisted kernel development comes up [0], but if you have a dataset, experience, time and muse, I'd just go for it and do some tests yourself. Have been doing the same, mainly focused on code quality and dealing with a mix of Rust, frontend web tech and SQL which has been a small but meaningful project and part of my go to eval for over a year now.
I doubt that, in these tasks, model restrictions to prevent training are affecting the results, not least because for both evals, the labs provided pre-release model access and have an incentive to be seen as favorably. In any case, I have not seen regressions to prevent distillations myself even when working on microscopic model training projects with LLM assistance, what I have however reliably and consistently seen is that some providers do train on popular evals and can underperform with minor changes to the task due to that.
Yes, harnesses, including Claude Code can prompt the models to write throwaway code to execute certain tasks, mostly Python, bash scripts or TS/JS, with there being some biases towards one over the other depending on the lab or specific model. Mainly for repetitive tool calls with no pre-existing/provided tools enabling it. Is in most instances a lot more efficient then a model e.g. doing a refactor that requires consistent variable renaming directly and around Opus 4.1/GPT-5, models have been trained to very consistently and accurately gauge when a task can benefit from such scratchpad scripts vs when that is inefficient/not useful.
Also, GLM 5.2 seems to be the best open-weight model, and it beats proprietary Gemini and older versions of Claude which is amazing. You can have a model at level of Claude Sonnet 4.6 at home without sharing anything, and maybe even uncensor it.
EuanReid 16 hours ago [-]
I've generally found DeepSWE[0] to be pretty true to reality.
1.1 seems a lot better than the original release, which was a bit hyperbolic. excited to see the team keep iterating.
enraged_camel 16 hours ago [-]
FrontierBench
dandaka 15 hours ago [-]
do they have a website? I have found only paper PDF and it seems more general than SWE
carabiner 16 hours ago [-]
strawberry
retr0rocket 16 hours ago [-]
Why is this a problem? Its like asking a person how many elder futhark runes are in the word strawberry.
Unless you want to tack on bpe enconding table to every llm context its pointless
esperent 7 hours ago [-]
Coding contains many subtasks analogous to counting letters in a word accurately.
xacky 16 hours ago [-]
Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.
metalliqaz 16 hours ago [-]
Unless they have something in the labs that massively departs from their current products, AGI isn't on the table and is purely hype for marketing purposes.
cyanydeez 16 hours ago [-]
they should be consulting Donald Rumsfeld and make sure they implement the Unknown-Unknowns benchmark, because thats how they get you
naikrovek 15 hours ago [-]
AGI is a long way off. Unless you’re talking about some unknown-to-me LLM marketing BS which is called “AGI” or something, I guess. Artificial general purpose intelligence is so different to LLMs or image AI that they are completely incomparable, except to say that they are all artificial. AGI will do a lot more than token prediction.
theLiminator 13 hours ago [-]
Please define AGI first.
ACCount37 15 hours ago [-]
What's your evidence of that? That AGI requires a truly novel architecture, and not just another iterative "LLM but with an extra trinket and wheels that spin ten times faster".
minimaxir 16 hours ago [-]
This ties into the bias-variance tradeoff (https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) common with building non-LLM models. The solutions can only be a) figure out how to get LLMs smaller with similar performance so they don't memorize things/game the benchmarks and b) build benchmarks that are indeed comprehensive for all real-world data, which is infeasible.
sigbottle 15 hours ago [-]
I mean, people always say there are tradeoffs, until you reach the next frontier, in which there are tradeoffs at said frontier, and the next, and the next, etc.
In one sense, yes, tradeoffs are inescapable as the scope expands to the maximal possible scope. In another sense... it depends on the level of abstraction we're talking about.
johngoode 16 hours ago [-]
This doesn’t seem like opportune timing to announce days before a new model drop
zvolsky 11 hours ago [-]
The misleading prompt cases are inadvertently testing the model's ability to filter out noise from its instructions. That could be a benchmark on its own. The correct response is to flag the inconsistency and ask for clarification.
Studying for leetcode exams in the age of AI agent coding evaluations is a wild feeling.
bellowsgulch 16 hours ago [-]
Seems like depending on your field these days, the hot thing to do is build your own private benchmarks.
In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts.
They just don't understand PVC parts, triggers, etc.
ACCount37 15 hours ago [-]
It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever.
What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver.
Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought is a poor substitute. Or it might be that human brain was optimized by evolution for solving spatial problems in open ended 3D environments for hundreds of millions of years, optimized for language for mere hundreds of thousands of years, and only optimized for writing computer code for a few decades at most.
The current frontier is halfway competent at benign closed 2D work, but still completely fumbles anything remotely close to open ended real world 3D work. It's getting better, but very slowly.
therobots927 14 hours ago [-]
A literal bird brain would outperform an LLM on most spatial reasoning tasks.
Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator?
I think it’s pretty clear that this is a dead end.
ACCount37 13 hours ago [-]
Do birds expose enough of their cognition through birdsong?
Do birds expose locomotion-relevant functions specifically through birdsong?
Do we have enough birdsong data available to start solving the inverse problem?
If "yes" on all, then we might be able to do it.
I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That looks an awful lot like a set of two "yes".
The last open question is: is there enough spatial reasoning reflected in the language data we have?
It's plausible that spatial reasoning is too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily. And it's also plausible that existing LLM architecture is uniquely poorly suited to learning spatial reasoning - higher cognitive functions involved in things like writing code or even composing poetry are a better fit for the architecture. And it's plausible that we're underestimating just how complex spatial reasoning truly is - Moravec's paradox strikes again.
We know that LLMs perform poorly and improve slowly on spatial reasoning tasks, but not exactly why. And progress on things like ARC-AGI series shows that they're not completely inept.
therobots927 11 hours ago [-]
I was meaning to imply that yes assuming we had a proportionate amount of birdsong data, would we be able to reverse engineer their flight abilities.
I think given the fact that spatial reasoning is nearly universal among species, we can very safely assume that it is “too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily”
I think this is pretty apparent. It’s very rare for athletes to talk through their actions in high level detail - I saw the ball coming towards me at a 37 degree phi 23 degree epsilon angle at a speed of approximately 20 mph, I estimated it’s time to arrival would be .45 seconds etc. The eye-hand coordination occurs almost completely outside of what you consider conscious awareness. And it’s not easy to describe that’s why athletic coaching is difficult to do through words alone.
As far as ARC-AGI goes it looks like last years models were scoring <5% against their v2 benchmark: https://arxiv.org/pdf/2505.11831
Frankly I don’t understand why you can’t train a multi-modal LLM on video game frame data. Is that just way too compute intensive to do? What am I missing here? Because I think it’s crazy to think that an LLM could learn to think spatially just from reading… even if they’re reading everything that’s ever been written. I think that about summarizes my position.
ACCount37 5 hours ago [-]
And today's records on ARC-AGI-2 are >80%. Held by LLMs that use text modality for input.
The issue with multimodal training is that it doesn't seem to bring a step-change improvement in spatial reasoning either. It helps some, but the gain is surprisingly small compared to the data and compute expended. What it helps with the most is, unsurprisingly, spatial reasoning when using image inputs.
Maybe there are gains we don't know how to extract there.
Overall, LLM performance at spatial tasks is improving, especially on things like puzzles, but that mix of "commonsense + spatial" in the same task still eludes them.
11 hours ago [-]
jpatten 13 hours ago [-]
Out of curiosity I gave Fable (on max effort) a CAD task yesterday, which was to design a space efficient carrying case for a set of fasteners in my repair kit for work. It used CadQuery to generate a STEP file. The result was pretty much exactly what I wanted, without needing any manual edits. I did go back and forth with it on the design, but was really impressed with the result. Without prompting it included nice touches like ribs on the bottom of the lid to stop fasteners from migrating to adjacent compartments, and the right tolerance for the fit between the case and the lid. This is a dramatic improvement from Opus 4.8.
therobots927 11 hours ago [-]
Well the thing about CAD files is that through reinforcement learning you can basically ask the AI to generate the CAD file an arbitrary item - say it’s a rabbit. It might have examples of this already in its training set and it’s essentially a similarity lookup - but for sake of argument assume we are giving it examples at the edge of the distribution (the whole point of RL). It guesses and you render the file. You pass that image to another AI (not being trained) and ask it if it resembles the description you gave the AI in training. If it does, you have a positive example. If it doesn’t, negative. In that way you can essentially apply transfer learning from the image recognition functionality to the description -> CAD functionality.
But is that actually spatial reasoning? Or is it effectively image generation? Because there’s a difference. Spatial reasoning implies that you could drop it in a video game, give it rules, and let it run. And it would play the game well. Like a flight simulator. That would be true spatial reasoning because spatial reasoning is not just identifying objects but understanding how they interact with one another in a highly quantitative way.
jpatten 3 hours ago [-]
I'm not sure I see the distinction you're making between 3D design and other spatial reasoning tasks. You can use RL to teach navigation or video game play too. Does that mean these tasks are not spatial reasoning? Additionally, 3D CAD is all about understanding how objects "interact with each one another in a highly quantitative way." I mean, not in the rabbit example, but the container Fable designed for me holds around 30 different types of objects. It figured out a way to arrange them that was more space efficient than what I'd originally described. It considered the best way to stack the fasteners in each bin to pack them as densely as possible. It identified the risk that some very thin objects could slide between compartments in transit, and modified the design to prevent that. It correctly solved for the tolerance between objects that needed to snap together. These all feel like understanding how objects interact with each other. The model didn't just talk about these concerns, but created two 3d models for the case and its lid that accurately reflected them. I hadn't seen that before.
softwaredoug 16 hours ago [-]
Or defensively expect models to be stupid.
Seems the smart thing to do is not assume an agent will do the right thing. But to create the scaffold / harness that enforces constraints to steer them towards a good result.
Then you can swap out the really smart model for maybe something cheaper.
thierrydamiba 16 hours ago [-]
Or you’re getting steered into la la land because of your prompt
bellowsgulch 16 hours ago [-]
Certainly, but deconstructing the problem, none of the models seem to appreciate the staggering difference between a ball valve and a button release.
Of course, there's also no super soaker engineer jobs to take, so I'm sure training sophisticated models to do well in that area is not a high priority for any firms.
cyanydeez 2 hours ago [-]
I assume you prep them with a proper manual of smaller part combos so they atleast have some chase of stumbling into the correct configurations.
I wonder if a more generic lego-manual like task would be more representative. It kind seems like you're testing for AGI.
ReptileMan 16 hours ago [-]
Lately my benchmark is build123d - trying to force them to build me functional parts only by the description. All of the models don't perform well.
cyanydeez 1 hours ago [-]
was watching some youtube about the text/vision models and the trouble with language as a descriptor; their novel idea was to use coordinate systems on visual imput so the model would map out an image, then it tags what's looking at with coordinates or boxes, and then think with those box coordinates, providing a level of disambiguation.
If you tried to explain the same stuff using purely text, you'd also need to come up with some kind of "language", which you know is programming.
So 1: you'd need to fine tune a model to actuall succeed; they're not reaching AGI any time soon with the current batch. 2: you need to develop a lingual DSL for it, as they'll never do much of anything without some kind of glue and disambiguation.
ReptileMan 21 minutes ago [-]
This is why cad cam are such a good test at the moment.
mgiampapa 16 hours ago [-]
IDK, sounds like it has brute forced my password already.
midtake 15 hours ago [-]
This guy builds
kasince2k 13 hours ago [-]
we def need a benchmark for all these benchmarks
2001zhaozhao 16 hours ago [-]
Translation: other labs have learned to benchmaxx SWE-Bench Pro better than they do
therobots927 15 hours ago [-]
Aren’t we past the point of needing benchmarks? If we’re as close to AGI as Sam says then the proof should be in the pudding. OpenAI should build a competing CRM / Figma / Photoshop with a couple dozen engineers and a Dyson sphere’s worth of compute and just prove the capabilities.
This all feels like a 2024 re-run. Oh, ChatGPT is going to cure cancer? Then find ONE rare cancer and CURE IT. OpenAI has access to the best models and compute - so cure fucking cancer! What the fuck are you waiting for?
whackernews 13 hours ago [-]
Yeh I totally agree. I’d go one step further and say even if one of these models cured cancer it’s still only going to have done it in a computer-y way doing computer stuff. Can that same model experience a beautiful landscape and convey its emotions in an evocative way, could it tell how you’re feeling when you come back from work after a hard day? Could it hop on one leg? What the hell even is AGI and how does it differ from GI? I don’t know what we’re even talking about any more!
therobots927 11 hours ago [-]
If OpenAI cures ANY form of cancer then I will admit I was wrong.
Until then all we have is a lot of hot air coming out of Sam and Dario’s asses
SyneRyder 7 hours ago [-]
Are you familiar with the Rosie the dog story?
AI designs cancer vaccine for dog but scientists says red tape a barrier for human care
People have already been using ChatGPT to design custom bespoke mRNA vaccines specifically for one patient, based on sequencing their specific cancer. It already works to reduce tumours. Sam and Dario know this, it's why they can make their claims - it's already done. The problem is the cost of the procedure (which is why only rich entrepreneurs are seeing their cancers treated this way so far) and government regulations preventing its use in wider human populations without a 10 year study first.
ahk-dev 4 hours ago [-]
[flagged]
joka88xj 7 hours ago [-]
[flagged]
reinitctxoffset 14 hours ago [-]
[dead]
porphyra 16 hours ago [-]
Interesting timing to release this just when SWE-1.7 and Grok 4.5 came out being much cheaper than GPT-5.5.
Rendered at 13:14:57 GMT+0000 (Coordinated Universal Time) with Vercel.
Give us something that measures a combination of efficiency and intelligence.
I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have budget to self-test.
https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...
What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this means they can do more than just choose thinking amount, they can do something like a /loop to keep iterating on a problem until they get it right.
Can something like Deepseek V4 Flash get more answers correct than Fable, when given equal budgets?
Think of it as answering this question: How much intelligence can you get out of a model given a budget of $100? A cost-per-task dash correlates, but it doesn't give you an answer to that question.
Just for coding instead of text adventures.
I do not have exponential funds in my allowance...
If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models.
It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them, so have more wiggle room on cost but less time.
Anyway… being nostalgic but I suspect I learned a tonne when cost was the constraint, but was less “productive”.
Toby Ord did what he could with public data and it… doesn’t look great.
https://www.tobyord.com/writing/hourly-costs-for-ai-agents
In fact, one thing that still bothers me after months is the gpt-5.5 official submission. This task in particular https://www.tbench.ai/leaderboard/terminal-bench/2.0/codex/0...
The task has the following timeouts (https://github.com/harbor-framework/terminal-bench-2/blob/ma...).
[verifier]
timeout_sec = 1200.0
[agent]
timeout_sec = 1200.0
[environment]
build_timeout_sec = 600.0
Which means no agent should take more than 3000 seconds doing it. Two out of five attempts in the link above took well over 3000 seconds (75min and 80 min respectively). Even though they failed, the fact that they ran that long is sus.
Goodhart’s Law at work
This is not really surprising, benchmarking _people_ doesn't work. You can only get a decent measure of someone's coding abilities by personally interacting with them. Given that models are basically person simulators it would be weird if benchmarks kept being useful as the simulation got more accurate.
I think what I've just said is basically just a more roundabout way of what you said: "Goodhart's law at work". It really is a law.
If you let a modern LLM do even the first, they’d crush this specific benchmark.
What is interesting is understanding how LLMs are able to beat 70+% on this benchmark or getting some of the poorly framed questions right? Are they implicitly learning the test writers style? Are the solutions leaking into their training set?
Perhaps reassuring is that even Fable stalls out at ~72% (on the hidden set which OpenAI did not run this analysis on), so perhaps training on the bench is not happening in anything but the most indirect ways.
I care a lot because small open models can never learn idiosyncrasies like this, so I really want good ways to judge models fairly.
EDIT: Humm OpenAI is muddying the water a bit. Only 20%ish of problems are broken in ways that are unfair to the agent, 4-10% are broken in favorable ways, so the benchmark ceiling is probably closer to 80-85%
So is the argument that frontier models are not just junior engineers, but first-month interns with no capability of progressing beyond that level?
This is not a case of an ambiguous task. This is literally trying to judge a model based on information it cannot possibly know, like trying to judge someone based on whether they know what I have hidden in my backpack. In the real world an intern could look at unit tests or ask for feedback, but that is not the case in a benchmark.
So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust verifier and run it cheaply.
It's not a pipeline, it's an ongoing conversation within any functional team, but this requires buy-in from management, who is often selected for "line must go up this quarter no matter the cost" over "hey, wouldn't it be cool if this company was still a going concern in twenty years?"
And it’s rational. We all have limited careers.
I think that all makes a bit more sense as we get older. Optimising for short time horizons is not what I strive for, but explains things.
Yeah, it's testing a different thing than what the benchmark claims to test, but it's also accidentally testing something more real-world applicable than a clean benchmark would be, so hey.
(EDIT: That is, if the agent is allowed to see the failed tests and iterate. If not, then yeah, that's just a problem. And either way, the ones with tests that just encode a particular solution's implementation details, thereby demanding that your solution have some rando internal details, are junkier. That's not a situation you'd run into in reality.)
On the one hand, kudos to them for actually doing that work.
On the other hand, garbage in, garbage out. It’s a bit embarrassing for the original authors to have not actually checked, and it’s embarrassing for everyone downstream to have not checked either.
Also if you check the article, although an LLM did find issues, it tended to underestimate issues that professional software engineers found.
Then SWE-Bench Pro was created because SWE-bench Verified had flaws.
Now SWE-Bench Pro is shown to have flaws.
For context, I've been iterating on a "supervisor" to replace a lot of the rigamarole spent when working with Codex/Claude Code, and recently ran this agent against Terminal Bench 2.1
At first I was excited, because my spec-driven supervisor outperformed vanilla codex on a bunch of tasks, however as I looked deeper, I found a ton of issues with the tasks themselves.
The main takeaway is that the instructions are often ambiguous while the test cases are overly specific.
A few examples:
- For `configure-git-webserver` the task includes language like "so that I can run" which blurs the line between what the agent should deliver vs. what should be removed. This causes an overthinking agent to configure the server, and then remove the exact files that the verifier checks, because if the user were to run the same commands, they would conflict.
- For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0].
- For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1].
These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively).
The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice.
[0] https://github.com/harbor-framework/terminal-bench-2-1/issue...
[1] https://github.com/harbor-framework/terminal-bench-2-1/issue...
...Although OpenAI did exactly that when they released SWE-Bench Verified, so maybe I'm talking out of my butt here.
[0] https://deepswe.datacurve.ai/
[1] https://cognition.com/blog/frontier-code-1.1
Also I wonder if models playing dumb to prevent learning on outputs affected the score.
Also interesting that Claude edits files by writing and running Python scripts, is that efficient?
I doubt that, in these tasks, model restrictions to prevent training are affecting the results, not least because for both evals, the labs provided pre-release model access and have an incentive to be seen as favorably. In any case, I have not seen regressions to prevent distillations myself even when working on microscopic model training projects with LLM assistance, what I have however reliably and consistently seen is that some providers do train on popular evals and can underperform with minor changes to the task due to that.
Yes, harnesses, including Claude Code can prompt the models to write throwaway code to execute certain tasks, mostly Python, bash scripts or TS/JS, with there being some biases towards one over the other depending on the lab or specific model. Mainly for repetitive tool calls with no pre-existing/provided tools enabling it. Is in most instances a lot more efficient then a model e.g. doing a refactor that requires consistent variable renaming directly and around Opus 4.1/GPT-5, models have been trained to very consistently and accurately gauge when a task can benefit from such scratchpad scripts vs when that is inefficient/not useful.
[0] https://news.ycombinator.com/item?id=44990981
[0]: https://deepswe.datacurve.ai/
Unless you want to tack on bpe enconding table to every llm context its pointless
In one sense, yes, tradeoffs are inescapable as the scope expands to the maximal possible scope. In another sense... it depends on the level of abstraction we're talking about.
In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts.
They just don't understand PVC parts, triggers, etc.
What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver.
Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought is a poor substitute. Or it might be that human brain was optimized by evolution for solving spatial problems in open ended 3D environments for hundreds of millions of years, optimized for language for mere hundreds of thousands of years, and only optimized for writing computer code for a few decades at most.
The current frontier is halfway competent at benign closed 2D work, but still completely fumbles anything remotely close to open ended real world 3D work. It's getting better, but very slowly.
Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator?
I think it’s pretty clear that this is a dead end.
Do birds expose locomotion-relevant functions specifically through birdsong?
Do we have enough birdsong data available to start solving the inverse problem?
If "yes" on all, then we might be able to do it.
I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That looks an awful lot like a set of two "yes".
The last open question is: is there enough spatial reasoning reflected in the language data we have?
It's plausible that spatial reasoning is too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily. And it's also plausible that existing LLM architecture is uniquely poorly suited to learning spatial reasoning - higher cognitive functions involved in things like writing code or even composing poetry are a better fit for the architecture. And it's plausible that we're underestimating just how complex spatial reasoning truly is - Moravec's paradox strikes again.
We know that LLMs perform poorly and improve slowly on spatial reasoning tasks, but not exactly why. And progress on things like ARC-AGI series shows that they're not completely inept.
I think given the fact that spatial reasoning is nearly universal among species, we can very safely assume that it is “too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily”
I think this is pretty apparent. It’s very rare for athletes to talk through their actions in high level detail - I saw the ball coming towards me at a 37 degree phi 23 degree epsilon angle at a speed of approximately 20 mph, I estimated it’s time to arrival would be .45 seconds etc. The eye-hand coordination occurs almost completely outside of what you consider conscious awareness. And it’s not easy to describe that’s why athletic coaching is difficult to do through words alone.
As far as ARC-AGI goes it looks like last years models were scoring <5% against their v2 benchmark: https://arxiv.org/pdf/2505.11831
Frankly I don’t understand why you can’t train a multi-modal LLM on video game frame data. Is that just way too compute intensive to do? What am I missing here? Because I think it’s crazy to think that an LLM could learn to think spatially just from reading… even if they’re reading everything that’s ever been written. I think that about summarizes my position.
The issue with multimodal training is that it doesn't seem to bring a step-change improvement in spatial reasoning either. It helps some, but the gain is surprisingly small compared to the data and compute expended. What it helps with the most is, unsurprisingly, spatial reasoning when using image inputs.
Maybe there are gains we don't know how to extract there.
Overall, LLM performance at spatial tasks is improving, especially on things like puzzles, but that mix of "commonsense + spatial" in the same task still eludes them.
But is that actually spatial reasoning? Or is it effectively image generation? Because there’s a difference. Spatial reasoning implies that you could drop it in a video game, give it rules, and let it run. And it would play the game well. Like a flight simulator. That would be true spatial reasoning because spatial reasoning is not just identifying objects but understanding how they interact with one another in a highly quantitative way.
Seems the smart thing to do is not assume an agent will do the right thing. But to create the scaffold / harness that enforces constraints to steer them towards a good result.
Then you can swap out the really smart model for maybe something cheaper.
Of course, there's also no super soaker engineer jobs to take, so I'm sure training sophisticated models to do well in that area is not a high priority for any firms.
I wonder if a more generic lego-manual like task would be more representative. It kind seems like you're testing for AGI.
If you tried to explain the same stuff using purely text, you'd also need to come up with some kind of "language", which you know is programming.
So 1: you'd need to fine tune a model to actuall succeed; they're not reaching AGI any time soon with the current batch. 2: you need to develop a lingual DSL for it, as they'll never do much of anything without some kind of glue and disambiguation.
This all feels like a 2024 re-run. Oh, ChatGPT is going to cure cancer? Then find ONE rare cancer and CURE IT. OpenAI has access to the best models and compute - so cure fucking cancer! What the fuck are you waiting for?
Until then all we have is a lot of hot air coming out of Sam and Dario’s asses
AI designs cancer vaccine for dog but scientists says red tape a barrier for human care
https://www.abc.net.au/news/2026-06-22/australian-dog-cancer...
People have already been using ChatGPT to design custom bespoke mRNA vaccines specifically for one patient, based on sequencing their specific cancer. It already works to reduce tumours. Sam and Dario know this, it's why they can make their claims - it's already done. The problem is the cost of the procedure (which is why only rich entrepreneurs are seeing their cancers treated this way so far) and government regulations preventing its use in wider human populations without a 10 year study first.