Are AI benchmarks useless?

No. They are how the field tracks progress, and a system that scores poorly across many benchmarks is genuinely weaker. The problem is using small differences between leading systems to make a purchasing decision. A large gap tells you something. A three-point gap usually does not, because most published comparisons report no statistical test.

What is benchmark contamination?

It is when material from the test has appeared in the model's training data, so the model can partly recall the answer rather than work it out. Research on SWE-bench found models identifying the correct file from an issue description alone at up to 76% accuracy with no access to the repository, against up to 53% on repositories outside the benchmark.

How do I test AI properly for my own business?

Assemble thirty to fifty real tasks from your own work, write down what a good answer looks like before you start, then run every candidate on the same set and run it twice. If two systems come out close, decide on price, data handling and reliability instead. It takes a morning and it keeps working as new models appear.

A vendor told me their evaluation uses a private test set. Is that better?

Not necessarily. The February 2026 saturation study found that resilience to saturation came from expert curation rather than from keeping test data private. A privately held test set is also one you cannot inspect, so it is reasonable to ask how it was designed and what it contains.

What benchmark scores do not tell you

Someone is going to show your board a chart where their AI scores 94 and the alternative scores 91. Three separate studies published since late 2025 explain why that chart cannot carry the weight being put on it, and what to do instead.

C4 Studios29 August 20267 min read

Benchmark scores are a poor basis for choosing an AI system for your business. A review of 445 benchmarks found only 16% used any statistical test when comparing results, so most published gaps have no error bars. A study of 60 language model benchmarks found nearly half have saturated, meaning they no longer separate the leading systems. And at least one heavily cited benchmark has been shown to reward memorisation rather than reasoning. None of this makes the models bad. It makes the scores a weak reason to prefer one over another, and it means the only test that answers your question is one run on your own documents.

Why this matters to a firm that is buying, not building

You are not going to read an evaluation paper. You are going to sit in a room while someone shows you a slide, and the slide will have numbers on it. The question worth answering is how much those numbers should move your decision.

The short answer is less than the slide implies, and the reasons are now documented in the research rather than being a matter of opinion. That is a useful thing to know before you spend money, and it is a fair thing to raise with anyone selling to you.

Most published comparisons have no error bars

In November 2025 a group of researchers published a systematic review of 445 language model benchmarks, assessed by 29 expert reviewers. They were looking at construct validity, which is the question of whether a test measures the thing it claims to measure.

The finding that matters most for a buyer is blunt. In the paper's own words: "Once the responses were scored, 16.0% used uncertainty estimates or statistical tests to compare the results."

Six out of seven benchmark comparisons reported no statistical test at all. So when a chart shows 94 against 91, there is usually nothing in the underlying work telling you whether that three-point gap would survive being run again. It might be a real difference. It might be noise. The published result frequently does not say.

The rest of the review is consistent with that. Only 53.4% presented any evidence for construct validity. 47.8% were measuring phenomena the field does not clearly agree on. 40.7% used constructed tasks rather than real-world ones, and 81.3% scored answers by exact matching, which is the crudest method available and punishes a correct answer phrased differently.

Six out of seven benchmark comparisons report no statistical test. When someone shows you 94 against 91, the underlying work usually cannot tell you whether that gap is real.

Nearly half of benchmarks have stopped discriminating

A second study, published in February 2026, looked at benchmark saturation across 60 language model benchmarks using 14 properties. Saturation is what happens when the leading systems all cluster near the ceiling. The test still produces a number, but the number stops telling you which system is better.

The finding: "nearly half of our benchmarks exhibit saturation, with rates increasing with age." The older a benchmark is, the more likely it has stopped being useful, which is awkward because the older benchmarks are the ones with the most name recognition and therefore the ones most likely to appear on a slide.

The same paper contains a result that runs against common sense and is worth carrying into a vendor conversation. The authors found that "resilience to saturation is impacted by expert-curation, not by public test data." Keeping the test set secret is not what protects a benchmark. Careful expert design is. That matters because "our test set is private" is a claim vendors make, and on this evidence it is not the reassurance it sounds like.

At least one famous benchmark rewards memory

The third study is the most concrete, and it has an article of its own here. Researchers asked whether a model could name the file containing a bug when given only the issue description and no access to the repository. It should not be able to. It managed up to 76% accuracy anyway, falling to merely up to 53% on repositories outside the benchmark.

The short version is that part of a headline score can reflect having seen the answer during training rather than working it out, and from outside you cannot tell which. "When the model has seen the test" walks through that study in full.

What benchmarks are still good for

This is not an argument that evaluation is worthless, and you should be suspicious of anyone who takes it that way. Benchmarks are how the field makes progress legible, and a model that scores badly across the board is genuinely worse than one that does not.

The honest position is narrower. A benchmark tells you something useful when it matches the work you actually do, when it has not saturated, and when the result comes with some indication of variance. Most of the numbers put in front of buyers meet none of those conditions. A large gap between a leading system and a laggard is informative. A small gap between two leading systems usually is not.

What to do instead

Build a test set out of your own workThirty to fifty real tasks, taken from documents your firm actually handles, redacted where needed. The point is that it looks like your work rather than someone else's benchmark. This is a morning's job and it outlasts every model release.
Write down the right answers firstBefore you run anything, record what a good answer looks like for each task. Doing this afterwards means grading against whatever the machine produced, which is how firms talk themselves into being impressed.
Run every candidate on the same setSame tasks, same order, same instructions. If two systems come out close, treat them as equivalent and choose on price, data handling and reliability instead. Those differ far more between vendors than capability does.
Run it more than onceThese systems are not deterministic. Running each task twice will show you how much the output moves on its own, which is the variance the published comparisons mostly do not report.
Ask the vendor which benchmark and how old it isA fair question, politely asked. If the score comes from a well-known benchmark several years old, saturation is a live possibility and it is reasonable to ask what a newer or task-matched evaluation shows.
Keep it and re-run itWhen a new model appears, you already have the test. This is the difference between having an opinion about AI and having evidence about it, and it costs you one morning.

Why we publish this

We sell AI training for businesses and we get asked which model is best. The honest answer is that it depends on your documents, and that the way to find out takes a morning and produces a better answer than any chart.

We would rather hand over the method than have clients choose on a number nobody has tested. It also makes us easier to check, because the same test that grades a vendor grades us.

Where the deciding factor is confidentiality rather than capability, the comparison changes shape entirely, and that is what private AI is about. Our own position on using these tools in client work is at how we use AI.

Sources

Measuring what Matters: Construct Validity in Large Language Model Benchmarks — Bean, Kearns, Romanou, Hafner, Mayne et al., arXiv:2511.04703 (2025)
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation — Akhtar, Reuel, Soni, Ahuja et al., arXiv:2602.16763 (2026)
The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — Liang, Garg & Zilouchian Moghaddam, arXiv:2506.12286 (2025)

Common questions

Are AI benchmarks useless?: No. They are how the field tracks progress, and a system that scores poorly across many benchmarks is genuinely weaker. The problem is using small differences between leading systems to make a purchasing decision. A large gap tells you something. A three-point gap usually does not, because most published comparisons report no statistical test.
What does benchmark saturation mean?: It means the leading systems have all clustered near the maximum score, so the test no longer separates them. A February 2026 study of 60 language model benchmarks found nearly half had saturated, and that older benchmarks saturate more. Those are also the benchmarks most likely to be quoted at you.
What is benchmark contamination?: It is when material from the test has appeared in the model's training data, so the model can partly recall the answer rather than work it out. Research on SWE-bench found models identifying the correct file from an issue description alone at up to 76% accuracy with no access to the repository, against up to 53% on repositories outside the benchmark.
How do I test AI properly for my own business?: Assemble thirty to fifty real tasks from your own work, write down what a good answer looks like before you start, then run every candidate on the same set and run it twice. If two systems come out close, decide on price, data handling and reliability instead. It takes a morning and it keeps working as new models appear.
A vendor told me their evaluation uses a private test set. Is that better?: Not necessarily. The February 2026 saturation study found that resilience to saturation came from expert curation rather than from keeping test data private. A privately held test set is also one you cannot inspect, so it is reasonable to ask how it was designed and what it contains.