AI assistance produces more code and a smaller share of it reaches production. In a randomised trial, experienced developers believed AI had made them 20% faster and measurement showed it made them 19% slower. Across 8.1 million pull requests, AI-authored code merged 32.7% of the time against 84.4% for human-authored code, and waited more than five times longer for a reviewer. Neither study says the tools do not work. Both say the bottleneck moved from writing code to reviewing it, and almost nobody has budgeted for that.
The trial where everyone guessed wrong
In 2025 a research group ran a randomised controlled trial on sixteen experienced open-source developers across 246 real tasks. These were not toy problems. They were jobs in mature codebases the developers had worked in for an average of five years. Each task was randomly assigned to allow or forbid AI tooling.
Before starting, the developers forecast that being allowed to use AI would cut their completion time by 24%. After finishing, having lived through it, they estimated it had cut their time by 20%.
Measured, allowing AI increased completion time by 19%.
The gap between what they experienced and what happened is the finding. It was not a small error and it did not go in the direction of caution. Two independent expert groups were asked to predict the result beforehand: economists said 39% faster, machine learning researchers said 38% faster. Everyone was confidently wrong in the same direction.
They believed it made them 20% faster. Measurement said 19% slower. Nobody in the study noticed while it was happening.
What eight million pull requests show
The second study is much larger and asks a different question. It looked at 8.1 million pull requests across 4,800 teams in 42 countries, and compared what happened to AI-assisted code against human-authored code once it entered review.
Adoption is not in question. Around 88% of developers in that dataset now use AI regularly, up from just under 72% in early 2024. The tools are in the building.
What happens next is the interesting part. AI-authored pull requests merged 32.7% of the time. Human-authored ones merged 84.4% of the time. Fewer than half as often.
They also waited 5.3 times longer before a reviewer picked them up, and they were substantially bigger, running 408 lines at the 75th percentile against 157 for unassisted work. The report also found roughly 1.7 times more issues per pull request in AI-authored code.
Put plainly, more code is being written, it arrives in larger pieces, it sits in the queue longer, and a much smaller share of it survives review.
Why the two studies agree without being about the same thing
One study measured individual task time under controlled conditions. The other counted outcomes across thousands of teams. They share no authors, no method and no dataset.
What they share is a direction. Generation got faster and cheaper. Review did not. Every line an assistant produces still has to be read by someone who understands the system it is going into, and that person is the same person they were last year.
That is the whole mechanism. The constraint was never typing.
What this does not say
It does not say the tools are useless. Both studies measure experienced people working in code they know well, which is the scenario where assistance helps least. Other research consistently finds the largest gains go to less experienced people on narrower, well-defined tasks.
It does not say AI-written code is wrong. The measured claims are about merge rate, review latency, pull request size and issues raised in review. None of them measures the correctness of code that actually shipped.
And it does not say the effect is permanent. Both studies describe tools at a moment in time, and that moment was 2025 into 2026.
What it does say is that self-reported productivity is not evidence. The developers in the trial were not careless or dishonest. They were wrong about their own experience by 39 percentage points, and they had every reason to notice.
What to do with this if you run a team
- Stop surveying people about whether it helpsThe trial is the clearest evidence available that asking produces a confident answer unrelated to what happened. If your only measurement is how the team feels, you do not have a measurement.
- Measure the thing you actually care aboutNot lines written or suggestions accepted. Time from work starting to work merged, and the share of work that merges at all. Both studies point at the same two numbers.
- Look at your review queue before you look at your toolingIf AI-assisted work waits five times longer to be picked up, the constraint is reviewer capacity. Buying more generation makes that worse rather than better.
- Watch the size of what arrivesLarger pull requests are harder to review properly and more likely to be waved through or abandoned. A 408-line change and a 157-line change are not the same object.
- Expect gains where the evidence puts themNewer people, narrower tasks, well-defined work, unfamiliar code. Not senior people doing complex work in systems they built.
- Run it for a fortnight before you decideTwo weeks of merge-rate and cycle-time data on your own team beats any published figure, including these ones.
Why we publish this
We build software and we use these tools daily. We would rather say what the measurements show than repeat a productivity multiplier from a vendor deck.
The same discipline applies to everything else we recommend. Our position on that is at how we use AI, and where the honest answer is that a benchmark cannot support the claim being made from it, we have written about that too in what benchmark scores do not tell you.
If you are being sold a productivity figure for your development team, ask which study it came from and how it was measured. That question is usually enough.
Sources
Every figure in this article was read at the document itself.
Common questions
- Does this mean AI coding tools do not work?
- No. It means the measured effect on experienced developers working in code they know well was negative, and that a smaller share of AI-authored code survives review. Other research consistently finds real gains for less experienced people on narrower, better-defined tasks. The tools work; the question is who benefits and where.
- Why did the developers think they were faster?
- The study does not settle that. What it establishes is the size of the gap: they forecast 24% faster, estimated 20% faster afterwards, and measurement showed 19% slower. Two expert groups predicted around 38% faster. The useful takeaway is that self-report is not evidence, however experienced the person reporting.
- What does a 32.7% merge rate actually mean?
- Of AI-authored pull requests raised across that dataset, roughly one in three was merged. For human-authored pull requests it was closer to five in six. It measures what survived review, not whether the code was correct.
- How big was each study?
- The randomised trial used 16 developers across 246 tasks, which is small but controlled and randomised. The benchmarks report covered 8.1 million pull requests across 4,800 teams in 42 countries, which is large but observational. Their weaknesses are different, which is part of why the agreement between them is interesting.
- What should we measure instead of asking the team?
- Time from work starting to work merged, and the proportion of work that merges at all. Track the size of pull requests and how long they wait for a reviewer. Two weeks of your own data is worth more than any published figure.