AI Made Expert Developers 19% Slower. What the AI Productivity Study Really Shows
Why the most important number in AI productivity research is not the speedup, but the gap between feeling and fact.
In early 2025, the research group METR ran a randomized trial with 16 experienced open-source developers. They worked on 246 real tasks in their own projects. Each task was randomly assigned: AI allowed or not.
With AI, the tasks took 19 percent longer. After the study, the developers still estimated that AI had made them 20 percent faster.
That is the AI productivity study most people have heard of. Most people stop at "AI slows you down." That is the wrong lesson.
The conclusion is not stupid. It is just too narrow. Other studies found clear gains. And in February 2026, METR published a follow-up that points in a different direction again.
I want to show you what the METR study and its follow-up actually found. I want to show you why studies with gains and studies with losses are both right. And I want to give you a simple system to measure your own AI productivity, so you stop guessing.
Key takeaways
- In the 2025 METR trial, experienced developers took 19 percent longer with AI, while believing they were 20 percent faster.
- Other trials found real gains: 15 percent more resolved issues in customer support, 25.1 percent faster work on consulting tasks inside the AI frontier.
- The durable finding is the perception gap. Your feeling about AI speed is not data. Measure it by task type.
What the AI productivity research actually shows
The common belief has two versions. Version one: AI makes everyone faster. Version two, since July 2025: AI secretly makes experts slower.
Both are too simple. The research shows something more useful.
The METR trial, read closely
The 16 developers were not beginners. They had an average of five years on their repository. The projects averaged more than 22,000 GitHub stars and over a million lines of code. Tasks took about two hours each. Developers were paid 150 dollars per hour.
When AI was allowed, they mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, the frontier models at the time. Before starting, they forecast that AI would cut their completion time by 24 percent. Experts in economics forecast 39 percent, and machine learning experts 38 percent.
The measured result: 19 percent more time. The confidence interval ran from 2 to 39 percent slower.
The appendix explains where the time went. Developers accepted less than 44 percent of AI generations. About 9 percent of their time with AI went to reviewing and cleaning AI output. 75 percent said they read every line of AI code. They also spent more time waiting and idle.
And here is the detail almost nobody quotes. 69 percent of the developers kept using Cursor after the study ended. The researchers read that as a sign that the tool gave them something real, even if it was not speed.
What is a randomized controlled trial? A randomized controlled trial assigns a treatment by chance instead of by choice. In the METR study, a random draw decided for each task whether AI was allowed. That removes the biggest bias in productivity claims: people tend to use AI on the tasks where they already expect it to help.
The follow-up nobody talks about
METR repeated the experiment from August 2025 with newer tools. It included 57 developers, 143 repositories and more than 800 tasks. Pay dropped to 50 dollars per hour.
The raw numbers changed direction. For the 10 developers from the original study, METR now estimates 18 percent less time with AI. The confidence interval runs from 38 percent less to 9 percent more. For the new developers, the estimate is 4 percent less time.
But METR itself calls this data unreliable. Many developers refused to join because they did not want to work without AI. Between 30 and 50 percent said they held back tasks they did not want to do without AI. One developer completed none of the tasks assigned to the no-AI condition.
In other words, the study broke because AI became too useful to give up. METR believes developers are likely faster in early 2026 than in early 2025. It also says its own data is only very weak evidence for how much.
The studies with clear gains
Two other trials show where AI helps.
In a study of 5,172 customer support agents, Erik Brynjolfsson, Danielle Li and Lindsey Raymond found 15 percent more issues resolved per hour with an AI assistant. The version published in the Quarterly Journal of Economics in 2025 reports that. An earlier 2023 working paper, the one most articles quote, reported 14 percent and 34 percent for novice and low-skilled workers. The gains were largest for less experienced workers. The most skilled agents saw small gains in speed and small declines in quality.
In a preregistered experiment with 758 Boston Consulting Group consultants, researchers from Harvard Business School and partner universities tested GPT-4. On 18 tasks inside what the authors call the jagged frontier, consultants with AI completed 12.2 percent more tasks and finished 25.1 percent faster, with significantly better quality.
On one task chosen to sit outside the frontier, the picture flipped. Consultants with AI were 19 percent less likely to get the right answer. Co-author Ethan Mollick reported that consultants got it right 84 percent of the time without AI, and only 60 to 70 percent with it.
In short: AI is not a speed multiplier you can apply to your whole day. It helps a lot on some tasks, hurts on others, and you cannot tell which by how it feels.
The three mistakes people make when judging AI at work
Once you put these studies side by side, three mistakes stand out.
Mistake 1: Trusting how fast it feels
The developers in the METR trial were experts. They still misjudged the effect by about 40 percentage points, as METR later summarized it. Watching a model write 50 lines in seconds feels fast. The 20 minutes of review, rework and waiting do not feel like part of the task.
If you want to see how strong that bias is outside of AI, the article on the planning fallacy shows the same pattern with deadlines. People trust the story over the record.
Mistake 2: Using one verdict for every task
"AI makes me faster" and "AI slows me down" are both claims about an average. The BCG study shows that the same tool, the same people and similar difficulty can produce opposite results.
Here is an example. A solo founder uses AI for everything: first drafts, data cleanup, pricing analysis and contract questions. The drafts and cleanup really are faster. The pricing analysis comes back confident and wrong, and nobody notices for a month. On average it feels like a win. In the one task that mattered most, it was a loss.
Mistake 3: Counting speed instead of finished value
The customer support study measured resolved issues, not typed words. That is the right unit. In a separate May 2026 survey of 349 technical workers, METR found a median self-reported speed gain of 3x. The self-reported gain in value was lower, between 1.4x and 2x.
The gap has a simple cause. When something becomes cheap, you do more of it, even if it does not matter much. A dashboard nobody asked for is fast output, not productivity.
Does AI make you more productive? The AI Time Ledger
You do not need a lab to answer this for yourself. You need the same three things the good studies had: a baseline, a fair comparison, and honest time logs. Here is a version for one person.
Step 1: Baseline. For two weeks, log every work task with three fields: task type, minutes spent, and whether it is done. Do not change how you work yet. Ten task types are enough: writing, research, analysis, coding, email, planning and so on.
Step 2: Map your frontier. Mark each task type as inside, edge or outside. Inside means the work is common, easy to check and low risk if wrong. Outside means it needs context the model does not have, or errors are hard to spot. This is a guess at first. The next steps replace it with data.
Step 3: Flip. For edge tasks, alternate between AI and no AI on similar tasks. Let a coin decide, not your mood. This is the core of the METR design, and it is the only way to stop picking AI for the tasks that were going to be easy anyway.
Step 4: Log all the time. Count prompting, waiting, reading and fixing as part of the task. A week later, note any rework the task caused. If you switch to email while the model is thinking, that switch counts as well. The article on context switching explains why it costs more than it seems.
Step 5: Review monthly. Compare average minutes and rework per task type, with and without AI. Move task types between zones. Default to AI where your own numbers say it wins. Keep the coin flip running on the edge.
Measuring real output only works if planned work and finished work live in one place. That is the idea behind Gaapp.io, a goal tracking app that connects yearly goals to daily steps and shows what is due today. Disclosure: I am building it. A paper notebook works too. What matters is that the record is written down, not remembered.
Ask yourself this week:
- For which three task types do I actually know, in minutes, whether AI saves me time?
- Where am I using AI on work that is hard to check?
- Am I measuring finished value or just faster output?
Conclusion: measure, do not guess
AI may well make you faster. The one thing the research proves is that you cannot feel it.
The METR result was never a verdict on AI. It was a verdict on self-reporting. The same tools that slowed experts in early 2025 became so useful by 2026 that developers refused to work without them. Customer support agents and consultants saw real gains on the right tasks.
Being pro-AI and being precise about it are not opposites. The operators who get the most out of AI will be the ones who know exactly where it pays off, because they wrote it down.
What you learned today
- In the METR trial, experts took 19 percent longer with AI and still believed they were 20 percent faster.
- The 2026 follow-up points to a speedup, but METR calls its own data unreliable because developers would not work without AI.
- Gains are real inside the frontier and negative outside it. A two-week baseline and a coin flip show you which is which.
FAQ
Does AI make you more productive? It depends on the task. Randomized trials found 15 percent more resolved issues for customer support agents and 25.1 percent faster work on consulting tasks inside the AI frontier. But experienced developers in a 2025 trial took 19 percent longer, and consultants made more mistakes on a task outside the frontier. Measure by task type.
What did the METR AI productivity study find? METR ran a randomized trial with 16 experienced open-source developers on 246 tasks in early 2025. With AI allowed, tasks took 19 percent longer. Developers had forecast a 24 percent speedup and still believed afterwards that AI had made them 20 percent faster. A 2026 follow-up suggests a speedup, but METR calls that data unreliable.
Why were developers slower with AI? The researchers found several likely factors. Developers accepted less than 44 percent of AI generations and spent about 9 percent of their time reviewing and cleaning AI output. They knew their large, mature codebases very well, so the AI added less. They also spent more time waiting on the model and idle.
What is the jagged frontier of AI? The jagged frontier describes how AI ability is uneven across tasks of similar difficulty. In a study of 758 BCG consultants, AI users finished 12.2 percent more tasks and worked 25.1 percent faster on tasks inside the frontier. On a task outside it, they were 19 percent less likely to find the correct answer.
How do you measure your own AI productivity? Log your tasks for two weeks with task type and minutes. Sort task types into inside, edge and outside the AI frontier. On edge tasks, let a coin decide whether you use AI. Count prompting, waiting, review and rework. Compare the averages each month and adjust where you use AI.
About the author
Tristan Eilenburg. One person. AI systems. Global reach. I write about building systems that work, with evidence instead of hype.
- X: @MrB2Beast
- LinkedIn: Tristan Eilenburg
- Instagram: @mrb2beast
- Building Gaapp.io (Instagram: @gaapp.io)
I write one of these every week. If this one taught you something, the newsletter is where the next one lands first.
Sources
- Becker, J., Rush, N., Barnes, E., and Rein, D. (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv:2507.09089. https://arxiv.org/abs/2507.09089
- METR (2025, July 10). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
- Becker, J., Rush, N., Cunningham, T., Rein, D., and Mahamud, K. (2026, February 24). We are Changing our Developer Productivity Experiment Design. METR. https://metr.org/blog/2026-02-24-uplift-update/
- Becker, J. (2026, May 11). Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity. METR. https://metr.org/blog/2026-05-11-ai-usage-survey/
- Brynjolfsson, E., Li, D., and Raymond, L. R. (2025). Generative AI at Work. The Quarterly Journal of Economics, 140(2), 889-942. https://doi.org/10.1093/qje/qjae044 (preprint: https://arxiv.org/abs/2304.11771)
- Brynjolfsson, E., Li, D., and Raymond, L. R. (2023). Generative AI at Work. NBER Working Paper 31161. https://www.nber.org/papers/w31161
- Dell'Acqua, F., McFowland III, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., and Lakhani, K. R. (2026). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality. Organization Science, 37(2), 403-423. https://doi.org/10.1287/orsc.2025.21838 (working paper: https://papers.ssrn.com/abstract=4573321)
- Mollick, E. (2023, September 16). Centaurs and Cyborgs on the Jagged Frontier. One Useful Thing. https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged
