Developers Think AI Makes Them 24% Faster. The Data Says 19% Slower.

AI coding assistants promise dramatic productivity gains. Developers believe them. A rigorous study from METR tested these claims with experienced open-source developers. The results contradict everything the industry believes about AI productivity.

The Study That Shocked Researchers

METR researchers expected to find positive results. "We initially were broadly expecting to see positive speedup," the authors stated explicitly. Instead, they discovered something troubling.

The study tracked 16 experienced developers from large open-source repositories—projects averaging 22,000+ GitHub stars and over 1 million lines of code. These developers worked on 246 real issues from their own repositories. Half worked with AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet), half without.

Key point: When allowed to use AI, developers took 19% longer to complete their tasks.

The 40-Point Perception Gap

Before using AI tools, developers predicted a 24% speed increase. After working with AI and experiencing the actual slowdown, they still believed AI had made them 20% faster.

The math is stark:

• Expected improvement: +24%
• Actual performance: -19%
• Post-experience belief: +20%
• Total perception gap: ~40 percentage points

Common mistake: Relying on developer self-reports for productivity metrics. As the METR study noted, "virtually every corporate AI productivity claim relies on self-reporting" that may not reflect actual productivity.

Why Developers Cannot Tell

The most unsettling finding isn't the slowdown itself. According to the researchers: "The deepest finding is not that AI makes developers slower. It is that developers cannot tell."

Several factors create this cognitive blindness:

New collaboration overhead. Developers spend time crafting prompts, interpreting suggestions, and validating AI output. This feels like productive work—it isn't.

Legacy system friction. Integrating AI-generated code into existing million-line codebases requires extensive adaptation. The study focused on established projects, not greenfield development.

Perception of activity. Constant interaction with AI tools creates a sense of momentum. Developers feel busy and productive even while accomplishing less.

In plain terms: The collaboration itself becomes the work, not the output.

What This Means for Teams

These findings challenge fundamental assumptions about AI adoption. If experienced developers working on familiar codebases cannot accurately assess their own productivity, what does that mean for:

• Performance reviews based on self-assessment
• Team velocity estimates
• ROI calculations for AI tool investments

Try it: Track actual completion times for similar tasks with and without AI tools. Ignore feelings—measure results.

The disconnect between perception and reality extends beyond individual developers. Teams reporting dramatic AI-driven improvements might be experiencing the same cognitive bias at scale.

The Context That Matters

This study specifically examined experienced developers working on large, established codebases. Other research shows different results for different contexts—some studies report 26-56% improvements for specific task types.

The difference matters. Greenfield projects, simple scripts, or well-defined tasks might benefit from AI assistance. Complex debugging in million-line codebases apparently doesn't.

Tested in production: The researchers used frontier models (Claude 3.5/3.7 Sonnet) with Cursor Pro—the best tools available in early 2025. Tool quality wasn't the issue.

Frequently Asked Questions

Why do developers still believe AI made them faster even after experiencing the slowdown firsthand?

The collaboration with AI tools creates a sense of productive activity. Developers focus on the interaction—crafting prompts, reviewing suggestions—rather than measuring actual task completion time. This cognitive bias persists because the work feels engaging and purposeful.

What specific types of tasks does AI slow down the most for experienced developers on large codebases?

The study examined real issues from established open-source projects with over 1 million lines of code. Complex debugging, system integration, and modifications requiring deep contextual understanding showed the most significant slowdowns compared to simpler, isolated coding tasks.

Why did developers who were familiar with established open-source repositories experience slowdowns while other studies show 26-56% speedups?

Context determines outcomes. The METR study focused on experienced developers working within complex, existing codebases they knew well. Other studies showing speedups often involve greenfield projects, simple scripts, or well-defined tasks without legacy constraints.

What to Try Right Now

Measure your actual task completion times for one week—log start and end times for every coding task, with and without AI assistance. Compare the data, not the feelings.