AI-powered coding? Study finds it may take more time than it saves

A new study published by METR, a non-profit AI research organisation, has found that using AI-powered coding tools may actually slow down experienced developers despite expectations that such tools would speed up software development.

METR conducted a randomised controlled trial involving 16 experienced open-source developers who completed 246 real tasks drawn from large codebases they regularly contribute to. Each developer was randomly assigned tasks either with or without permission to use AI coding tools. 

According to the study, before beginning the tasks, developers estimated that AI would reduce their completion time by 24%. However, research shows that allowing AI actually increases completion time by 19% which means developers are slower when using AI tools.

While nearly all participants had prior experience using web-based AI tools in coding, only 56% had used Cursor, the main tool tested in the study. METR stated that they did provide training to ensure developers were familiar with the tool's features before the trial.

The researchers suggest several reasons why AI tools might have slowed down performance. Developers often spent more time formulating prompts and waiting for AI responses than writing code themselves. The tools also struggled to navigate the complexities of the large codebases used in the study.

Despite the findings, the researchers cautioned against drawing broad conclusions. They do not claim that AI coding tools fail to improve productivity across the board. In fact, other large-scale studies have shown efficiency gains in various software engineering contexts. METR also acknowledged the rapid pace of progress in AI, noting that the results might differ if the study were conducted again just a few months later.

Read more

Vibe coding: what is it?