These studies are important but nevertheless they don't tell the whole history because the outcome will always depend on the model that is used. If you only test one model, of course the result cannot be generalized to all software development tasks. Ultimately, I'm afraid these studies will only contribute to increase the perception against AI:

#AI #GenerativeAI #CoPilot #LLMs #SoftwareDevelopment #Programming: "Many developers say AI coding assistants make them more productive, but a recent study set forth to measure their output and found no significant gains. Use of GitHub Copilot also introduced 41% more bugs, according to the study from Uplevel, a company providing insights from coding and collaboration data.

The study measured pull request (PR) cycle time, or the time to merge code into a repository, and PR throughput, the number of pull requests merged. It found no significant improvements for developers using Copilot.

Uplevel, using data generated by its customers, compared the output of about 800 developers using GitHub Copilot over a three-month period to their output in a three-month period before adoption.

(...)

There’s a difference between writing a few lines of code and full-fledged software development, Gekht adds. Coding is like writing a sentence, while development is like writing a novel, he suggests.

“Software development is 90% brain function — understanding the requirements, designing the system, and considering limitations and restrictions,” he adds. “Converting all this knowledge and understanding into actual code is a simpler part of the job.”"

https://www.cio.com/article/3540579/devs-gaining-little-if-anything-from-ai-coding-assistants.html