I don't think the study is flawed. It just seems rather narrow:

"We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience."

So the question is what other kinds of software development tasks this result applies to. Moderate AI experience is fine. This applies to many other situations. But 5 years of experience with a single code base is an outlier.

That said, they used relatively large repositories (1.1 million LOC) and the tasks were randomly assigned. So developers couldn't pick and choose tasks in areas of the codebase they already knew extremely well.

I think the study does generalise to some degree, but I've seen conclusions drawn from this study that the methodology doesn't support. In my view, it doesn't generalise over all or even most software development taks.

Personally, I'm a bit sceptical (but not hostile) about LLMs for coding (and some other thinking tasks), because the difference in quality between requests for which there are many examples and tasks for which there are only few examples is so extreme.

Reasoning capabilities of LLMs still seem minimal.

My argument is limited to “we don’t know and one study with significant limitations with regards to participant adaptation doesn’t settle anything definitively for the long-term”.

Your argument seems to project significantly more certainty and spittle.

The LLM crowd always sees themselves as messianic and victims, eerly reminding me of the NFT crowd back 1 year ago. I would not be surprised if a lot of those are the same folks.

The burden of proof is on the ones saying a new concept/tool (LLMs/NFT) is revolutionary or useful. I provided studies showing not only the new concept is not revolutionary, but that it is a step back in terms of productivity. Where are the studies and evidence proving that LLMs are a revolution?

NFT boosters tried for years to make us believe something that wasn't there. I will take the LLM crowd more seriously when I actually see the impact and usefulness of LLMs. For now, it's simply not there.

https://fortune.com/2025/08/18/mit-report-95-percent-generat...

> Your argument seems to project significantly more certainty and spittle.

I am not surprised that a bunch of folks outsourcing their critical thinking to a fancy autocomplete don't have any arguments nor studies though, to refute a pretty simple argument with some receipts behind it. Spittle? Please, at least there is an argument and links.

From the LLM cult crowd there is usually nothing, just crickets. Show me the studies, show me the links, show me the proof that LLMs are the revolution you so desperately want it to be.

Until then, I got the receipts that, if anything, LLMs are just another tool but hardly a revolution worth paying attention to.

“I submitted studies”

You submitted one study and claimed it’s the only in existence (it’s not)

“I got the receipts”

You have one receipt that you misrepresent by saying it scientifically settles things the paper itself points out that it explicitly does not claim

> is the same as good ol' scrum: a bunch of fanatic gaslighters.

Oh no, please no. I can't take it one more time. Is it just me or are devs the absolute worst profession in the regards of self-inflicted dogmas?