Right now, AI is making science publishing slower, worse, and more expensive

Not Faster, Better, Cheaper

I’m out with a column today about the challenges that scholarly publishing is having coping with the effects of artificial intelligence on the output of scientific research. Large language models (LLMs) are absolutely opening new frontiers in the research itself, particularly in materials science and protein structure prediction, with many more fields to come. And AI tools are making it much easier to search the literature and prepare manuscripts, which has the potential to allow more people to participate in the scientific literature. But in the short term, at least, the errors generated by AI and the speed with which papers are generated is taxing the system and requiring more human effort, not less so instead of faster, better, cheaper, the industry is slower, worse, and more expensive.

This is a bleak view, I admit, and it could improve. But here I lay out a little more of the evidence for why I’m pessimistic, at least at the moment.

Errors are inevitable and likely here for the duration

In a recent column, Zeynep Tufecki makes the case that the errors produced by LLMs are unlikely to be completely eliminated based on the way the models predict the next word rather than actually reasoning. Tufecki is mainly addressing the idea that AI is going to immediately replace human jobs by making the case that as long as LLMs continue to make errors, humans will be needed to check them. And these errors are here to stay:

Large language models are not reasoning machines. They’re plausibility engines. It’s not just that they don’t test their outputs to make sure they’re correct or logical, or that they fail to do so in certain instances. They can’t, and they’ll never be able to on their own. They can only assess which answers are probable, based on the data on which the models have been trained. And that holds true whether they’re trained on the full breadth of human output or only on peer-reviewed scientific articles. It’s baked into the way they operate.

For the condensed version: just see the bottom of the ChatGPT web interface:

Tufecki also argues that new advances in LLMs will not eliminate these errors, because it’s impossible to imagine every constraint that needs to be placed on the paths it can go down and because its need to keep users engaged by flattering us — detailed here in a recent Science paper — also encourages wrong answers.

Plenty of other researchers are no doubt more optimistic, time will tell. But at the moment, there’s no questions that the errors abound.

AI is definitely boosting volume

Science and our journals can anecdotally point out that our submissions are way up and that they have lots of AI errors. For a more scholarly take, a recent study by James Evans and co-workers shows that researchers who use AI publish more papers than those who don’t and get more citations. However, Evans observes that these well circulated papers don’t necessarily drive science in new directions:

On average, AI adoption helps individual scientists publish 3.02 times more papers, receive 4.84 times more citations and become team leaders 1.37 years sooner. This probably results from improved modelling and prediction of field-specific data, resulting in higher performance on recognized benchmarks. The substantial academic benefits of AI use may be a driving force behind its accelerated rate of adoption; however, we also find unintended consequences from the increased prevalence of AI-augmented research. In all fields, AI-augmented research focuses on a narrower scope of scientific topics and reduces the scientific engagement of follow-on research, leading to more overlapping research work that slows the expansion of knowledge. Further, with a greater concentration of collective attention to the same AI papers, the adoption of AI seems to induce authors to converge on the same solutions to known problems rather than create new ones.

One critic of Evans’ work on this pointed out that he used AI to determine that the expansion of knowledge will be slowed, but we’ll eventually know whether they got the right answer.

And while AI will help you write up these findings, there is at least some evidence that the agents that do this will be more likely to produce findings that might not actually be in the data through things like p-hacking.

For now, we’re struggling

Maybe all of this will be improved. But for now, we know that we’re getting more papers with more errors that require more human effort to get them prepped to be in the literature accurately. This is occurring while the tech bros are all over YouTube telling us that we’re going to live in radical abundance with lots of free time on our hands.

If only.

Still, ChatGPT did make this awesome graphic for me for this article:

Be careful out there.