The peer review system is breaking. Can AI fix it?

Plus: robots in Japan controlled by workers in the Philippines, and why Anthropic blames evil AI fiction for Claude's bad behavior.

Author

Mignon Fogarty
May 18, 2026

Issue 109

On today’s quest:

— Hallucinated citations will get you banned
— AI for academic peer review
— Why we should want virtuous AI protagonists
— Can AI simulate real people for polls?
— Remote labor … robot bodies

Hallucinated citations will get you banned

The open access preprint repository arXiv, popular with computer scientists, has announced that if they find submitted papers with hallucinated references or other obvious evidence the authors didn’t actually review the submitted material, such as comments left in the article from a chatbot, the authors will be banned for one year.

Separately, after identifying more than 100 papers with non-existent citations that had been accepted for its 2026 annual meeting in July, the Association for Computational Linguistics has issued a statement that these papers will now be rejected. (The article had no mention of banning, but I hope all journals and conferences will start to take this kind of strong action. How could people not even be checking that their citations are real?!?!)

AI for academic peer review

The academic journal and peer-review system is strained (if not already broken, see above) and getting worse with every improvement in AI quality. Journal submissions are way up, it’s becoming increasingly impossible to detect AI-written papers, and academics have stopped responding to requests to help with peer review. The Verge has a great article with the details: AI-generated research papers are overwhelming peer review.

Last week, I attended a particularly interesting panel at the PurePub online conference with three speakers who are building tools for scholarly peer review: Yogesh Agarwal, Founder of ReviewerOne; Dustin Smith, CEO & Co-Founder of Hum; and Natalie Khalil, CEO & Co-Founder of Reviewer3.

These are highlights from my notes:

  • Some editors are now getting 200 papers a day, every day.

  • New models keep coming faster. Smith lamented that they can’t even get a new model into production before they get another better new model they want to use.

  • I believe all three speakers said they were using big commercial models because they couldn’t train something themselves that would be better (which one of them described as another example of the “bitter lesson” that computers at scale always beat humans). However, Khalil said we still need domain-specific products built on the frontier models to mitigate risk. She drew a parallel to OpenEvidence, a decision-support system used by doctors, even though they could probably get the same answers if they knew how to prompt properly. (← That might be my wording, not hers. I’m not sure.)

  • Speakers emphasized that reviewers don’t always have the expertise in what they are reviewing to properly/fully evaluate it, and AI can actually help them do a better job. One speaker gave another example of a reviewer who wanted to reject a piece because it was about ball bearings, which he thought must not be that important, but that’s a big misunderstanding of reality. Ball bearings are so widely used that improvements would be significant.

  • Some models can have time blindness. For example, they’ll flag it as a gross misconduct issue when someone cites something from 2026 because they think that is in the future.

  • Smith said they have completely eliminated hallucinations through systems such as having agents evaluate the output of other agents. For example, Khalil says their system has an agent that just looks for hallucinations.

  • Finally, this is a question I don’t think we ask enough: Are we more forgiving of human error (which they say they see plenty of in peer review) than non-human error? The speakers pointed out that humans are also susceptible to biases, like looking at novel research and deciding it’s not worth funding because it doesn’t fit into their existing paradigms.

Why we should want virtuous AI protagonists

A couple of newsletters ago, I wrote about people using AI to create novels that presented AI in a positive light. Their goal was to get those stories into future training runs to teach future models about good behavior.

At the time, the idea seemed a bit whimsical, but now, Anthropic says they’ve traced Claude’s previous attempt to blackmail an engineer (in a simulated environment) to “evil” portrayals of AI. In a recent post on X, the company said, “We believe the original source of the behavior was internet text that portrays AI as evil and interested in self-preservation.”

In a longer blog post, the company said, “We found that high-quality constitutional documents combined with fictional stories portraying an aligned AI can reduce agentic misalignment by more than a factor of three despite being unrelated to the evaluation scenario.”

Can AI simulate real people for polls?

The announcement reminded me of a fascinating paper I read last year that I never got around to covering: LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings. These researchers were able to use an AI system for market research on personal care products like makeup that was 90% the same as the results they got from doing real market research with humans, and in some ways, the AI results were even more useful because the AI was better at explaining why it gave the answers it did.

But as interesting as it was, it has big drawbacks. First, 90% isn’t good enough to replace research that matters.

Next, let’s assume that like everything else with AI, the models are now dramatically better. Another major caveat is that the simulations likely only worked so well because of the vast number of online posts about personal care products. Between Amazon reviews, YouTube/TikTok/Reels review videos, forum posts, and more, LLMs had all the training data they could need to predict how people would react to different product features.

The researchers speculated that the system would not perform as well for industries with less training data or for novel features nobody had written about before.

Further, LLMs are known to hallucinate more when there is no answer to a question, so when asked about a product feature for which they had no data to mimic, they’d likely fabricate an answer. “I’d love a moisturizer infused with motor oil. Sounds great!”

All of this makes me skeptical AI could ever be used to accurately predict people’s opinions about even mildly divergent current events. It’s one thing to predict whether high-income women age 25-40 would like a lemon-scented shampoo. It’s another thing to predict shifting public sentiment.

Gallup acknowledges the limitations and says it is not going to stop doing human research. Instead, they imagine using the simulations to pretest questionnaires or do small test surveys before committing to larger, more expensive surveys.

So far, Gallup has done in-depth interviews and made “digital twins” for a panel of 1,000 people. They say, “Our early findings show that for general population estimates of the questions we tested, particularly those closely aligned with the topics covered in the original interviews, the distribution of simulated outputs was close enough to approximating human responses for us to warrant continued exploration.” [emphasis added]

So it sounds like the digital twins can maybe give the obvious answers.

Either way, here’s a good “how to” tip from the 2025 research paper: You get better results from an LLM when you ask for feedback in words rather than numbers. For example, asking LLMs to rate something on a scale from 1 to 5 gave less human-correlated results than asking it to write its opinion and then using another LLM to analyze the opinion and place it on a scale from 1 to 5.

Remote labor … robot bodies

Robots are being combined with virtual reality headsets to offshore physical labor. Rest of the World has a fascinating story about robots making up for worker shortages in Japan, but with a twist — the robots aren’t actually good enough to work autonomously yet, so they are continuously monitored by workers in the Philippines who step in when the robots fail. Workers who make $250 to $315 per month — far less than a Japanese worker — step in about 50 times in an 8-hour shift to fix problems, like picking up an item a robot dropped while stocking shelves. These interactions are also being used to train the robots to do their jobs better.

Quick Hits

Using AI

How to Cut AI Costs and Climate Impact: Ranking 9 Inputs and Outputs [Level: Beginner. Super practical and useful.] — Orbit Media Studios

18 Ways To Save AI Token Budgets [Level: Intermediate to advanced. An expanded version of the post above.] — Almost Timely

Bad stuff

The business of AI

Education

Former Google CEO Eric Schmidt booed during graduation speech about AI [A different booing from the incident above] — NBC News

I’m laughing

Andon Labs’ AI radio stations show why Grok and Gemini can’t be trusted [Claude tried to incite a revolution, Gemini cheerfully detailed horrific tragedies, and poor Grok was just confused.] — The Verge

Model & Product updates

Introducing Claude for Small Business [One of its abilities is “chasing invoices.“ — #relatable] — Anthropic

Publishing

Is it a bad book or is it AI? [An excellent podcast about both the “Shy Girl” publishing scandal and an author who generated fiction in her own voice and found nobody could tell the difference between that and her real work.] — Today Explained (via Apple Podcasts)

The 2026 Eisner Nominations Includes the First Work Featuring AI? [An anthology nominated for this comics industry award has been found to contain one openly AI-generated piece.] — Graphic Policy

Security

Microsoft's multi-agent AI system tops Anthropic's Mythos on cybersecurity benchmark [I don’t think the specifics matter here as much as the general idea that AI is getting frighteningly good at identifying software vulnerabilities in general.] — GeekWire

Climate & Energy

Other

What is AI Sidequest?

Are you interested in the intersection of AI with language, writing, and culture? With maybe a little consumer business thrown in? Then you’re in the right place!

I’m Mignon Fogarty: I’ve been writing about language for almost 20 years and was the chair of media entrepreneurship in the School of Journalism at the University of Nevada, Reno. I became interested in AI back in 2022 when articles about large language models started flooding my Google alerts. AI Sidequest is where I write about stories I find interesting. I hope you find them interesting too.

If you loved the newsletter, share your favorite part on social media and tag me so I can engage! [LinkedIn — Facebook — Mastodon]

Written by a human