Human summaries beat AI summaries in a new study

Plus, Business Insider says writers can use AI first drafts without disclosure

Author

Mignon Fogarty
September 18, 2025

Issue 85

On today’s quest:
— Plain-language summaries by humans are better than those from AI
— Business Insider says AI first drafts are OK
— 41% of Gen Z workers are sabotaging their company’s AI initiatives
— ChatGPT-5 ❤️s first person
— Chatbots are better than multiple choice surveys

Sponsored
MarketingAlecSkip the AI hype, get real results. Join 12,000+ marketers learning the AI tools and prompts that drove 40% better performance. Power prompts Mon, Fresh tips Wed, Deep dives Fri.

Plain-language summaries by humans are better than those made by AI

Researchers tested how well LLM plain-language summaries of scientific papers conveyed information compared to human-written summaries and found that although readers rated the AI summaries as equivalent to the human summaries, and the AI summaries actually had better readability scores, people who read the AI summaries scored lower on tests about the material.

In other words, the AI summaries looked great by all measures, but fell flat at the actual job of educating readers. This study suggests we need to be careful about how we measure the quality and effectiveness of AI output. — Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation (arXiv)

Business Insider says AI first drafts are OK

In a leaked FAQ sent to staffers, Business Insider told its employees that it is fine to use AI to write the first draft of their stories and that they don’t need to disclose this use to readers.

This is the first such policy at a large online publication I’m aware of, and most comments I saw on the posts were negative, saying the move would undermine trust.

The FAQ did say writers need to make sure the final work “is yours” and interestingly also seemed to handwave to the value of human writing, saying, “Writing is a valuable critical thinking process, and strong writing makes our journalism better and more distinct.” — Status (Oliver Darcy on Bluesky) (A screenshot of more of the article.)

41% of Gen Z workers are sabotaging their company’s AI initiatives

A survey back in March from an AI company called Writer (which doesn’t seem to have anything to do with writing) found that 41% of Gen Z workers admitted to actively working against their company’s AI initiatives. The number was 31% for all age groups.

This feels like a shocking number to me, and I can’t find a lot about the survey, so I’m skeptical of its quality, but it wouldn’t surprise me if this was a problem at some level in corporations.

Kevin Chung, Chief Strategy Officer at Writer, said employees may be worried about AI being biased or unethical or may fear that AI will “diminish their value, creativity or job security.”

He also noted that 35% of employees said they pay for their own AI tools, which suggests they aren’t satisfied with the tools being supplied by their employers (something that I’ve seen in other reports). — Forbes

ChatGPT-5 ❤️s first person

Lech Mazur runs various writing tests on different LLMs and recently looked at how the different models write when given a flash fiction assignment.

The screenshot above shows that ChatGPT-5 (with medium reasoning) dramatically preferred a first-person point of view (“I need to go to the store”) compared to all the other models, which leaned heavily toward a limited third-person POV (“Mignon thought about going to the store”).

Llama 4 Maverick was an outlier on parameters called “narrative modality” and “motif strategy.”

Clearly, all models are not equivalent. Big picture, if you’re not getting the results you want for any kind of project, you might want to try a different LLM.

Chatbots are better than multiple choice surveys

In an ongoing study of universal basic income, researchers checked in with participants using online surveys, in-depth phone interviews, or chatbot interactions.

According to Semafor, “Those who opted to talk to the bot spent a median time of 16 minutes each on the survey, often offering more comprehensive answers that would not have been possible with a multiple-choice format. More importantly, the bot could probe respondents’ answers and ask them to clarify or expand, giving OpenResearch a much richer set of data.”

The story did not say how the chatbots compared to the human phone interviews but did note that for general polling, where you can’t as easily reach people as you can in a research study, online surveys have exploded given how hard it is to reach potential respondents by phone. Therefore, giving people the option of talking with a chatbot online could, theoretically, improve the accuracy of results.

The study was conducted by OpenResearch, a nonprofit research arm of OpenAI.

Quick Hits

Using AI

How people are using AI

Philosophy

I’m laughing

Job market

Model updates

Education

Music

Government

AI slop

AI and coders

Other

What is AI Sidequest?

Are you interested in the intersection of AI with language, writing, and culture? With maybe a little consumer business thrown in? Then you’re in the right place!

I’m Mignon Fogarty: I’ve been writing about language for almost 20 years and was the chair of media entrepreneurship in the School of Journalism at the University of Nevada, Reno. I became interested in AI back in 2022 when articles about large language models started flooding my Google alerts. AI Sidequest is where I write about stories I find interesting. I hope you find them interesting too.

If you loved the newsletter, share your favorite part on social media and tag me so I can engage! [LinkedIn — Facebook — Mastodon]

Written by a human