Are you really so sure?

Fabricated quotes, false AI accusations, and fake doctors — a week of people trusting and distrusting AI for all the wrong reasons.

Author

Mignon Fogarty
May 30, 2026

Issue 110

On today’s quest:

— Anthropic releases Claude Opus 4.8
— Are you really so sure?
— AI-generated podcasts claim to be hosted by doctors
— What is WRONG with people?
— How to check for changes in a document
— More people are discovering AI fact checking

Anthropic releases Claude Opus 4.8

A new Claude model just dropped, and people who had early access say it is noticeably better. Simon Willison called it “a modest but tangible improvement,” and Ethan Mollick said he was “impressed with it.” Mollick posted a video it made and a “sophisticated, if minor, academic paper” it ideated and wrote.

The Every blog said it is “legitimately great” — enough so that Anthropic could have called it Opus 5.0 — and that it’s “the best model we’ve tested for writing and knowledge work.”

Anthropic says Opus 4.8 is better at “honesty,” being “more likely to flag uncertainties about its work and less likely to make unsupported claims.”

The new model is only available on paid plans.

Are you really so sure?

I’m becoming increasingly worried about unprovable accusations of people using AI.

A Trinidadian writer named Jamir Nazir won the prestigious Commonwealth Short Story Prize and had his story published online in Granata.

But some people who read the story thought it sounded like it had been written by AI and put it into the Pangram AI detector, which flagged it as 100% written by AI.

HOWEVER, AI detectors are known to work less well on writing that isn’t standard English. When they fail, they often fail on writing from people for whom English isn’t their first language. English is the official language of Trinidad, but Trinidadian English is different from standard American and British English.

Linguist Jack Grieve, who does work on AI systems, wrote on LinkedIn:

“My basic worry is just because a tool performs very well on a test dataset, we shouldn’t assume the results port over to every other text.

In this case, the results being cited to justify the certainty with which the guilt of this author is being proclaimed are grounded in the performance of a tool tested on a single prompting stategy, on a different register (first 1,000 words of novels), and most notably presumably not on Trinidadian English.

The baseline style of human authored Trinadian short stories is not obvious to me, and so I’m surprised it is to an AI system much less to the American computer scientists making these claims.”

Following that debacle, people started putting past winners of the prize through Pangram, and “identified” other stories that were supposedly written by AI. In one case, the author has publicly insisted she did not use it (and I saw a friend vouching for her on social media). Troublingly, the story, which is about enslaved Africans also doesn’t seem to conform to western storytelling norms. In an interview in the Globe and Mail, the author said, “This story is rooted in Vincentian oral storytelling tradition, which by nature is formulaic and relies on repetition, rhythm, and patterned language as core elements of their structure.”

Pangram is the tool of choice these days for making accusations, and someone also used it to suggest that the pope of used AI to write his recent encyclical, Magnifica Humanitas. Pangram said parts of the pope’s commentary on AI were 40% to 100% written by AI, which I seriously doubt. And again, the pope’s language is likely not in line with most of the training data used on these models.

Finally, I have seen multiple stories of writers putting their own human-written work into Pangram to find it flagged as AI, and I have seen it do so with writing that I am also sure is human-written.

People need to be a lot more skeptical about Pangram results — and the results of even worse tools. The sad thing is that Pangram actually is the best tool when tested head-to-head with others. And the ironic thing is that people who are anti-AI are using an AI tool in such a credulous and harmful way.

Dealing with AI writing is a rapidly growing problem for anyone who works with writers, and it’s about more than just detection. If you read one other thing this week, I highly recommend Jane Friedman’s thoughtful piece on policing AI writing in publishing: Writing & Publishing Awards Have Difficult Decisions to Make Regarding AI.

AI-generated podcasts claim to be hosted by doctors

The Podcast Business Journal reports that Inception Point, the company that releases about 10,000 AI-generated podcast episodes each week, has been releasing content with voices that claim to be doctors. At The Podcast Show conference, CEO Jeanine Wright said they will stop the practice but will still have hosts that identify themselves as experts or “having been well read and well studied in a space.” When questioned about whether they would remove the existing shows that pretend to be hosted by doctors, Wright said no.

I’m no expert, but I did find a law firm website that said it is illegal to “impersonate a doctor by advertising or holding oneself out as a doctor or other medical professional without the proper licensure” in every state in the U.S.

(And as an aside, I also learned about “the Dr. Oz paradox”: because of the First Amendment, doctors can give potentially harmful health advice to the general public that would subject them to malpractice liability if they gave it to an actual patient.)

What is WRONG with people?

Another day, another person being stupid with AI. Maybe I shouldn’t be surprised anymore, but the New York Times reports that a buzzy new book about truth in the era of AI includes quotations fabricated by AI.

The author, Steven Rosenbaum, says he accepts responsibility for the errors and is looking into them. And yes. He should accept responsibility. But he should have accepted responsibility — for checking the facts — before he submitted his manuscript.

I’ll never understand what it is about AI that seems to compel people to think they don’t have to read the actual quotations or citations they put in their work.

It’s fine to use AI as a search engine. It’s not fine to not check the original sources because we know that some percent of the time, AI makes things up.

I think the thing that gets me most, though, is the apparent lack of shame. In an astonishing attempt to put a good spin on the situation, his final comment in the article is if the episode “serves as a warning about the risks of A.I.-assisted research and verification, then that is why I wrote the book.”

Mission accomplished, I guess.

UPDATE! I read a follow-up interview with Rosenbaum that made me even more annoyed. The man who put himself forward as an expert on trust in the age of AI told Ars Technica, “he ‘learned a lesson’ and is ‘going to be much more suspicious’ and ‘reticent to trust’ AI outputs going forward.”

How to check for changes in a document

I feel like I owe you a useful tip if you’ve read this far through my rant, so if you ever ask AI to make changes to a document, you can use a simple free tool called DiffChecker to make sure it hasn’t made changes you didn’t want.

I used it just last week to make sure a contract I was signing hadn’t been changed since the last version I read.

More people are discovering AI fact checking

I’ve mentioned AI for fact-checking multiple times in this newsletter, and this week, I saw Wharton professor Ethan Mollick recommend it too. (I find the same thing he mentions below — ChatGPT can be nitpicky about small details that don’t matter.)

  • They used the free models in their tests (something I’ve repeatedly told you to avoid).

  • They highlighted the slop that comes from Google’s AI Overviews (again, something we should all know is terrible by now).

  • In some cases, they cited accuracy statistics from 2024 and 2025 (anyone paying attention knows the models are dramatically better now).

  • They confused hallucination rates on generated output with accuracy in fact-checking.

Again, very disappointing. I’ve been a Wired subscriber since almost when it launched, and I would have expected better.

So again … don’t believe anything you get from an LLM. Especially don’t believe anything you get from a free LLM or Google’s AI Overview. Fact-check everything, and if you choose to use AI, it can be an excellent fact-checker.

To fact-check with AI, enter your text into one of the best paid models with thinking, and say, “Fact check this piece. Show your sources.” Then check every source. It won’t be perfect, but it’s faster and more thorough than entering every fact one by one into a search engine, which also isn’t perfect.

I also sometimes tell it to use a green check emoji for things that are correct, a yellow caution emoji for things it’s not sure about, and a red police car light emoji for things that are clearly wrong. That helps me instantly see the scope of what I need to check.

Whether you do Google searches or have an LLM give you a list of questionable points with its sources and links, you still have to use your own brain and judgment, and of course, neither system will be able to fact-check things like comments a writer got directly from a source and information behind paywalls (e.g., AP and Chicago style). Google can’t find things that don’t exist on the open web and neither can an LLM.

I think the reason this makes me so angry is that there are plenty of true things that are bad about AI. It has helped children commit suicide, it’s short-circuiting our education system, and it helps stupid and evil people be stupid and evil at scale, just to name a few. But if you don’t tell the truth about what it can do, you are misinforming your readers, leaving them ill-prepared to deal with the challenges they are going to face. And when they eventually realize you’ve been giving them strawman arguments, you’ll have less credibility to make accurate arguments.

Quick Hits

My favorite recent pieces

AI is increasing the need for human oversight, not removing it. In this report from two professional nonfiction writing conferences, PerfectIt said, “The conversation is no longer really about whether AI will be used. It’s about how organizations maintain control, quality, and consistency as AI becomes embedded into increasingly complex workflows.” — PerfectIt

Choosing to stay human — Ethan Mollick

Using AI

How to Cut AI Costs and Climate Impact: Ranking 9 Inputs and Outputs [Level: Beginner. Super practical and useful.] — Orbit Media Studios

18 Ways To Save AI Token Budgets [Level: Intermediate to advanced. An expanded version of the post above.] — Almost Timely

Claude Prompting Best Practices [Level: Intermediate] — Anthropic

How to Use Claude to Turn Your Handwriting Into a Custom Font or just make any kind of custom font you draw [Level: Beginner] — Inc.

Agents

Audio

Bad stuff

ChatGPT and other [Free] AI bots made huge errors before Scottish election, study finds [As I’ve said before, the free products hallucinate a lot.] — The Guardian

The number of fabricated references in scientific papers increased more than 12x from Jan. 2023 to Feb. 2026. “Fabricated references are embedded in the peer-reviewed literature at scale, and that the rate of fabrication is accelerating.” — The Lancet

The business of AI

Education

Government

State Department Unveils AI-Driven Visa-Scheduling Tool During Secretary Rubio’s India Trip [The platform claims to assign “earlier interview slots to travelers whose trips the algorithm deems most likely to strengthen U.S. economic or strategic interests.] — VisaHQ

White House Approves $9 Billion for Spy Agencies to Catch Up on A.I. [“The C.I.A. and N.S.A. cannot fully deploy the latest models on their classified systems because of a shortage of cutting-edge chips.”] — New York Times

I’m laughing

You can no longer Google the word 'disregard' — TechCrunch (plus, more examples on Bluesky, and more) [I believe this has been mostly fixed now, but it was still funny.]

Images

Job market

Meta cut 8,000 workers as Mark Zuckerberg warns ‘success isn’t a given’ in the AI race [About 7,000 employees were also moved to AI roles.] — CNBC

Two seemingly contradictory stories out of Uber. Both are based on a comments from talks, so I wonder about the bigger context:

Model & Product updates

Gemini app rolling out ‘Extended’ thinking level [and new third-party integrations including Spotify, WhatsApp, and GitHub] — 9to5 Google

At big launch event, Google announces a new model (Gemini 3.5 Flash), new video generation capabilities, a Daily Brief feature, an agent called Gemini Spark, a new agent app called Antigravity 2.0, and plans for a future MacOS app. [One of these agents works 24/7 in the cloud, which seems like a big advantage because you can use it from anywhere and don’t need to keep your computer on and awake.] — Google

Philosophy

Publishing

Robotics

LimX Luna demo [impressively fluid and complex motion] — YouTube

Science & Medicine

Security

Video

Other

The social contract of writing [that the writer will have spent more time on a piece than the reader] — jola.dev

8 AI bots now write 50% of X’s Community Notes. [AI notes are rated as more helpful than human-written notes, but because they usually use Grok as the LLM, viewpoints are more centralized and susceptible to Grok’s biases.] — Indicator

What is AI Sidequest?

Are you interested in the intersection of AI with language, writing, and culture? With maybe a little consumer business thrown in? Then you’re in the right place!

I’m Mignon Fogarty: I’ve been writing about language for almost 20 years and was the chair of media entrepreneurship in the School of Journalism at the University of Nevada, Reno. I became interested in AI back in 2022 when articles about large language models started flooding my Google alerts. AI Sidequest is where I write about stories I find interesting. I hope you find them interesting too.

If you loved the newsletter, share your favorite part on social media and tag me so I can engage! [LinkedIn — Facebook — Mastodon]

Written by a human