After I published Knowledge Is Cheap Now, Murray MacRae PhD asked me on LinkedIn how you identify an AI hallucination. My answer was quick, and I still think it was right: if you’re an expert in the subject you can see it, and if you’re not, you mostly can’t. His question stayed with me, and in the weeks that followed I ended up on both ends of it, and the uncomfortable end was the one where I didn’t see it at all.
The first time ran the other way round. On 9 September I was running an audit of the files that hold my writing system, 5 phases, one Claude model doing the evaluation and another carrying out the changes, and one finding said the Te Pūkenga restructure that ended my last role had been invented. It beat itself up about it, stating that my redundancy and my early retirement in November 2024 had been fabricated and hallucinated, and should come out. It only backed down when it went back to The Coach Said Ageism is a Myth, where I wrote about that restructure in June, and found the context it had been missing. I was made redundant. I retired early. I was there. Nobody else reading that audit would have known which way round the truth ran; I knew because it’s my life, and there’s no one more expert in it than me.
Consistent, and wrong
That audit checked my writing against what it had in front of it, and what it had in front of it was partial. I don’t write my Strategic Kata pieces in chronological order, some run ahead of when they happened and some reach back years, and the system that helps me edit holds its own selective memory of my career. Against that partial record, a restructure it couldn’t place looked invented. Internally consistent, externally wrong, and delivered with complete confidence.
OpenAI’s researchers published a paper in September 2025 on why language models hallucinate,1 and their explanation is plainer than I expected. The models are trained and scored like students sitting a multiple-choice exam. If a model doesn’t know someone’s birthday, a guess has a 1 in 365 chance of being right, while “I don’t know” scores zero every time, so over thousands of questions the model that guesses looks better on the scoreboard than the one that admits it isn’t sure. So it guesses, and it guesses fluently. The way I read it, hallucination will always be there when the model hasn’t got the right context. I’m seeing less of it in my own work than when I started with AI a couple of years ago, and I’m not saying it’s gone, because September gave me three of them.
Some hallucinations you can see from across the room. When I trained Gemini to draw my comics, it kept giving mouths to characters who don’t have mouths. Anyone could spot that. The ones that cost you are written in well-formed, confident sentences about something you don’t know well enough to argue with, and that’s where Murray’s question bites.
Where I could see it
On my own ground I catch it, and I was catching the human version of it long before AI arrived. At IRD, BNZ and Stats NZ nearly everything ran on Microsoft, and I read the vendor reports, the release notes and the updates because that was part of the job. When a supplier brought in a product person from headquarters, flown in from Europe or Australia or the US to present the new release, they tended to assume the room knew less than they did. Often the room knew more. Some of my team had been running the beta for weeks because they loved to tinker, and they knew what the product actually did, so every now and then what the presenter claimed didn’t match what we had seen with our own hands. The presenter believed every word, and was fluent about a product they hadn’t used the way we had.
At ABN AMRO, when we moved into a new building, the data centre went into office space on the top three floors, and the project manager handed me the specification for the floor where the AS/400 and the other servers would sit. There was no raised floor in it. It had simply been left out, with no budget argument or short cut behind it. A raised floor is where the cooling air moves, where the power cables, data cables and ventilation run safely out of the way, and where the engineers get at all of it for maintenance, even on an office floor. I only saw what was missing because before ABN AMRO I had run a data centre at a local bank, in a different role. The raised floor went in, after some objection and a few rounds of discussion with leadership and finance. The specification had been wrong by omission, the same way AI can be: incomplete, with nothing on the page to tell you something is missing.
A couple of years ago I asked ChatGPT to help revise my CV. It inflated my percentages, it translated my New Zealand job titles into what they would mean in the US, and it turned work that was done locally, and done well, into global impact. I laughed at it, because it’s a very good wordsmith with a strong sense of marketing. I have never sent that CV to anyone. I couldn’t stand behind evidence it had made up.
On my own ground I can even see what I don’t know. In September, while I was drafting Does the Exit Interview Work? about our body corporate changing managers, Claude wrote that the outgoing manager ran 40 other buildings, and presented it as fact. I had no idea how many buildings they ran. It wasn’t my business, and they never told us. That’s how I knew the number was made up: I never had one to give, and nothing in the material said it. It came out before the piece went live.
Where I couldn’t
Now the other half. The same article had a second one in it, and that one I didn’t see. The evidence in it was a 2016 Harvard Business Review study by Everett Spain and Boris Groysberg.2 Claude compressed the study’s method into “188 executives across 210 organisations”, which can’t be right, since you can’t spread 188 people across 210 organisations. They had surveyed 188 executives and interviewed 32 senior leaders, who between them represented 210. The draft also left out where the authors land: exit interviews can work if a second-line manager runs them. I knew that. I had said the same about the concept in my first blog post on exit interviews in 2022, and I part company with them only because in practice I have never seen one work. Claude reported the citation as verified. I read it, and I published it.
The next day a reader who knew the study wrote to me privately and told me it was wrong, twice. The correction sits at the foot of that piece, dated 24 September. I had read and scanned the study, and I never checked whether the numbers on my own page added up. The reader knew the study well enough to see they couldn’t. On that paper, I was the non-expert.
Big firms miss it too. Australia’s Department of Employment and Workplace Relations paid Deloitte AU$440,000 for a 237-page review of its welfare compliance system.3 The report cited academic work that doesn’t exist and a quote from a Federal Court judgment that isn’t in the judgment. Deloitte had used GPT-4o to help draft parts of it. A firm full of professionals put its name on it. Chris Rudge, a University of Sydney researcher in health and welfare law, read it and saw the problem, because he knew the literature it was citing. Deloitte refunded the final instalment of the contract in October 2025.
Closer to home, in February this year the New Zealand Supreme Court dealt with an applicant representing himself in a Family Court matter. His submissions cited cases that looked real: real case names, attached to citations that pointed somewhere else. The court said they appeared to have been hallucinated by an AI application,4 and warned that relying on false citations, including unverified AI output, may in serious cases amount to obstruction of justice or contempt of court. He had no lawyer, and little chance of seeing it, and the warning landed on him anyway.
What a non-expert can do
Back to Murray’s question. My first instinct, when I dictated the draft of this piece, was that a non-expert could paste the paragraph back into AI and ask whether it holds up. I have talked myself out of that. It’s like asking the con artist to tell you he’s honest, and if you can’t judge the original, you can’t judge the verdict either. It will catch some things, a dead link, a case that doesn’t exist at all. It’s much less reliable on a real study with its method bent out of shape, which is exactly the kind that got past me.
What caught all three was a person who knew the original well. My reader knew Spain and Groysberg’s numbers. Chris Rudge knew the work Deloitte claimed to be citing. The Supreme Court had the law reports. So this is what I do now when I’m outside my own ground. I open the source myself, the actual paper or judgment or release note the summary was made from, and check that the sentence carrying the weight says what I’ve been told it says. And when it matters, I borrow an expert, someone who has walked that ground, and ask them to read it before I rely on it.
Niklas Gruhn wrote a blog post in August about the meat proxy,5 the person who forwards whatever the AI produced without reading it, understanding it or checking it. The burden of verification sits with whoever puts their name on the output, and for Strategic Kata that’s me. My process already had checks in it: the Wispr Flow voice draft, the excavation, the back and forth, an audit against what I’ve published before and what’s online. It still let one through. So every citation now gets two rounds before it goes out: I open the link to see that it lands where it says it does, and I search the source for the passage I’m leaning on and read it there. It’s a quick read. It’s also the price of my name on the page.
Murray was right. I just didn’t expect the non-expert to be me.
- Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, Why Language Models Hallucinate, OpenAI, September 2025: https://arxiv.org/abs/2509.04664 ↩︎
- Everett Spain and Boris Groysberg, “Making Exit Interviews Count”, Harvard Business Review, April 2016: https://hbr.org/2016/04/making-exit-interviews-count ↩︎
- “Deloitte to partially refund Australian government for report with apparent AI-generated errors”, Associated Press via Fortune, 7 October 2025: https://fortune.com/2025/10/07/deloitte-ai-australia-government-report-hallucinations-technology-290000-refund ↩︎
- Jones v Family Court at Whangārei [2026] NZSC 1, 11 February 2026, at [7]: https://www.courtsofnz.govt.nz/assets/cases/2026/2026-NZSC-1.pdf ↩︎
- Niklas Gruhn, blog post on the meat proxy, 3 August 2026: https://gruhn.me/blog/2026-08-03/ ↩︎