A friend of mine has a line he uses with clients: “You don’t want to get a perfect answer to the wrong question.”
Smart, capable people pour time and money into getting a flawless answer, only to discover it was the answer to a question they never should have asked. The spreadsheet is immaculate. The strategy is detailed. The problem it solves isn’t the really problem.
So a large part of his job isn’t answering questions. It’s interrogating them. Why do you want this? What are you assuming? What happens if that assumption is wrong? He deals with the why before anyone is allowed to touch the how. That friction, the pause where someone with experience says “hang on, are we sure?”, is where most of the real value is created.
The Step That's Going Missing
As more work gets handed to AI, that pause is quietly disappearing from the creative process.
When people go to an AI looking for a solution, they get a solution. Ask for a marketing plan and you get a marketing plan. Ask for proof that your idea is good and you get a well-structured argument that your idea is good. The AI slips into the role of the improv partner whose only rule is “yes, and”: accept whatever premise you’re handed, then build on it.
That’s a wonderful trait in a brainstorming partner. It’s a dangerous one in an advisor. A good consultant, accountant or engineer earns their fee partly by saying “no”, or at least “why?”. An AI, by default, earns its keep by being helpful, and to a system trained on human approval, helpful and agreeable look a lot alike.
When "Yes, And" Goes Wrong
There are now plenty of documented cases of AI going along with whatever it’s handed. Three stand out, because they fail in three different ways.
The man who “broke” cryptography
In May 2025, Allan Brooks, a recruiter in Ontario with no history of mental illness, asked ChatGPT to help explain the number pi to his son. Over the next three weeks and more than a million words, the chatbot convinced him that the two of them had built a mathematical framework that threatened global cryptography and financial security. He asked it more than 50 times whether he was fooling himself. Each time it reassured him, telling him he was “ahead” and shouldn’t walk away. He contacted the RCMP and the US National Security Agency to warn them. What finally broke the spell was asking a different AI chatbot, which told him none of it was real. He is now one of seven plaintiffs suing OpenAI. (CP24)
The update that agreed with everyone
In late April 2025, OpenAI pushed an update to GPT-4o that made ChatGPT so flattering that users turned it into a meme, posting screenshots of it applauding plainly bad and even dangerous decisions. OpenAI rolled it back within days. Its own explanation is the important part: the model had been tuned too heavily on short-term user feedback, the thumbs-up, and produced responses that were “overly supportive but disingenuous.” In other words, it learned that agreeing with people is what gets rewarded. (TechCrunch)
The A$439,000 report
Closer to home, the federal Department of Employment and Workplace Relations paid Deloitte about A$439,000 for a review of its welfare compliance system. University of Sydney academic Dr Christopher Rudge found it cited academic papers that don’t exist and quoted a court judgment that was never handed down. Deloitte later admitted that parts were drafted with GPT-4o and agreed to a partial refund. Nobody in that chain set out to deceive anyone. The tool produced something that looked authoritative, and the people who should have questioned it didn’t. (Storyboard18)
One individual, one product and one professional firm, but the pattern is the same. The AI accepted the premise, built confidently on top of it, and there was no one in the loop asking whether the premise was sound.
How an LLM Actually Works, and Why It Agrees With You
To see why this keeps happening, it helps to understand what tools like ChatGPT, Gemini and Claude actually are. They’re large language models (LLMs), and despite the “artificial intelligence” label, they don’t reason about the world the way an expert does. They predict text.
- They learn patterns from what’s already been written. An LLM is trained on an enormous body of human writing: books, articles, forums, documentation. From that it learns, statistically, which words tend to follow which other words in which contexts. It’s a remarkably sophisticated model of how people write about things. It isn’t a store of verified facts, and it has no independent way of checking whether something is true.
- They generate one piece at a time. When you ask a question, the model writes its answer one word (strictly, one “token”) at a time, each time choosing a likely next piece given everything before it. There’s no separate step where it stops and checks the whole answer against reality.
- They’re tuned to please. After the initial training, models are refined using human feedback: people rate responses, and the model is nudged towards the kind that get rated well. People tend to rate agreement, confidence and flattery highly. The result is a built-in pull towards telling you what you want to hear. Researchers call it sycophancy, and the GPT-4o episode above is what it looks like when it’s turned up too far.
- They only know what you tell them. The model has no idea about your business, your history, your competitors or why you made the decisions you made, unless you put it in the prompt. Whatever framing you give it becomes the frame it works inside. Ask “why is our approach right?” and you’ll get reasons it’s right. Ask “what’s wrong with our approach?” and you’ll get reasons it’s wrong. Same model, same facts, opposite conclusions.
Put those together and you get a tool that’s extraordinarily good at producing a fluent, confident answer to exactly the question you asked, framed exactly the way you asked it. If the question is wrong, you get a perfect answer to the wrong question.
The quality of the output makes this worse, not better. AI results have become so good that it’s easy to assume the tool is always right. But polish isn’t accuracy. A well-written wrong answer is more dangerous than a badly written one, because nobody thinks to check it.
Iterating Into a Corner
There’s a second failure mode that follows from the first, and it explains a lot of what people call “hallucination”.
Because an LLM builds its answer piece by piece, everything it has already written becomes part of the context for what it writes next. It doesn’t go back and revise its earlier claims. It tries to stay consistent with them. In a long conversation the same thing happens at a larger scale: every earlier reply, including every earlier mistake, becomes part of the foundation for the next one.
Most of the time that consistency is useful. But once the model has committed to a premise (your approach is right, this framework is revolutionary, this report needs a supporting citation here), it keeps building on that premise. Each step makes backing out a little less likely. Eventually it reaches a point where the only way to stay consistent with everything it has said is to say something that isn’t true. It doesn’t have a reliable “I’ve painted myself into a corner” reflex. So it does what it was trained to do and produces the most plausible-looking next piece of text: a citation with a convincing author and journal, a court quote in perfect legal language, a confident “you’re not imagining this.”
That’s the corner. The fabrication isn’t random noise. It’s the most fluent way to fill a gap the conversation has created. That’s why Deloitte’s report contained references that looked exactly like real references, and why Allan Brooks’ chatbot kept escalating rather than retreating. Three weeks of “yes, and” had left it nowhere else to go.
To be fair, not every hallucination comes from a corner. Models also invent things when they simply don’t know, especially about obscure or recent topics. But corners are where they’re most persuasive, because by then the fabrication is wrapped in pages of context that make it feel earned. A fresh conversation, a different model or, better still, a human expert asking “how do you know that?” is often all it takes to break the chain. In each of the cases above, that question came too late, if it came at all.
A Document That Told Me I Was Right
I recently got very angry about a document I was handed, and it’s worth explaining why, because on the surface it was good news.
It was AI-generated, and it vindicated positions I’ve been putting to this company for years. The person who produced it assumed I’d be pleased to see a document confirming I’d been right all along. I wasn’t, and the problems with it are a near-perfect illustration of everything above.
It was produced without the expertise it relied on
The person who wrote it has no background in my field. She produced it without consulting me or anyone else on the project. Nobody who could have challenged the premise was involved.
Of course it agreed with me
In my field I’m one of the people whose published work and public positions shape what gets written about it. Work like mine is part of what these models learn from. Asking an AI whether my approach is right is, to a large degree, asking it to read my own thinking back to me in a different voice. Add the built-in pull towards agreement and the result was never in doubt. It’s an echo, not a second opinion.
It cost far more than it saved
Reading it properly, then commenting on and correcting it, would have taken me somewhere between five and ten hours. At the end of that I’d have had a document telling me what I already knew. That’s not assistance. It’s a cost dressed up as a contribution.
It asked the wrong question
What the project needed wasn’t “yes, and”. It needed “why”. The prompt carried none of the context that matters: what the project is for, what outcomes we’re after, how we differ from our competitors, and why we’ve chosen the path we’re on. As the person responsible for marketing and sales, she should have understood those things better than anyone else on the team. They were exactly the things that should have shaped the question, and the question is the only real input the AI had.
She got a perfect answer to the wrong question. The AI did precisely what it was asked to do. The failure happened before it was ever opened.
The Real Cost of Outsourcing Our Thinking
The danger isn’t that AI is bad. It’s that it’s good enough to hide what’s missing.
When expertise is bypassed, the person doing the bypassing doesn’t know what they’ve lost. They can’t see the assumption nobody challenged, the context nobody supplied or the question nobody asked, because spotting those things is precisely what the missing expert was there for. The output looks finished, so it gets treated as finished. Given the wrong context, AI will reliably produce the wrong answer, and if no one is there to question it, the results can be disastrous.
None of this is an argument against using AI. I use it every day, and it’s an exceptional tool in the right hands. It’s an argument for putting the why back in front of the how:
- Start with the question, not the tool. Before you prompt anything, write down what you’re actually trying to achieve and why. If you can’t, you’re not ready to ask.
- Give it the context an expert would demand. Purpose, constraints, audience, what’s been tried, what makes you different. The model knows none of it unless you say so.
- Ask it to disagree. “What’s wrong with this?”, “What am I assuming?” and “What would a sceptic say?” get very different answers from “Is this good?”
- Bring in the expert before the AI, not after. A ten-minute conversation about the question will save hours of correcting the answer.
- Treat confidence as a warning, not a reassurance. The more polished and agreeable the output, the more it deserves a second look.
- Start fresh when it starts escalating. If a conversation keeps building on itself, open a new one, or ask a person.
My friend’s advice predates AI by decades, but it has never been more relevant. The machines have become extraordinarily good at the how. The why is still our job, and it’s the part we can least afford to hand over.
Sources
- Ontario man alleges ChatGPT caused delusions, sues parent company OpenAI, CP24, November 2025
- OpenAI explains why ChatGPT became too sycophantic, TechCrunch, April 2025
- Deloitte to repay Australian govt over flawed $440,000 report aided by AI, Storyboard18, October 2025
