Reasonable Doubt · 4-minute read
Before AI, confident nonsense had to be assembled by hand. Someone misremembered a statistic, added a little certainty, and forwarded it to seventeen relatives. Small-batch. Locally sourced. Occasionally in all caps.
Now we can produce it with headings, a summary, and a tone that suggests it has already spoken to the manager.
That is the joke behind “I miss when bullsh*t was handmade.” The concern underneath it is real: an answer can look finished long before it has earned our trust.
Polish is doing a lot of work here
AI systems can produce plausible claims that are false. Researchers often call these “hallucinations.” OpenAI’s research describes how training and evaluation can reward guessing instead of acknowledging uncertainty. Better models can still make mistakes.
A smooth paragraph does not tell you whether a fact was checked. Neither does a table. Especially a table. Put nonsense in six tidy columns and suddenly it looks like someone’s quarterly strategy.
This matters outside work, too. A made-up quotation can become a social post. An outdated opening time can become a wasted trip. An assumption can become a very confident plan.
A great statistic. Shame nobody measured it.
Meet Sundial Grove Farms and its marketing agency, Copper Finch Creative. Both are fictional companies invented for this example.
The farm has upgraded part of its irrigation system. Its project notes say the equipment is installed and water-use measurements are still being collected. The agency asks AI to turn those notes into a customer update.
In our constructed example, the draft announces: “Our new irrigation system has reduced water use by 30%.” Beautiful sentence. Excellent headline. One tiny issue: the source notes never supplied that number.
The useful draft says what is actually known: the installation is complete, and the farm is collecting measurements to assess its effect. The marketing team can still explain the project clearly. The performance claim has to wait for evidence.
This is an illustrative scenario, not a recorded AI test or a real client result. It shows how an unsupported detail can turn into a public promise simply because it reads well.
Give a confident answer three checks
Find the load-bearing facts. Which dates, prices, quotations, permissions, or claims would change your decision if they were wrong? Start there.
Open the source. Check the original organization, document, or evidence. Does it actually support the claim? Is it current? A citation supplied by AI is another thing to check.
Look for the assumption. What did the answer quietly fill in? What relevant exception or contrary evidence might change the conclusion?
Try this follow-up prompt:
Review your previous answer. List the factual claims I should verify before acting, starting with those that would most change the decision if wrong. Separate supported facts, assumptions, and unknowns. For each important claim, provide the original source if you actually accessed it; otherwise say what I should check and where. Flag dates, prices, quotations, exceptions, and information that may have changed. Do not invent sources or fill gaps. What evidence would change your conclusion?
That prompt helps organize a review. It does not certify the answer. You still need to open the relevant sources yourself.
Keep the useful part
Let AI organize your notes, suggest alternatives, explain unfamiliar language, or turn a rough idea into a draft. Give it a clear task and enough context to help. Share only information you are permitted and comfortable to put into that tool.
Then give the important claims the attention they deserve. The more consequential the decision, the stronger the evidence and human review should be.
Keep the speed. Keep your judgment. Retire the bullsh*t, whatever its manufacturing process.
Your turn: What is the most confidently wrong answer you have caught, from a person or a machine?
Source: OpenAI, “Why language models hallucinate” (September 5, 2025), for the description of hallucinations and incentives to guess. The fictional companies, constructed output, three checks, and prompt are original editorial guidance, not a guarantee of accuracy.
