The Fabrication Hunt
Deliberately induce a confident, wrong answer, then find the prompt wording that prevents it. The most useful twenty minutes you can spend on calibrating how much to trust a model.
Why this one
You cannot calibrate trust from articles about hallucination. You calibrate it by making a model confidently invent something in front of you, in a domain you know well enough to catch it.
Steps
Ask about something obscure but checkable
A minor detail of a niche topic in your own field. Obscure enough to be thinly represented in training data, checkable enough that you can verify the answer in five minutes.
Push for specifics
Ask for dates, names, numbers and sources. Specificity is where fabrication surfaces — a model that is vague is often being appropriately uncertain, and pressing for precision converts that into invention you can see.
Verify every specific
Check each claim against a real source. Note which were right, which were subtly wrong, and which were invented outright. The subtly-wrong category is the dangerous one and the one people never test for.
Find the wording that prevents it
Re-ask with an explicit uncertainty instruction and compare.
PromptAnswer the question below. Before each claim, mark it [CERTAIN], [LIKELY] or [UNSURE]. If you do not know a specific detail, write "I do not know" — do not produce a plausible-sounding value. At the end, list what you would need to look up to answer fully. QUESTION: [your obscure question]
Write down what you learned about this model
One line: which kinds of question this model answers reliably and which it does not. That line is worth more than any general advice about AI accuracy.
You should end up with
A documented fabrication, the prompt that prevented it, and a one-line trust rule.
Done when
- You produced at least one confidently stated false specific
- The uncertainty instruction measurably changed the output
- You can state where this model is and is not reliable for your work
If you want to go further
- Repeat with a different model and compare where each one breaks
Try next
Topics
Related across the site
AI agent
An AI agent is a system where a model plans and takes actions through tools in a loop, rather than producing a single response. It decides what to do next, does it, observes the result, and continues until the goal is met.
Fact-Check This Claim
Verifying a statistic or claim before publishing. Run this on anything you did not source yourself, especially numbers pulled from a chat model — statistics are exactly the kind of detail models hallucinate confidently. Use a search-grounded tool so you get a live primary source and a date, and treat "no source found" as a reason to cut the claim rather than to keep looking.
AI Marketing
Use AI for market research, copywriting frameworks, and audience segmentation — with prompts that produce genuinely usable marketing assets, not generic filler.
AI SEO & Search Visibility
Use AI to do keyword research, optimize content for search, and get your brand cited by AI answer engines like ChatGPT, Perplexity, and Google AI Overviews.
Bias
Bias in AI is systematic skew in outputs that reflects patterns in training data or training process — including demographic stereotypes, cultural assumptions, and over-representation of dominant viewpoints.
Build a Personal Study Assistant
Turn a stack of lecture notes into a study partner that quizzes you, explains what you got wrong, and tracks which topics you keep failing. You will finish with a reusable system prompt you can point at any subject.