Ask an AI assistant about a GST rule and it may give you a clear, confident answer, with a section number that doesn't exist. Ask for a statistic and it may produce a precise-looking figure with no real source. This is usually called hallucination, and anyone using AI for business work needs to understand it.
Why it happens
AI language models are trained to produce the most likely next words, given everything before them. They are very good at producing text that sounds right. Most of the time, sounding right and being right coincide. But the model has no built-in sense of "I don't know". When it lacks the facts, it can fill the gap with something plausible.
Newer models hallucinate less than older ones, and tools that search the web or read your documents are more reliable. None of them are free of it.
Where it's most likely in business work
- Numbers: prices, rates, percentages, totals in a summary
- Laws and rules: section numbers, deadlines, thresholds, GST rates for specific items
- Names and citations: court cases, studies, people, quotes
- Recent events: anything after the model's training cut-off, unless it searched the web
- Your own business: if you didn't tell it your prices or policy, it will guess them
- Long documents: summaries can merge or misplace details
It's least likely on general writing tasks: rephrasing, tone, structure, or turning your notes into a cleaner message.
Six habits that catch it
1. Give it the source. Instead of "What's the cancellation policy for…?", paste your actual policy and ask it to answer from that. AI working from a document you provide is far more reliable than AI working from memory.
2. Ask it to quote. "For each point, quote the exact sentence from the document it came from." If it can't quote it, treat the point as unverified.
3. Ask "how sure are you, and what would you check?" Models often flag their own weak spots when asked directly.
4. Check numbers against the original. Every price, date and figure in anything that leaves your business.
5. Use search-enabled tools for current facts, and open the sources they cite. A citation that doesn't contain the claim is a warning sign.
6. Keep a person on the send button for anything going to customers, tax authorities, banks or lawyers. That's the core of human in the loop.
What this means for customer-facing AI
A chatbot or WhatsApp assistant answering customers can hallucinate too: inventing a discount, a timing or a service you don't offer. Good setups reduce this by:
- answering only from your approved information (price list, FAQs, policies),
- being told to say "I'll check with the team" when the answer isn't there,
- handing over to a person for anything about money, health or legal matters,
- logging conversations so someone reviews a sample every week.
If a vendor can't explain how their assistant avoids making things up, ask before you buy. See questions to ask an AI agency.
Checklist: catching AI mistakes
- Paste your own source documents instead of asking from memory
- Ask the AI to quote where each fact came from
- Check every number, date and rate before it leaves the business
- Use search-enabled tools for anything recent, and open the sources
- Customer-facing assistants answer only from approved information
- A person approves anything legal, financial or medical
- Weekly review of a sample of AI conversations
For more on using AI safely, read is ChatGPT safe for business data?. If you're building a customer-facing assistant, we set them up to answer only from your approved information. See AI agents.
