AI hallucination
An AI hallucination is an answer that is fluent, confident and wrong, produced because the model generated plausible text rather than checking anything.
What it means in practice
A customer asks whether a dress can be returned after thirty days. Your policy says fourteen. The bot says thirty. Three weeks later the customer asks for a refund, with a screenshot of your own chat. You now have to choose between honouring a rule you never set and arguing with a customer who has it in writing.
This has already reached a tribunal. In February 2024, British Columbia’s Civil Resolution Tribunal ruled against Air Canada after its website chatbot told a passenger he could claim a bereavement fare discount after travel, which the airline’s policy did not allow.
| Moffatt v. Air Canada, 2024 BCCRT 149 | Detail |
|---|---|
| Decision date | 14 February 2024 |
| What the chatbot said | Book a regular fare, then claim the bereavement discount as a partial refund within 90 days |
| Airline’s defence | The chatbot was a separate entity responsible for its own actions. The tribunal rejected this. |
| Finding | Negligent misrepresentation. Air Canada did not take reasonable care that its chatbot was accurate. |
| Damages | C$650.88, plus interest and tribunal fees |
Source: Civil Resolution Tribunal of British Columbia, Moffatt v. Air Canada, 2024 BCCRT 149.
The amount was small, but the tribunal held the airline responsible for what its chatbot told the customer, in the same way as anything else on its website.
Why it happens
A language model produces likely text. When it has the facts, likely text is usually true. When it doesn’t, it still produces text, because nothing in the process makes it notice the gap. Newer models make fewer of these errors, but a text predictor asked a question it has no material for will still guess. So the work is to make sure it always has the material, or hands over.
Four things that reduce it
- Answer from your content. Retrieval puts your real policy in front of the model before it writes.
- Look up customer facts in the system that holds them. Order status, booking slots and stock should come from a query. See WISMO.
- Allow “I don’t know”. A system allowed to hand over will hand over instead of inventing. See live chat handover.
- Enforce limits in the software. If the refund ceiling is applied by the system, no wording gets the model past it.
How owners put it
Business owners rarely say “hallucination”. One request we have seen: “Respond only using the content I provide, without making things up.” Customers worry about it too. In a Gartner survey of 5,728 customers in December 2023, AI giving wrong answers was among the top concerns about AI in customer service, after difficulty reaching a person and job losses (Gartner, July 2024).
Related terms
Read next
Test it against your own policies
Start your 7-day free trial. No credit card needed. Load your returns and delivery pages, ask AeroChat the awkward questions your customers ask, and check every answer against what you wrote.
On Shopify? Install AeroChat from the Shopify App Store.