Postmortem: our support assistant invented a refund policy

A short postmortem, shared with permission from the team and with details changed enough that I am comfortable posting it.

Summary

For 41 hours, a retrieval-augmented assistant that drafts replies for our support agents told some customers that orders could be refunded within 60 days. The policy is 30 days. Agents approve every draft before it is sent; 23 drafts with the wrong number were sent anyway. We honoured all 23.

Timeline (local time)

• Tue 09:10: A help centre article about a holiday promotion ("extended 60-day returns on December orders") is re-published during a content migration, without its expiry date.
• Tue 09:40: The hourly indexer picks it up. It is now the most semantically similar document to "what is your refund policy".
• Tue 10:15: First draft citing 60 days.
• Wed 23:30: A night-shift support lead notices three drafts with 60 days in a row and escalates.
• Thu 03:15: Article unpublished, index rebuilt, queued drafts regenerated.

What the model did, precisely

It did not invent the number from nothing, which is what "hallucination" usually suggests. It retrieved two chunks: the real policy (30 days) and the promotion (60 days, no date). It chose the more specific-sounding one. The prompt told it to "always give the customer a clear, confident answer". It did exactly that.

I mention this because the word "hallucination" sends people looking at the model. In our case the model was the least broken part of the system.

Contributing factors

1. The prompt rewarded confidence over abstention. There was no instruction for what to do when sources conflict, and no allowed answer like "I am not sure, escalating".
2. Retrieval had no notion of time. Chunks carried no published or expiry date in their metadata, so nothing could prefer current policy.
3. Our eval set had no conflicting-source cases. 400 examples, every one with exactly one correct document. We were testing the easy version of the problem.
4. Agent review degraded. By the second month agents accepted 94% of drafts unedited. Human in the loop becomes human next to the loop very quickly when the drafts are usually right.

What we changed

• Policy answers (refunds, warranties, shipping times) now come from a small structured table maintained by the policy owner, not from retrieval. The model formats the answer; it does not look it up.
• Every chunk carries published_at and expires_at. Expired chunks are filtered before ranking, not after.
• The prompt has an explicit conflict rule: if retrieved sources disagree on a number, say so and route to a human.
• We added 60 eval cases with conflicting or stale sources. The old pipeline failed 38 of them. The new one fails 3, all of which escalate rather than answer, which is the failure mode we want.
• Drafts that contain a number of days, an amount or a percentage are highlighted for the agent. Small UI change, noticeably more edits.

What I would actually conclude

The cheapest fix was not a better model. It was taking the one category of answer where being wrong costs money out of the generative path entirely. If an answer must be exactly right and changes rarely, it belongs in a table, and the model's job is to read the table out loud.

The second lesson is about evals: we had a good number on a test set that did not contain the hard case. A high score on easy examples is not evidence about hard ones.