What I learn after 3 months building eval set for our support bot

Three months ago our support bot go to production. Two weeks after, it tell a customer there is a discount code "SALE50". There was no such code. Customer was very angry, my boss was more angry. This is how we started to take evals seriously. Here is what I learn, maybe useful for someone.

1. Start from real conversations, not invented questions

At first we write 50 test questions ourself. Bot pass almost all. Then we sample 300 real tickets from the logs (after removing personal info), and the bot fail many of them. Real customers write short, mix Vietnamese and English in one sentence, make typo, send only "còn hàng ko" (still in stock?). Our invented questions were too clean.

2. Separate retrieval eval and answer eval

When an answer is wrong there are two very different reason: the right document was not retrieved, or it was retrieved and the model still answer wrong. We label for each test question which document should be retrieved and measure recall@5 separately. Our first problem was retrieval (recall around 0.6), not the model. After we fix the chunking and add product names to the chunk header, recall go to 0.85. Without separate metric we would keep changing the prompt for nothing.

3. LLM-as-judge is ok, but calibrate it

We use a bigger model as judge with a rubric: correct, partially correct, wrong, and "should have said I don't know". But first we label 100 answers by human (me and two colleague from the support team) and check how often the judge agree with us. First rubric agree only 70%. We rewrite the rubric with an example for each grade, now around 90%. Not perfect, but good enough to catch regression.

4. Test the "I don't know"

Most important category for us. We have 40 questions where the correct answer is "I don't have this information, let me connect you with our staff", like asking about a discount that does not exist. Before, the bot answered confidently on 60% of them. Now discount codes, prices and stock only come from tool call results, never from model memory, and the prompt say if the tool return nothing, say you don't know. Now it pass 37 of 40.

5. Run eval on every change

Every prompt change, model change or chunking change, we run the full set (about 450 case now) in CI. It take around 12 minute and cost less than one coffee. We compare with the last result per category, not only the total score, because sometime the total go up but one category go down a lot.

Still many thing to improve. But now when boss ask "is the new version better?", I can show number, not feeling. That is already big change.