How I build evals for an LLM feature (the boring version)

This topic exists because I kept seeing launches justified with a demo video and a feeling. So here is the first real post: how I evaluate an LLM feature before it ships. Nothing here is novel. That is the point.

The feature, for context: classifying inbound support tickets into one of 14 categories and extracting an order ID if there is one. It is currently handled by a gradient-boosted model plus a regex (more on regex in some future post, I suspect). The question from leadership was "can the LLM do it better?"

1. Write down what "better" means before you look at anything

We agreed on three numbers up front:

• Macro-averaged F1 across the 14 categories (macro, because the rare categories are the expensive ones: chargebacks, legal threats).
• Exact-match accuracy on order ID extraction, including correctly returning nothing when there is no ID.
• p95 latency and cost per 1,000 tickets.

And one decision rule: ship only if macro F1 improves by at least 3 points with the 95% confidence interval excluding zero, at no more than 2x the cost of the current approach per ticket... which we immediately had to relax, as you will see. Writing the rule first still matters, because after you see the numbers everyone becomes very creative about which metric "really" counts.

2. Build a golden set you are slightly embarrassed by

We sampled 1,200 tickets from the last 90 days, stratified by category so the rare ones had at least 40 examples each. Two people labelled each ticket independently. Inter-annotator agreement (Cohen's kappa) was 0.71 on the first pass, which is decent and also tells you something important: humans disagree on roughly one ticket in six. Your model cannot be more accurate than your labels, and it is unfair to grade it as if it could.

We resolved disagreements in a 45-minute meeting per 200 tickets, wrote down every rule we invented ("a refund request that mentions a chargeback is a chargeback"), and that rule document became the labelling guide and, not coincidentally, most of the prompt.

(The golden set took about 30 person-hours. This is the step people skip. It is also the step that matters most.)

3. Split it, and lock one part away

• 300 tickets: development set. Look at these as much as you like while iterating on prompts.
• 900 tickets: test set. Run it at the end, a small number of times, and do not read individual failures until you have decided.

Prompt iteration is training. If you tune your prompt against the same examples you report on, you are reporting training accuracy. I have seen a 9-point "improvement" evaporate on a held-out set for exactly this reason.

4. Baselines first

Before any LLM call, the existing model on the test set, then the candidates:

                     macro F1   order ID exact   p95 latency   cost / 1k tickets
current (GBM+regex)    0.78         0.962            40 ms          ~$0.02
LLM, zero-shot         0.74         0.948          2,100 ms         $4.10
LLM, guide in prompt   0.83         0.951          2,300 ms         $6.80
LLM + 8 examples       0.84         0.955          2,600 ms         $9.30

Notice the zero-shot result is worse than the old model. Most demo comparisons are against no baseline at all, so that never shows up. Notice also that "2x the cost" was a silly rule against a model that costs two cents per thousand tickets; we replaced it with an absolute monthly budget, explicitly, in writing, before going further.

5. Confidence intervals, because 900 is not infinity

I bootstrap the test set (resample with replacement, 2,000 times) and compute the metric difference each time:

import numpy as np
from sklearn.metrics import f1_score

def bootstrap_diff(y_true, pred_a, pred_b, n=2000, seed=0):
    rng = np.random.default_rng(seed)
    y_true, pred_a, pred_b = map(np.asarray, (y_true, pred_a, pred_b))
    diffs = []
    for _ in range(n):
        idx = rng.integers(0, len(y_true), len(y_true))
        a = f1_score(y_true[idx], pred_a[idx], average="macro")
        b = f1_score(y_true[idx], pred_b[idx], average="macro")
        diffs.append(b - a)
    return np.percentile(diffs, [2.5, 50, 97.5])

For "guide in prompt" against the current model: median difference +0.05, 95% interval [+0.02, +0.08]. Passes the 3-point rule on the median, and the interval excludes zero. The 8-example version adds a point, with an interval that overlaps the cheaper one, at 37% more cost. We dropped it.

6. LLM-as-judge, only where you must, and calibrated

Classification does not need a judge: you have labels. We used a judge for one free-text field only (a one-line summary for the agent). Before trusting it, two humans graded 150 summaries on a 3-point rubric and we compared. The judge agreed with the humans 81% of the time, and it was systematically lenient on summaries that were fluent but missed the customer's actual request. So we report the judge score with that caveat, and re-check agreement every time the judge prompt or model changes.

7. Read the failures (the actual point)

Once the decision was made, I read all 140 test failures. Roughly:

• 38: the label was arguably wrong (see kappa above).
• 31: two plausible categories; the model picked the other one.
• 27: tickets in Spanish or Portuguese, where performance dropped noticeably. This became its own eval slice.
• 22: order ID formats missing from the dev set (an old system's IDs with a different prefix).
• 22: genuinely bad outputs, including 4 where it invented an order ID. Four invented IDs out of 900 sounds small until you remember an agent will look up the wrong customer's order.

The invented IDs alone justified a post-check: any extracted ID must appear verbatim in the ticket text. Three lines of code, four failures gone.

8. Put it in CI

The dev set runs on every prompt change (about 4 minutes, under $3). The test set runs before release. If macro F1 on the dev set drops more than 2 points, the PR fails. This is not sophisticated. It catches regressions a demo never will, including the time someone "tidied up" the prompt and removed the chargeback rule.

What I would actually conclude

• The LLM version is better on the thing we cared about, by a real but modest margin, at more than 300 times the cost per ticket of the old model. At our volume that is still a small number of dollars, which is the only reason it shipped.
• Most of the improvement came from writing down the labelling rules, which also improved the old model when we used them to clean its training data. Draw your own conclusions about where the value was.
• If you cannot produce a table like the one in section 4 for your feature, you do not know whether it works. You know that it demos.

Questions and corrections welcome. Particularly corrections.