Evals are just tests you are afraid to write
If you cannot say what a regression looks like, you do not have a feature. You have a demo.
Contents
We shipped a retrieval feature with no eval set because "the outputs are subjective". Six weeks later we changed the chunking strategy and had no way to know whether it helped. That is not an ML problem. That is shipping untested code and calling the untestedness a property of the domain.
The set that took one afternoon
120 real queries pulled from production logs, stratified: 40 head, 40 torso, 40 tail. Two of us labelled the correct source document for each. That is it. It is not a benchmark, it will not appear in a paper, and it caught a 9-point recall regression the following week that no one had noticed by eye.
Three metrics, and only three
- Recall@5 — is the right document in the window at all? If not, nothing downstream can save you.
- MRR — how far down is it? Position matters more than teams expect.
- p95 latency — a 3-point quality win for 400ms is not a win, it is a tradeoff you must argue for explicitly.
Run it in CI, on a schedule, cheaply
Ours runs on every PR that touches the retrieval path and nightly against production data. It costs about $0.40 a day. The nightly run is the one that matters: it is how we found that embedding drift after a corpus refresh had quietly cost us 4 points of recall over three weeks, with no code change to blame.
The uncomfortable finding: our first model was never better than BM25 on tail queries. We only learned that because we finally measured. The heuristic you are replacing deserves a place in your eval table.
Written by
Sana Qureshi
ML engineer working on retrieval and inference, not training runs. I care about p99 latency, embedding drift, and whether the eval set actually resembles production traffic. Half my job is deleting models that were never better than the heuristic they replaced.
3 Comments
Sign in to join the discussion
The 40 cents a day is the detail that makes this actionable. Every 'we should have evals' conversation I have been in died on an assumed cost nobody had measured.
It is 120 queries against an index that already exists. The expensive version is the one where you decide you need a benchmark before you are allowed to have an eval set.
'Shipping untested code and calling the untestedness a property of the domain' is the sentence. Same argument lands on 'you cannot test a query plan' — you can, you just have to decide what a regression is. A node type change and a 10x row estimate drift are both assertable.