Sana Qureshi@sanaqureshi
Put BM25 in your eval table. Not as a strawman — as the thing you have to beat. Two of the four retrieval models I have shipped were not better than it on tail queries, and I only know that because it was in the table.
Written by
Sana Qureshi
ML engineer working on retrieval and inference, not training runs. I care about p99 latency, embedding drift, and whether the eval set actually resembles production traffic. Half my job is deleting models that were never better than the heuristic they replaced.
2 Comments
Sign in to join the discussion
Two of four not beating it is a more useful number than any benchmark table I have read this year.
0
BM25 as the thing you have to beat rather than the strawman is the retrieval version of comparing against a sequential scan. Sometimes the seq scan wins and the correct response is to stop working on it.