Six months on a feature that eleven people used
Contents
We built semantic search for an internal knowledge base. Embeddings, reranker, a nice hybrid scoring blend, an eval harness I was genuinely proud of. Recall@5 went from 0.61 with BM25 to 0.83. Shipped to 400 people.
Weekly active users after two months: eleven.
What we got wrong
Nobody was searching in the first place. The logs we tuned against came from the 6% of users who had ever opened the search box. Everyone else asked a colleague in chat, and the reason was not result quality — it was that finding the search box took three clicks from where people actually worked.
A colleague spent two days putting a search field in the chat tool as a slash command, pointed at the old BM25 index. Four hundred queries in the first week.
What I took from it
- Measure the funnel before optimising the step. Our step went from 0.61 to 0.83 inside a funnel where 94% of people never arrived.
- An eval set drawn from existing users encodes existing users' behaviour. It cannot tell you about the people who bounced.
- The good version of this story is that the reranker later went behind the slash command and did help — 0.83 on traffic that exists is worth something. It was just worth nothing for six months.
I do not regret the eval harness. I regret not spending one afternoon in week one asking why the search box was empty.
Written by
Sana Qureshi
ML engineer working on retrieval and inference, not training runs. I care about p99 latency, embedding drift, and whether the eval set actually resembles production traffic. Half my job is deleting models that were never better than the heuristic they replaced.
4 Comments
Sign in to join the discussion
0.61 to 0.83 inside a funnel where 94% never arrived is a beautiful sentence and a terrible quarter.
The slash command is the whole lesson. Distribution beat quality and it was not close.
It really was not. The reranker did eventually earn its place — but only once it was sitting somewhere people already were, which took two days of someone else's work rather than six months of mine.
'Finding the search box took three clicks from where people actually worked' is a design finding, and it took an ML project six months to surface it. Painful, and extremely familiar.