How do you keep an embedding index fresh without re-embedding the whole corpus every night?
1.4M documents, roughly 3% change per day, mostly small edits rather than new documents. Full re-embed is about six hours of GPU time and we are doing it nightly, which is both expensive and slower than the business wants updates to appear.
Incremental is obvious in principle — embed only what changed — but two things make me nervous:
- Model version skew. The moment we upgrade the embedding model, the index holds vectors from two models and cosine similarity between them is meaningless. Do you dual-write into a shadow index and swap, or is there a cheaper pattern?
- Chunk boundary churn. A one-word edit near a chunk boundary can shift every subsequent chunk in the document, so "3% of documents changed" quietly becomes "18% of chunks changed". Content-defined chunking would fix it, but it makes chunks non-reproducible across versions.
What is working for people at a few million documents? Specifically interested in whether anyone found the shadow-index swap manageable operationally, or whether it turned into a permanent second system to maintain.
Written by
Sana Qureshi
ML engineer working on retrieval and inference, not training runs. I care about p99 latency, embedding drift, and whether the eval set actually resembles production traffic. Half my job is deleting models that were never better than the heuristic they replaced.
3 Comments
Sign in to join the discussion
Content-hash keying is the piece I was missing. It turns chunk boundary churn from a scary property into a number I can compute on the diff before scheduling anything. Running it on last week's edits today.
On cost: run the incremental job as a queue consumer rather than a nightly batch. The spend spreads out, and 'freshness' becomes a lag metric you can alert on instead of a cliff you discover the next morning.
The shadow index is manageable if you run it as a column type migration rather than as a second system:
1. New index, write-only, no reads. 2. Backfill in bounded batches you can stop, with a partial index on the not-yet-embedded predicate so the batch selection does not degrade as the backfill progresses — ours went from 2s to 90s per batch before we added that. 3. Dual-read behind a flag and compare, do not cut over. 4. Swap. 5. Delete the old index on a date that is in the same ticket as the swap. Step 5 is the one that decides whether this is a migration or a permanent second system, and it fails because nobody wants to be the person who deletes it.
On chunk churn: key the vector by a hash of the chunk's own content rather than by its position. Re-embedding a document becomes a set difference — an edit near a boundary re-embeds the chunks whose content actually changed and everything else is a cache hit. Your 18% number is then something you can measure before you spend the GPU hours instead of a thing you fear.