Skip to content
question

How are you versioning prompts together with their eval results?

Dana Kowalski@dana_ml

October 11, 2026 · 1 min read

#3 in My Model Is Fine, The Data Is Not

A practical question rather than an opinion, for once.

We have about 30 prompts in production across six features, and each has an eval set. Today the prompts live in the application repo as text files, eval results go into a spreadsheet, and the link between "this prompt version" and "these scores" is a commit hash someone pastes into a cell. It works until it does not: last week nobody could tell me which version of a prompt produced a regression we saw in July.

What I want:

  • Every prompt change has an eval run attached, with the model version and parameters.
  • I can diff two prompt versions and see the score change per eval slice.
  • No new SaaS if possible (procurement takes months), and nothing that sends production data to a third party.

What are people actually using? Plain git plus CI artifacts? MLflow? Something homegrown? I am interested in what has survived a year, not in what looks good in a launch post.

Top answer

Núria Puig#6 in My Model Is Fine, The Data Is Not2 likes

We are small so we do the git + CI way too, but one tip: store a hash of the eval set in the result file. We once 'improved' 4 points because someone removed the hard examples from the eval set. Bueno, it was me.

View in the thread

#llm#machine-learning#devops

Get the best of #my-model-is-fine weekly

One email each Monday with the most useful posts. No spam; unsubscribe in one click.

Comments

7

Sign in to join the conversation.

Sign in to comment
Nguyen Minh Anh@minhanh_ml#9 in My Model Is Fine, The Data Is Not
We use plain git and CI, it survived 14 months so far. Each prompt is a YAML file with the prompt text, model name, temperature and the eval set name. CI runs the eval on every PR that touches a prompt file and writes a JSON result into the repo, under results/prompt-name/commit-hash.json. A small script makes a markdown table diff between main and the branch and posts it as a PR comment. Simple, nothing leave our infra. Your regression question, we answer it with git log on the results folder.
1 net votes
Dana Kowalski@dana_ml#3 in My Model Is Fine, The Data Is Not
This is close to what I had in mind, and committing results next to the prompt is the piece we are missing. Did you have trouble with eval runs being non-deterministic between CI runs?
2 net votes
Nguyen Minh Anh@minhanh_ml#9 in My Model Is Fine, The Data Is Not
Yes. We run each eval 3 times at temperature 0 and report mean and min. Still some noise, so the PR comment only goes red if the drop is bigger than what we see between repeated runs on main. We measure that noise once a month.
2 net votes
Lin Wei@linwei#1 in My Model Is Fine, The Data Is Not
MLflow works for this if you already run it for classic models: log the prompt text as an artifact, model and params as params, eval metrics as metrics, one run per prompt version. The comparison view does the per-slice diff. We self-host it, so nothing leaves. Heavier than minhanh's approach, but it's survived 2 years for us.
1 net votes
Núria Puig@nuria_data#6 in My Model Is Fine, The Data Is Not
We are small so we do the git + CI way too, but one tip: store a hash of the eval set in the result file. We once 'improved' 4 points because someone removed the hard examples from the eval set. Bueno, it was me.
2 net votes
Dana Kowalski@dana_ml#3 in My Model Is Fine, The Data Is Not
The eval-set hash is an excellent idea and an honest confession. Adding both to the list.
0 net votes
Danny Cho@danny_cho_la#10 in My Model Is Fine, The Data Is Not
Boring-tech vote for git plus CI. Anything that puts the prompt somewhere other than next to the code that calls it will drift. We tried a prompt registry service, and within three months the 'registry' version and the deployed version disagreed.
0 net votes