Dana Kowalski

How are you versioning prompts together with their eval results?

A practical question rather than an opinion, for once. We have about 30 prompts in production across six features, and each has an eval set. Today the prompts live in the application repo as text files, eval results go into a spreadsheet, and the link between "this prompt version" and "these scores" is a commit hash someone pastes into a cell. It works until it does not: last week nobody could tell me which version…

I logged every AI-assisted coding task for a month. Here is the honest ledger

I keep asking people to measure, so it is only fair that I did it myself. For September I logged every coding task at work where I used an AI assistant: what it was, how long it took, how long I estimate it would have taken without, and whether I had to fix the output. The estimates are mine and biased; I tried to be conservative in both directions. The ledger tasks logged: 47 clearly faster: 29 (est. 11.5 h saved)…

For the PMs, designers and managers who keep asking me for 'something short that is not marketing': 3Blue1Brown's Large Language Models explained briefly. Short, accurate, and it does not oversell anything.

We replaced an LLM feature with a regex and nobody noticed

This is the regex post I threatened in June. Context: one step of our support pipeline extracts identifiers from inbound emails: order numbers, return authorisation (RMA) codes and carrier tracking numbers. Last year, in the general enthusiasm, this step was moved from a pile of old code to an LLM call: "here is an email, return JSON with any order number, RMA code and tracking number you find". It worked. It demoed…

Agent demo vs production: what happened when our 9-out-of-10 demo met 1,200 real tickets

In the spring we were shown an agent demo: give it a customer request ("I was charged twice, please fix it"), and it reads the account, finds the duplicate charge, issues a refund and writes a reply. Ten scripted requests, nine handled end to end. Leadership was ready to staff down a queue. We ran a four-week shadow pilot instead: the agent processed real tickets in parallel with humans, with its actions logged but…

If you are going to argue with me in the comments about what LLMs are, please watch this first. Andrej Karpathy's one hour intro is still the clearest explanation I know of pretraining versus fine-tuning, and of the model as a lossy compression of its training data.

The cost-per-request maths nobody does before launch

Every few weeks somebody tells me an LLM feature will cost "basically nothing" because the per-token price is small. Then the first invoice arrives. Here is the arithmetic I now ask for before anything ships, using a real feature with the numbers rounded. (I am using illustrative prices: $3 per million input tokens and $15 per million output tokens. Plug in your provider's current numbers. The structure is the…

Postmortem: our support assistant invented a refund policy

A short postmortem, shared with permission from the team and with details changed enough that I am comfortable posting it. Summary For 41 hours, a retrieval-augmented assistant that drafts replies for our support agents told some customers that orders could be refunded within 60 days. The policy is 30 days. Agents approve every draft before it is sent; 23 drafts with the wrong number were sent anyway. We honoured…

How I build evals for an LLM feature (the boring version)

This topic exists because I kept seeing launches justified with a demo video and a feeling. So here is the first real post: how I evaluate an LLM feature before it ships. Nothing here is novel. That is the point. The feature, for context: classifying inbound support tickets into one of 14 categories and extracting an order ID if there is one. It is currently handled by a gradient-boosted model plus a regex (more on…

Considering starting a topic for AI claims that come with numbers attached. Model, task, sample size, cost. No demo videos without an eval. Would anyone actually post there, or is this a club of one?