The cost-per-request maths nobody does before launch

Every few weeks somebody tells me an LLM feature will cost "basically nothing" because the per-token price is small. Then the first invoice arrives. Here is the arithmetic I now ask for before anything ships, using a real feature with the numbers rounded.

(I am using illustrative prices: $3 per million input tokens and $15 per million output tokens. Plug in your provider's current numbers. The structure is the point, not the price.)

Step 1: the naive estimate

The feature drafts replies to customer emails. Someone measured a typical request:

• System prompt: 1,200 tokens
• Retrieved context: 1,000 tokens
• Customer email: 300 tokens
• Output: 400 tokens

input  = 2,500 tokens x $3  / 1,000,000 = $0.0075
output =   400 tokens x $15 / 1,000,000 = $0.0060
total per request                       = $0.0135

At 40,000 emails a day: $540 a day, about $16,200 a month. This is the number that goes on the slide. It is wrong in at least five ways.

Step 2: what actually happens per email

1. It is not one call. Our pipeline classifies the email first (a small call), then retrieves, then drafts, then runs a check for policy numbers, and redrafts when the check fails. Measured average: 3.4 model calls per email, not 1.
2. Context grows. Email threads. The median email had 1 prior message; the 90th percentile had 7. Average input for the draft call was 4,100 tokens, not 2,500.
3. Retries. Timeouts, malformed JSON, rate limits. 6% of calls were retried at least once.
4. Regeneration. Agents clicked "regenerate" on 11% of drafts.
5. Evals and development. The nightly eval run, prompt experiments, staging traffic. Ours came to 9% of production spend. Nobody budgets for it.

Step 3: redo it

draft call:      4,100 in, 420 out   = $0.0123 + $0.0063 = $0.0186
classify call:     900 in,  10 out   = $0.0027 + $0.0002 = $0.0029
policy check:    1,500 in,  60 out   = $0.0045 + $0.0009 = $0.0054
redrafts (0.4 extra draft calls on average)  0.4 x $0.0186 = $0.0074
per email, before retries                                 = $0.0343
x 1.06 (retries) x 1.11 (regenerations)                   = $0.0404
x 1.09 (evals and dev)                                    = $0.0440

About $0.044 per email. At 40,000 a day that is $1,760 a day, roughly $52,800 a month. 3.3 times the slide.

(Yes, applying the regeneration multiplier to the whole email instead of just the draft call is slightly pessimistic. Pessimistic is what you want from a budget.)

Step 4: the levers that actually move it

• Prompt caching. Most providers now discount repeated prompt prefixes. Our system prompt and policy text are identical across requests, so putting them first and keeping them stable cut the effective input cost of the draft call by about 40%. Free money, but only if you do not put the date or the customer's name at the top of the system prompt (we did).
• A smaller model for the small calls. Classification and the policy check do not need the big model. Moving them to one roughly 10 times cheaper took about three quarters of a cent off every email, with no measurable quality loss on our eval set.
• Trim the thread. Summarise older messages once and store the summary instead of resending seven full emails.
• Cap the output. Set max tokens. One prompt bug produced 2,000-token drafts for a day. That day cost three times a normal one.

After those changes we were at about $0.026 per email, around $31,000 a month. Still nearly double the slide, but a number we chose with our eyes open.

What I would actually conclude

• Measure calls per user action, not price per call. The multiplier lives there.
• Budget 10% for evals and experiments. If you have no evals, you will spend that 10% on incidents instead.
• Do the maths at the 90th percentile of input size, not the median. Long threads are where the money goes.

The per-token price is the least interesting number in the whole calculation.