Agent demo vs production: what happened when our 9-out-of-10 demo met 1,200 real tickets

In the spring we were shown an agent demo: give it a customer request ("I was charged twice, please fix it"), and it reads the account, finds the duplicate charge, issues a refund and writes a reply. Ten scripted requests, nine handled end to end. Leadership was ready to staff down a queue.

We ran a four-week shadow pilot instead: the agent processed real tickets in parallel with humans, with its actions logged but not executed. Here is what we found. (Details are blurred; the numbers are real.)

The headline numbers

tickets in scope:                1,200
fully correct end to end:          61%
correct but needed a human step:   17%
wrong action proposed:             14%
gave up / asked for help:           8%
median tool calls per ticket:       9
p95 latency:                     94 s
model cost per ticket:          $0.38

61% is not nothing. It is also not 90%, and the 14% "wrong action proposed" bucket included refunds to the wrong order and one attempt to cancel a subscription the customer wanted to keep.

Why the demo said 90%

1. The demo requests were one-step problems in a costume

Every scripted request had exactly one account, one order and one obvious action. Real tickets have a customer with two accounts, an order split across two shipments, and a refund already partially issued by a colleague last week.

2. Errors compound

The median real ticket took 9 tool calls. If each step is right 95% of the time, independently, the chance that all nine are right is 0.95^9, about 63%. That is almost exactly what we measured. The demo tickets took 3 steps: 0.95^3 is about 86%. Same model, same per-step quality, very different outcome, purely because of length.

This is the single most useful piece of arithmetic for anyone evaluating agents. Per-step accuracy has to be extremely high before long tasks become reliable, and you will never see that in a short demo.

3. The environment was clean

The demo ran against a seeded test account. Production has tool timeouts, an internal API that returns HTTP 200 with an error message in the body, and a refunds endpoint that rate limits at the worst moment. 31% of the wrong actions traced back to the agent reading a tool failure as a success.

4. Nobody had defined "correct"

We spent the first week of the pilot writing a rubric with the support leads. Is a correct refund with a slightly rude reply correct? (No.) Is asking the customer a clarifying question a failure? (No, often it is the right answer.) Without a rubric, the demo's "nine out of ten" was nine out of ten according to the person who wrote the demo.

What we shipped instead

• The agent handles one narrow ticket type end to end: duplicate charges on a single order with no prior refund. That subset is 22% of volume, and the agent was correct on 96% of it in the pilot.
• For everything else it drafts a plan and a reply for a human to approve. That saves real time without letting it act.
• Tool wrappers now raise loudly on any non-success, including the 200-with-an-error API, so the agent cannot mistake a failure for a result.
• Every irreversible action (refunds, cancellations) needs a human click. This costs very little, because the agent's plans for these are mostly right, and the 4% that are wrong are expensive.

What I would actually conclude

• Ask for the step count of real tasks and the per-step success rate. Then multiply. It will tell you more than the demo.
• Shadow mode is cheap and decisive. Four weeks, no customer risk, and an argument replaced by a table.
• Narrow and reliable beats broad and impressive. Our agent handles a fifth of the queue alone and helps with the rest. That is a good result. It is not the result on the slide, and that is fine.

I expect the usual replies telling me I used the wrong model or the wrong framework. Possibly. Send me your shadow-mode table and I will happily read it.