We replaced an LLM feature with a regex and nobody noticed

This is the regex post I threatened in June.

Context: one step of our support pipeline extracts identifiers from inbound emails: order numbers, return authorisation (RMA) codes and carrier tracking numbers. Last year, in the general enthusiasm, this step was moved from a pile of old code to an LLM call: "here is an email, return JSON with any order number, RMA code and tracking number you find". It worked. It demoed beautifully. It ran for about nine months.

In August we were asked to cut costs on the pipeline. I went looking for the most expensive thing per unit of value, and this was it.

The numbers before

• About 55,000 emails a day.
• p50 latency 1.1 s, p95 2.8 s, on the critical path before routing.
• About $410 a day. Not a lot of money in isolation, and a lot of money for string matching.
• Exact-match accuracy on a 2,000-email labelled sample: 96.1%.

What the identifiers actually look like

This is the part nobody had looked at since the migration. I pulled 2,000 random emails, labelled them by hand over two afternoons (with help: thank you to the support leads, who accepted a box of pastries as payment), and looked at the formats:

• Order numbers: ORD- plus 8 digits, though customers write "ord 12345678", "order #12345678" or just the 8 digits after the word "order".
• RMA codes: RMA plus 6 alphanumeric characters, often with a space or a dash.
• Tracking numbers: the formats of our three carriers, which are documented and stable. One of them uses the UPU S10 format (two letters, nine digits including a check digit, two letters).

Identifiers are not natural language. They are designed by engineers to be machine-readable. That should have been the first clue.

The replacement

Roughly this, simplified:

import re

ORDER = re.compile(
    r"\b(?:ORD[-\s]?|order\s*(?:no\.?|number|#)?\s*:?\s*)(\d{8})\b",
    re.IGNORECASE,
)
RMA = re.compile(r"\bRMA[-\s]?([A-Z0-9]{6})\b", re.IGNORECASE)
S10 = re.compile(r"\b([A-Z]{2}\d{9}[A-Z]{2})\b")  # UPU S10 tracking numbers

def s10_check_digit_ok(code: str) -> bool:
    digits = [int(d) for d in code[2:10]]
    weights = [8, 6, 4, 2, 3, 5, 9, 7]
    total = sum(d * w for d, w in zip(digits, weights))
    check = 11 - (total % 11)
    check = 5 if check == 11 else 0 if check == 10 else check
    return check == int(code[10])

def extract(text: str) -> dict:
    orders = {m.group(1) for m in ORDER.finditer(text)}
    rmas = {m.group(1).upper() for m in RMA.finditer(text)}
    tracking = {m.group(1) for m in S10.finditer(text) if s10_check_digit_ok(m.group(1))}
    return {"orders": sorted(orders), "rmas": sorted(rmas), "tracking": sorted(tracking)}

Plus about 40 lines for the other carriers' formats and some normalisation (including stripping zero-width spaces, which customers paste in from PDFs more often than you would think).

Results on the same 2,000 emails

                      LLM        regex
exact match         96.1%       97.4%
false positives        31           9
missed identifiers     47          43
p95 latency        2,800 ms     0.3 ms
cost per day          ~$410        ~$0

The regex is slightly more accurate. Most of the LLM's false positives were plausible-looking numbers it promoted to order numbers (a phone number, a postcode followed by some digits), and three were order numbers that did not appear anywhere in the email. (Yes, again. See my earlier post on evals. Generation will generate.)

The misses are similar for both: identifiers split across a line break, or written with spaces in the middle ("ORD 1234 5678"). Some of those we fixed with normalisation; some are genuinely ambiguous to a human too.

The fallback

We did not delete the LLM. If the regex finds nothing and the email classifier says the email is about an order, we call the LLM, and we only accept an identifier from it if it appears verbatim in the text. That path runs on about 4% of emails. Cost went from about $410 a day to about $17.

"Nobody noticed"

We shipped it behind a flag, ran both versions in parallel for two weeks, compared outputs on every email, then switched. Support agents were not told. Nobody filed a ticket, nobody asked what had changed. The routing step got 2.8 seconds faster at p95, which somebody in support did eventually notice, and attributed to "the servers being upgraded".

What I would actually conclude

• When the input is designed to be machine-readable, use a machine-readable method. LLMs are remarkable at language. Order numbers are not language.
• The LLM version was not a mistake when it was built. It got the feature working quickly, and it gave us outputs to compare against. The mistake was never revisiting it.
• Check the boring option first, and check it again every time someone asks you to cut costs. "Could this be a regex, a lookup table or a SQL query?" takes ten minutes to answer.
• Regexes have their own failure modes (I have written some I am not proud of). Test them against labelled data exactly as you would test a model. The 2,000 labelled emails did more for this project than either the LLM or the regex.

I fully expect to be told this is obvious. It is. It was also running in production for nine months at about $150,000 a year.