We let AI write our tests for a quarter. Coverage went up, confidence did not

Quick context: 12 people, 5 engineers including me, TypeScript monorepo, Postgres, Vitest. In Q2 we had a mandate (from me, honestly 😅) to "use AI to fix our test debt". Coverage was 41% and our investors' tech DD person had made a face.

What we did

• Every engineer used an AI assistant to generate tests for untested modules, about 30 minutes a day.
• Rule: tests must pass and get reviewed in the PR like any other code.
• We tracked line coverage, mutation score (Stryker), and how many bugs a test caught before release.

The numbers after 12 weeks

line coverage             41%  ->  78%
test files                212  ->  604
CI time (tests)          3m40  ->  9m10
mutation score (billing)  52%  ->  55%
flaky tests per week       ~1  ->  ~6
bugs caught in CI        no real change

Coverage almost doubled. Mutation score barely moved. That is the whole story honestly.

Why

When we actually read the generated tests, a pattern: they test what the code does, not what it should do. The assistant reads the implementation and writes assertions that match it. If the code has a bug, the test enshrines the bug. Simplified example:

// implementation (bug: charges 19% VAT on reverse-charge invoices, which should have none)
export function invoiceTotal(inv: Invoice): number { /* ... */ }

// generated test: green, confident, wrong
it('calculates total for reverse charge invoice', () => {
  expect(invoiceTotal(reverseChargeFixture)).toBe(1428)
})

The correct number was 1200. Nobody checked 1428 by hand, because the test was green and there were 40 of them in that PR.

Mutation testing made it obvious: Stryker changes operators and values in your code and checks whether some test fails. Most generated tests survived mutations in the logic they "covered", because they asserted on snapshots or on whatever the function happened to return.

What actually worked

• Asking for test cases, not tests. "List the edge cases for reverse-charge invoices across EU countries" gave us a good checklist. Then a human wrote the expected values from the spec, not from the code.
• Boilerplate. Factories, fixtures, mocking our Postgres layer. Real time saved here, maybe 2 hours per engineer per week.
• Property-based tests with a human-written property. We write the invariant ("total is never negative", "sum of line items equals subtotal"), the AI writes the fast-check boilerplate.

What we changed

We deleted about 140 of the generated test files (mostly snapshot tests of things that change every week, which caused the flakiness). Coverage is 69% now, mutation score on billing is 71%. Lower coverage, much more confidence. CI is back to 5m20.

Tbh the lesson is old: a test is only as good as the person who decided what the right answer is. The AI cannot decide that for your business. Feierabend.