the renewal test that only failed one night a year

promised this one ages ago so here it is.

We had an end to end test for policy renewals. It ran every night at half twelve: create a policy starting today, jump forward a year, check the renewal quote and the end date. Green for months. Then one night at the end of October it went red, a policy that started on the 26th came out with an end date of the 25th the following year. Next night green again. Classic flake, everyone said, re-run it and move on.

I didnt let it go because it failed exactly once, at exactly that time, and real flakes arent that tidy.

The policy service calculated the end date by adding 365 * 24 * 60 * 60 * 1000 milliseconds to the start, in Europe/Dublin time. At half twelve that night we were still on summer time, the clocks only go back later in the night. A year later the same date was already winter time, because the change is on the last Sunday of October and that Sunday moves every year. So you add exactly 365 days of milliseconds, the offset is an hour different, and you land at half eleven the night before. Wrong day. And 365 days isnt even a year when a 29th of February gets in the way, so the bug had a second door.

The fix was boring. Things that are dates (a policy starts on a day, not at an instant) are stored as dates, and you add calendar units not milliseconds. In Java that was LocalDate.plusYears(1) instead of epoch maths. We also added tests that pin the clock to the nights of both DST changes and to the 29th of February.

Then the scary bit. Production had been doing this for years. A handful of real policies had end dates one day early. Small number, but its insurance, a day without cover isnt a rounding error, its a complaint to the regulator waiting to happen. We fixed them all and thank god nobody had a claim in that gap.

So yeah. When a "flaky" test fails on a date that sounds suspicious, it isnt flaky. Its telling you something.