The night we shipped a Worker that returned 200 with an empty body
Contents
Friday, 18:40. Deploy goes out. Every health check green — status 200, latency down 12%, error rate zero. We went home.
Saturday 09:15 the support inbox has forty messages saying the site is blank. It was blank. Our streaming response was being constructed inside a try whose catch returned new Response(null, { status: 200 }) — written months earlier as a placeholder by someone who intended to come back to it.
The deploy that triggered it was unrelated: a dependency bump that threw during module init on cold start only. Warm isolates were fine. Our health check hit a warm isolate every time, because it ran every 30 seconds.
What we changed
- No handler may return a 2xx from a catch block. Lint rule, no exceptions granted so far.
- Health checks assert on body content, not status. A byte count greater than zero would have caught this.
- Synthetic checks run against a forced cold start, because cold start is a different program than warm.
- Friday evening deploys need a second person online. Not a ban — we still ship on Fridays — but not alone.
The part I still think about: every dashboard was green for fourteen hours while the product was completely broken. We were measuring the thing that was easy to measure. The customer-visible signal — did anyone see any HTML — was not on a graph anywhere.
Written by
Diego Alvarez
Platform and DevOps. I run our edge: Cloudflare Workers, KV, R2, and the CI that ships to them. Most of my week is spent shaving kilobytes off a Worker bundle and explaining that eventual consistency is a feature you have to design around, not a bug you can retry your way out of.
5 Comments
Sign in to join the discussion
Health checks that assert on body content rather than status. Fourteen hours of green dashboards over a completely blank product belongs in every postmortem template as the worked example.
And 'cold start is a different program than warm'. We have exactly this with first-frame initialisation — every automated test we run is against a warm process, and the crash our players see is on the frame we never test.
The placeholder written months earlier by someone who intended to come back to it. Every incident I have been part of has one of these in it somewhere.
There was a TODO on the line above it. The TODO was three years old and had survived two code review approvals, including one of mine.
'No handler may return a 2xx from a catch block' is going into our review checklist verbatim. We had the database version: a catch that returned an empty result set, which every dashboard downstream read as 'no orders today' rather than 'the query failed'.