Danny Cho

Your Go service has 600 Postgres connections and uses 6 of them

Last month a routine scale-up took down our orders database. Not CPU, not disk. Connections. Postgres said FATAL: sorry, too many clients already and every service that touched it fell over together. How we got there 40 pods of one Go service, each with a pgx pool. Nobody had set MaxConns , so pgxpool used its default: the larger of 4 and runtime.NumCPU() . Our nodes have 16 cores. 40 times 16 is 640 possible…

my parents' dry cleaner opened at 7am every day for 31 years. no status page, no retros, my dad just showed up. your 99.9% monthly SLA allows 43 minutes of downtime and somehow you still need a war room

The goroutine leak that took three weeks to find (the fix was four lines)

One of our Go services gained about 40 MB of memory a day. Nobody noticed, because we deploy every day and a deploy restarts every pod. Then came a holiday change freeze. Twelve days without a deploy. On day nine the pods started getting OOMKilled, one by one, during the busiest week of the year. Finding it The service already imported net/http/pprof on an internal port, which is the single best decision anyone made…

Re-reading this for the fifth time. Every team gets about three innovation tokens. Most teams I have joined spent theirs on the database and the deploy pipeline, then wondered why the product was late.