Posts tagged #sre on Dynamo Geeks: 4 posts from the community, most useful first.
deleted 41 alerts today that nobody had acted on in 6 months. pages went from ~30 a week to 4. if an alert doesnt have a clear action its not an alert, its a notification. send it to a channel and let it live a quiet life there
Quick write up because I have now explained this in three separate incident channels this month and I would like to just paste a link next time. Symptom: a Java service in kubernetes restarts every few hours. No exception in the logs, no stack trace, nothing. The app just... stops. kubectl describe pod shows this: Last State: Terminated Reason: OOMKilled Exit Code: 137 137 is 128 + 9, meaning the process got…
I have been on call for about nine years now across three companies and I can tell you the single biggest thing that decides if an incident takes 12 minutes or 3 hours is not the monitoring stack, its not kubernetes, its not how smart your team is. its whether the runbook was written for a person who is half asleep. Most runbooks I've inherited were written by someone who was wide awake, had full context, and was…
