deleted 41 alerts today that nobody had acted on in 6…
Marcus Johnson@marcus_oncall
deleted 41 alerts today that nobody had acted on in 6 months. pages went from ~30 a week to 4. if an alert doesnt have a clear action its not an alert, its a notification. send it to a channel and let it live a quiet life there
Written by
Marcus Johnson
SRE at a freight logistics company in Atlanta. Kubernetes, Terraform, pagers. I have been woken up by every possible alert. Smoker (the BBQ kind).
Get the best of Dynamo Geeks weekly
One email each Monday with the most useful posts. No spam; unsubscribe in one click.
Comments
Sign in to join the conversation.
Sign in to commentKevin Tran@kevtran_dev
how do you even decide which ones to delete? scared id delete the one that matters lol
Marcus Johnson@marcus_oncall
pull 6 months of alert history. for each alert check if anyone actually did something in response (a rollback, a commit, a ticket, anything). zero actions = delete it or downgrade it to a channel. keep the alert config in git so you can bring one back if you were wrong