Insight · Observability
Signal Not Noise: Design Observability That Matters
Stop drowning in alerts. Design for what actually breaks your product — so your team responds with confidence.
What this article covers
How to design observability around user impact so alerts become more useful and incident response becomes faster.
Who it is for
Engineering and platform teams trying to reduce alert fatigue, improve MTTR, and make runbooks more practical.
Expected outcome
Fewer low-value pages, clearer signals during incidents, and more confidence in the alerts that remain.
The problem we see
You're getting 200 alerts a day. Your team ignores most of them. Then the one alert that matters gets missed, and suddenly customers are calling. By the time someone notices, MTTR has tripled.
Classic alert fatigue
- CPU spikes → false alarm 19 out of 20 times
- Disk at 85% → everything is fine until it suddenly isn't
- P95 latency elevated → could be legitimate traffic or a real problem
Signal vs Noise: The difference
❌ Noise (gets ignored)
"CPU above 80%"
No context. No action. Nobody knows what to do.
✓ Signal (gets action)
"Payment API returning 500s (15% error rate). Checkout flow impacted. Runbook: [link]"
Clear. Actionable. Your team knows exactly what to do.
Four design principles
Alert on user impact, not infrastructure
❌ Bad: "Database disk 90% full"
✓ Good: "Orders taking 10+ seconds. Checkout flow at risk."
Every alert needs context
If someone gets paged at 3am, they should know:
- ✓ What's broken? Not "CPU spike" but "Search API >5s"
- ✓ Who's affected? Which users, services, regions?
- ✓ What do I do? Link to the runbook. No guessing.
Runbook first, alert second
Don't create an alert unless you have a documented response. If there's no runbook, your team can't act anyway — so why alert them?
Tune alerts monthly
Track which alerts actually require action. Which ones fire but don't matter? Tune thresholds or delete them. An alert that cries wolf should not exist.
Real example: Order API
❌ Before (noise):
"API latency P95 > 2 seconds"
Engineers see this and ignore it. 80% false alarm rate.
✓ After (signal):
"Order creation API taking >5s. Checkout completion rate dropping. Customer impact: 0.3% of transactions affected. Runbook: [link]. Recent deployments: [link]."
Clear, actionable, contextual. Team responds immediately.
Five steps to implement
Map your critical paths
What are the 3-5 user journeys that matter most? (Checkout, login, search, payment processing)
Define SLIs
For each critical path, what does 'working' look like? (Checkout <5s, 99.5% success rate)
Alert on SLI violations
Alert when checkout success drops below 99.5%, not when database CPU spikes to 85%.
Write runbooks
For every alert, document the response. 'If order processing is slow, check: [checklist]'
Tune relentlessly
Monthly: Review alerts. What fired? Was it actionable? Tune or delete.
Common questions
What is observability signal versus noise?
Signal means alerts and telemetry that clearly point to meaningful user or business impact, while noise refers to frequent low-value alerts that teams learn to ignore.
Why does alert fatigue happen?
Alert fatigue happens when teams receive too many unactionable notifications without enough context, ownership, or clear response steps.
What makes an alert actionable?
An actionable alert explains what is broken, who is affected, and what the responder should do next, ideally with a linked runbook.