An Alert Nobody Acts on Should Be Deleted

I spent three years of my life maintaining a legacy payment gateway where the “state-of-the-art” dashboard had four hundred different metrics, yet none of them told me why the transaction success rate had plummeted to zero at 3:00 AM. We were drowning in telemetry, but we were starving for actual information. Most people think the solution is to buy more expensive observability suites or add more granular traces, but they’re missing the point of how to choose monitoring that gets read. If your alerts are just more noise to be filtered out by a tired engineer on a Sunday morning, you haven’t built a monitoring system; you’ve just built a digital landfill.

I’m not here to walk you through a vendor’s feature checklist or a “happy path” tutorial that assumes your infrastructure never fails. Instead, I want to talk about the messy reality of high-stakes backend systems. I’m going to show you how to strip away the vanity metrics and focus on the signals that actually trigger a meaningful response when the code starts breaking. My goal is to help you build a setup that provides clarity during a crisis, rather than just more data to ignore.

The High Cost of Ignoring Your Signal to Noise Ratio

Most monitoring setups suffer from a fundamental misunderstanding of what an alert is actually for. We’ve all been there: it’s 3:00 AM, and your phone is vibrating with a notification that a single microservice’s CPU spiked to 80% for three seconds. That isn’t an alert; it’s a diary entry. When you fail to manage your signal to noise ratio in monitoring, you aren’t just being noisy; you are actively training your engineers to ignore the very tools meant to protect them.

This is how you end up with a culture of apathy. When every trivial fluctuation triggers a page, you aren’t building resilience; you are just reducing alert fatigue by making everyone stop caring. The real danger isn’t the noise itself, but the fact that the one legitimate, catastrophic failure—the one that actually matters—gets buried under a mountain of irrelevant telemetry. If your team has learned to subconsciously filter out the “noise,” they will eventually filter out the signal too, right when the database starts melting down.

Why Symptom Based Alerting Beats Cause Based Chaos Every Time

I spent the better part of my thirties chasing ghosts in legacy payment gateways. Most of those late-night pages weren’t triggered by actual service outages, but by “cause-based” alerts: a CPU spike on a single container, a minor increase in disk I/O, or a single node rebooting. These are symptoms of something happening, sure, but they aren’t problems yet. When you alert on every minor fluctuation, you aren’t monitoring; you’re just creating a high-frequency noise floor that makes reducing alert fatigue an impossible dream.

The shift to symptom-based alerting is what actually saves your sleep. Instead of alerting because “Disk usage is at 80%,” you alert because “The checkout latency is exceeding 500ms.” One is a technical observation that might lead to nothing; the other is a direct signal that the user experience is degrading. By focusing on symptom-based alerting vs cause-based noise, you ensure that when your phone vibrates at 3:00 AM, it’s because something is actually broken, not just because a background process decided to be chatty. You want to know when the engine is smoking, not when the temperature sensor fluctuates by a single degree.

Five ways to stop your alerts from becoming background noise

• Alert on user impact, not CPU spikes. I don’t care if a container is sitting at 90% utilization if the transaction success rate is still 100%. If the customer isn’t feeling it, your pager shouldn’t be either.
• Enforce a “no action, no alert” rule. Before you click ‘save’ on a new monitoring rule, ask yourself: “If this fires at 3:00 AM, is there a specific, documented action I need to take?” If the answer is “just keep an eye on it,” delete the alert.
• Group by service, not by component. Getting ten different alerts for ten different microservices because a shared database is lagging is just a coordinated attack on your sanity. Your monitoring should tell you the service is degraded, not give you a fragmented puzzle to solve.
• Demand context in the payload. A notification that says “Error 500” is useless. A notification that says “Error 500 in Checkout Service; correlated with a spike in DB connection latency” is a starting point. If I have to hunt for the dashboard link, the alert has already failed.
• Audit your “resolved” notifications. If your team is seeing a constant stream of “Issue Resolved” messages without ever having been paged for the “Issue Detected” phase, your thresholds are tuned to the wrong frequency. You aren’t monitoring; you’re just watching a heartbeat.

The Summary: Stop Tuning for Perfection

If your alert requires a human to manually correlate three different dashboards to confirm a problem, it isn’t an alert; it’s a scavenger hunt.

Stop alerting on “CPU at 90%” or “Disk usage high.” These are causes, not symptoms. Unless a user can’t complete a transaction, keep the noise out of your inbox.

An alert that doesn’t come with a clear, actionable next step is just a way to ensure your team develops a Pavlovian response to ignoring notifications.

## The False Comfort of Green Dashboards

A dashboard that shows 100% uptime while your transaction latency is climbing into the stratosphere isn’t a monitoring tool; it’s a piece of fiction. If your alerts don’t force you to stop what you’re doing and actually fix something, they aren’t signals—they’re just digital background noise.

Pernille Askeland

Stop Tuning the Noise, Start Tuning the Signal

At the end of the day, effective monitoring isn’t about how many metrics you can scrape from a container or how many colors you can use in a Grafana dashboard. It’s about the distinction between a system that is merely “running” and a system that is actually delivering value. If you find yourself ignoring alerts because they’ve become part of the background hum, you haven’t built a monitoring system; you’ve built a distraction engine. Stop chasing every minor spike in CPU usage and start focusing on the symptoms that actually impact your users. If an alert doesn’t require a human to take a specific, decisive action, it shouldn’t be hitting your pager.

I’ve spent two decades fixing things that broke in ways no one saw coming, and I can tell you that the most dangerous state isn’t a system that is crashing—it’s a system that is silently failing. Good monitoring is an act of respect for your future self and your teammates. When you choose tools and thresholds that prioritize signal over noise, you aren’t just optimizing code; you are protecting your sanity. Build a system that tells you exactly when the world is on fire, so that when it finally does, you actually have the clarity to put it out.

Frequently Asked Questions

How do I actually distinguish between a symptom that matters and a cause that's just noise without drowning in more metrics?

Stop looking at the CPU usage. It’s a liar. High CPU might just be a background cron job doing exactly what it was told to do, which isn’t a crisis. Instead, look at the user’s experience: are the requests actually failing? Are the latencies spiking? A cause is a technical detail; a symptom is a broken promise to your customer. If the metric doesn’t translate to “the user can’t pay us,” it’s just noise.

If we move away from cause-based alerting, how do we prevent the team from ignoring "symptom" alerts that turn out to be false positives?

The fear is valid: if you alert on symptoms, you risk drowning in “false” positives that aren’t actually affecting users. The fix is strict SLO-based filtering. Don’t alert because “latency is up”; alert because “the 99th percentile latency is eating our error budget.” If a symptom doesn’t threaten your service level objective, it’s a metric to observe, not a page to wake someone up for. If it doesn’t hurt the customer, it’s just noise.

At what point does a monitoring setup become "too much," and how do I know we've crossed the line from visibility into pure overhead?

You’ve crossed the line when you stop investigating alerts and start playing whack-a-mole. If your team spends more time acknowledging “informational” notifications than actually fixing regressions, you aren’t monitoring; you’re just managing noise. When the cost of maintaining the dashboard exceeds the value of the data it provides, your visibility has become technical debt. If you can’t distinguish a critical outage from a minor latency spike in ten seconds, you have too much.