I once spent three hours on a Sunday morning staring at a dashboard that was glowing a perfect, serene green, while our transaction processing service was actually screaming in silence. The service was up, the container was running, and the endpoint was returning a 200 OK like everything was fine, but the database connection pool had quietly evaporated. That is the fundamental lie of the superficial ping; if you aren’t writing a health check that means something, you aren’t actually monitoring your system—you’re just monitoring its ability to lie to you.
I’m not here to give you a tutorial on how to return a static JSON object. I’ve spent twenty years maintaining legacy payment systems where “mostly working” is a recipe for a catastrophic outage, so I know exactly where the edge cases hide. In this post, I’m going to show you how to build checks that actually test the plumbing—from verifying downstream dependencies to checking resource exhaustion—so that when your system starts to choke, your alerts tell you why, not just that something is wrong.
Table of Contents
Why Your Shallow Health Checks Are Lying to You
Most developers treat a health check like a pulse check on a patient who is technically breathing but clearly unconscious. They implement a simple endpoint that returns a 200 OK as long as the web server is running. This is a shallow health check, and it is dangerous. In a microservices architecture, this creates a false sense of security. Your orchestrator thinks the service is fine, so it keeps routing traffic to it, even if the service can no longer reach its primary database or a critical downstream API.
This is where the distinction between meaningful liveness probes vs readiness probes becomes a matter of survival. A liveness probe should only tell the system if the process needs a restart. If you bake heavy dependency checking into your liveness probe, you risk a death spiral: a momentary network hiccup causes all your pods to fail their liveness checks, triggering a simultaneous restart across the entire cluster. Instead, use your readiness probes to signal when the service is actually capable of doing work. If you don’t differentiate the two, you aren’t monitoring your system; you’re just watching it lie to you until it finally crashes.
Meaningful Liveness Probes vs Readiness Probes in Production
The mistake I see most often in Kubernetes manifests is treating liveness and readiness as synonyms. They aren’t. If you point a liveness probe at a downstream database and that database hiccups, Kubernetes will see the failure and restart your pod. You haven’t fixed the database; you’ve just added a revolving door of crashing containers to your cluster, making the outage even harder to debug. This is how you accidentally trigger a death spiral.
Instead, you have to distinguish between “am I alive?” and “can I actually do work?” A liveness probe should be incredibly minimal—just enough to prove the process hasn’t deadlocked. For the heavy lifting, you use readiness probes. This is where dependency checking in health endpoints becomes vital. If your connection pool is exhausted or a required cache is unreachable, the readiness probe should fail. This pulls the pod out of the load balancer rotation without killing the process, effectively preventing cascading failures while the system tries to recover. It’s the difference between a controlled slowdown and a total meltdown.
Five ways to stop lying to your monitoring stack
- Don’t just check if the process is running; check if it can actually do its job. A service that can’t reach its primary database is just a very expensive way to return a 500 error.
- Implement dependency timeouts. If your health check waits indefinitely for a hanging downstream service, your health check itself becomes the bottleneck that kills your entire orchestration layer.
- Distinguish between transient and terminal failures. One failed socket connection shouldn’t trigger a container restart, but five consecutive failures should.
- Avoid the “circular dependency” trap. If your health check requires a service that is currently being checked by your health check, you’ve built a deadlock that no amount of YAML will fix.
- Log the ‘why’ behind the failure. A generic “unhealthy” status is a waste of my time; I need to see the specific connection pool exhaustion or the disk space alert in the logs the second the probe fails.
The Hard Lessons
If your health check doesn’t verify the downstream dependencies that actually matter to your business logic, you aren’t monitoring your service—you’re just monitoring whether the process is still breathing.
Stop treating liveness and readiness as synonyms; a service that is alive but can’t reach its database shouldn’t be killed by the orchestrator, it should just be pulled out of the load balancer rotation.
Design your probes with the expectation of failure, because a health check that only reports the “happy path” is just a way to delay the inevitable realization that your system is already failing.
## The Cost of a False Positive
A health check that only confirms the process is running is just a way to lie to your monitoring tools; I don’t care if the service is alive if it can’t actually reach the database or clear its own buffer.
Pernille Askeland
Stop Guessing, Start Measuring
At the end of the day, a health check isn’t a checkbox for your CI/CD pipeline; it’s a diagnostic tool for when you’re staring at a dashboard at 3:00 AM. If your probes only verify that the process is running, you aren’t monitoring your system—you’re just confirming its existence. You need to differentiate between a container that is simply alive and one that is actually capable of doing its job. By separating liveness from readiness and actually testing the critical paths—like your database connection pools and downstream dependencies—you move from reactive firefighting to predictable system management.
I spent years fixing legacy payment gateways where a “healthy” status was actually a mask for a silent, cascading failure. Don’t let your infrastructure lie to you. Build your checks with the assumption that things will break, and design them to tell you exactly how they are failing. When you write code that accounts for the edge cases and the connection timeouts, you aren’t just being thorough; you are building resilience into the foundation. Stop chasing the happy path and start preparing for the reality of the error.
Frequently Asked Questions
How do I prevent a deep health check from causing a cascading failure by overloading a struggling database with probe requests?
The trick is to stop treating your health check like a real user request. If your database is struggling, the last thing it needs is a fleet of containers hitting it every five seconds just to say “I’m alive.” I use a local cache for the deep check results. Let one background thread perform the heavy lifting and update a timestamp; the probe just reads that value. If the thread fails, the probe reports the failure.
At what point does a health check transition from being a diagnostic tool to becoming a source of noise in our alerting stack?
It happens the moment you start alerting on the symptom rather than the cause. If your health check fails because of a transient network blip or a single slow query, and your pager goes off, you haven’t built a diagnostic tool—you’ve built a noise machine. A health check should signal a state change that requires human intervention, not just a momentary hiccup that a retry policy could have handled.
How should we handle partial failures, like when a non-critical third-party API is down but the core transaction flow is still functional?
You have to categorize your dependencies before you write a single line of probe logic. If the third-party API is just for generating a loyalty points receipt, don’t let its failure trip your readiness probe. If you do, you’re turning a minor inconvenience into a self-inflicted outage. I treat non-critical failures as “degraded” states. The service stays up, the core transaction flows, but your telemetry should scream that a sub-system is offline.
