Most management handbooks treat handling an on call rotation like a resource allocation problem in a clean, simulated environment—as if you’re just moving blocks on a spreadsheet. They talk about “coverage windows” and “optimal staffing levels” with a level of abstraction that makes me want to scream. They never mention the specific, hollow feeling in your stomach when you realize the person on call is a junior dev who has never seen this particular database deadlock, or the way a “seamless handoff” actually feels like a game of telephone played during a total system meltdown.
I’m not here to give you a template for a scheduling tool or a list of polite Slack etiquette tips. Instead, I’m going to show you what happens when the documentation is wrong, the telemetry is lying to you, and the person who wrote the original service has been at a different company for five years. I want to talk about the uncomfortable reality of maintaining sanity when the pager starts screaming, focusing on the technical debt and human fatigue that tutorials always seem to ignore.
Table of Contents
When Incident Response Protocols Turn Into Chaos
Most incident response protocols are written in a vacuum of perfect conditions. They assume that when a service fails, everyone is sitting at a clean desk with a fresh cup of coffee, ready to execute a flawless sequence of commands. But in reality, when a critical payment gateway starts dropping transactions at midnight, the documentation usually feels like it was written by someone who has never actually been awake at that hour. You find yourself staring at a runbook that says “check the logs,” only to realize the logging service itself is what died.
This is where the gap between theory and reality becomes a liability. Without effective escalation workflows that account for human error and cognitive load, the process quickly devolves into a shouting match in a Slack channel. People start pinging the wrong stakeholders, the “primary” engineer is stuck in a loop of useless alerts, and suddenly, you aren’t solving a technical problem anymore—you’re managing a crisis of communication. If your protocol doesn’t account for the friction of a real outage, it isn’t a protocol; it’s just a suggestion that fails exactly when you need it most.
The Hidden Cost of Neglecting Engineer Sleep Hygiene
We tend to treat sleep like a variable we can just tune later, but in backend engineering, it’s a hard dependency. I’ve seen it happen too many times: an engineer gets paged at 2:00 AM for a minor database latency spike, spends three hours chasing a ghost in the connection pool, and then tries to push a hotfix at 7:00 AM. By the time they sit down for their actual shift, they aren’t thinking logically; they are just guessing. When you are operating on four hours of interrupted sleep, your ability to parse a stack trace evaporates, and that is exactly when you turn a minor hiccup into a systemic outage.
This isn’t just about being tired; it’s about the long-term erosion of technical judgment. If your team lacks effective escalation workflows that account for human exhaustion, you aren’t just risking a bug—you are risking the person. We talk a lot about on call burnout prevention in terms of frequency and rotation length, but we rarely discuss the physiological reality of the “recovery day.” If a developer is expected to jump straight from a midnight incident into a high-stakes sprint planning session, you have already lost the battle against technical debt.
The Survival Kit: Practical Realities for the Person on the Pager
- Stop treating “on-call” as a side task. If you are on rotation, your sprint capacity needs to drop by at least 30% immediately. You cannot expect someone to ship high-quality feature code while they are simultaneously triaging a database deadlock in a production environment.
- Audit your runbooks for “The Happy Path” fallacy. A runbook that only tells you how to restart a service is useless. I need the version that tells me what to do when the restart fails, the logs are silent, and the disk space is mysteriously full. If the runbook doesn’t include the “if this fails, do that” logic, it’s just a list of wishes.
- Implement a “No Blame, Just Data” post-mortem culture. When a deployment breaks the payment gateway at midnight, we don’t need a lecture on why the tests missed it; we need a precise timeline of the telemetry that failed us. We fix the visibility, not the person.
- Build an automated “noise filter.” If your pager goes off for a non-critical warning that doesn’t require human intervention, your alerting system is broken. Every false positive is a tiny withdrawal from your mental bank account, and eventually, you’ll go bankrupt and start ignoring the real emergencies.
- Hand over the baton properly. An on-call handoff isn’t just a Slack message saying “all good.” It’s a brief, technical summary of the ghosts in the machine—the weird latency spikes or the flaky service that behaved oddly during the shift. Don’t leave your successor to fight a war you already saw starting.
The Hard Truths of Staying Sane
Documentation is useless if it only describes the “happy path”; your incident response plan needs to be a survival guide for when the database is locked and the logs are silent.
Sleep isn’t a luxury you earn after the fix; it’s a technical requirement for maintaining a production system without introducing more bugs.
If your rotation relies on heroics rather than repeatable processes, you haven’t built a system—you’ve built a debt that will eventually come due.
## The Fallacy of the 'Seamless' Handover
“A handover isn’t a successful exchange of information if it only covers the metrics that stayed green; if you aren’t passing along the weird, edge-case latency spikes that almost triggered an alert at 4:00 PM, you aren’t handing over a rotation—you’re just handing over a ticking time bomb.”
Pernille Askeland
The Reality of the Rotation
At the end of the day, a successful on-call rotation isn’t measured by the absence of incidents, but by how your team handles the inevitable wreckage. We’ve talked about the chaos of poorly defined protocols and the physical toll of sleep deprivation, but the core issue is usually a lack of systemic empathy. You cannot patch a broken culture with a better pager settings configuration. If your documentation is a series of “happy path” assumptions and your engineers are running on caffeine and spite, you aren’t managing a rotation; you are managing a slow-motion burnout.
I’ve spent two decades staring at logs while the world slept, and if there is one thing I’ve learned, it’s that the most resilient systems are built by people who are allowed to be human. Stop treating your engineers like high-availability servers that never need a reboot. Instead, focus on building a culture where the post-mortem is a tool for growth rather than a search for a scapegoat. When we prioritize the stability of the person as much as the stability of the database, the code—and the team—finally starts to hold together.
Frequently Asked Questions
How do you actually push back against management when the "on-call" rotation is secretly just everyone being on call all the time?
You can’t argue with “vibes,” so stop using them. Management speaks in metrics and risk. If everyone is on call, your MTTR (Mean Time To Recovery) is likely tanking because nobody is actually rested enough to diagnose a race condition. I bring the data: I map out the “shadow on-call” hours—the Slack pings at 8 PM, the midnight context switches—and show how they correlate with the bugs we’re seeing in production. Turn the chaos into a cost analysis.
What’s the best way to document a fix for a legacy system so the next person doesn't spend four hours retracing my steps?
Don’t just document the fix; document the “why” behind the weirdness. If you patched a race condition in a twenty-year-old transaction module, don’t just say `fixed deadlock`. Tell me which specific lock contention caused it and, more importantly, why the “obvious” fix would have broken the downstream ledger. I write my commit messages before I touch the code because if I can’t explain the failure path clearly, I haven’t actually understood the bug yet.
How do you handle the handoff when the person coming on shift is inheriting a mess of unresolved alerts and half-baked documentation?
The “handoff” is usually a lie we tell ourselves to feel organized. In reality, it’s often just one exhausted engineer dumping a pile of unresolved alerts onto a fresh victim. If you’re inheriting a mess, stop trying to fix the bugs immediately. First, stabilize the documentation. If the alert is firing and the runbook is a single sentence of outdated nonsense, your priority isn’t the code—it’s documenting the failure so the next person doesn’t suffer.
