I remember sitting in a cold server room ten years ago, staring at a terminal while the hum of the cooling fans felt like it was vibrating inside my skull. I had just realized that a critical payment reconciliation task hadn’t just failed—it had executed concurrently, creating a nightmare of duplicate entries that no amount of “undoing” could easily fix. Most tutorials treat debugging a job that runs twice as a theoretical exercise in idempotency, but they never mention the sinking feeling in your gut when you realize the production database is currently being torn apart by a race condition you didn’t account for.
I’m not here to walk you through a sanitized, happy-path architecture diagram. Instead, I want to show you what actually happens when the scheduler loses its mind and your logic fails to protect the state. I’ll share the specific, messy patterns I’ve encountered while maintaining legacy systems, focusing on the actual traces left in the logs and the database locks that tell the truth. We aren’t going to talk about perfection; we are going to talk about how to find the ghost in your machine and stop it before it doubles your costs.
Table of Contents
Chasing Race Conditions in Scheduled Tasks
When I first encountered this, I assumed it was a simple configuration error in the cron trigger. But as I dug into the telemetry, the pattern looked less like a misconfiguration and more like a classic case of race conditions in scheduled tasks. In a distributed environment, you don’t just have one instance of your service; you have several, all waking up at the same millisecond, all seeing that the “last run” timestamp is still in the past. They all decide, simultaneously, that it is their turn to work.
The real headache starts when you realize that even with a database check, the timing is too tight. If two threads read the state before either has committed the update, they both proceed. You can try to tighten your database transaction isolation levels, but that often just trades one set of locks for another, potentially stalling your entire pipeline. I’ve spent many late nights realizing that the “safety” we build into our deployment orchestration is often the very thing that triggers the duplication. It isn’t a failure of the code logic itself, but a failure to account for the overlapping windows of execution inherent in distributed systems.
When Message Queue Retry Logic Goes Rogue
If the scheduler isn’t the culprit, you have to look at the transport layer. Most tutorials treat message queues like magic pipes that deliver a message once and then disappear. In reality, most systems are built on at-least-once delivery semantics. This means if your consumer takes a few milliseconds too long to acknowledge a task—perhaps due to a momentary spike in database latency—the queue assumes the worker died and sends the message to someone else.
This is where you see the “ghost” execution. You’ll find two different workers processing the same payload simultaneously, and if your logic isn’t airtight, they’ll both commit changes to the database. I’ve spent far too many late nights realizing that the fix wasn’t tweaking the retry intervals, but implementing a proper idempotent consumer pattern. You can’t rely on the infrastructure to be perfect; you have to write your code assuming that the same instruction will arrive twice, and ensure the second attempt does absolutely nothing.
Survival tactics for when the logs lie to you
- Stop looking at the application logs and start looking at the database locks. If your job is running twice, it’s often because the first instance is hanging on a row lock, causing the scheduler to think the previous run timed out and needs a replacement.
- Audit your idempotency keys. A job that runs twice is only a disaster if it isn’t idempotent; if you aren’t checking a unique transaction ID or a state flag before executing the core logic, you aren’t writing production code, you’re writing a landmine.
- Check your visibility timeouts. If you’re using a message queue, the most common culprit is a job that takes longer to process than the queue’s visibility timeout, causing the broker to assume the worker died and hand the same message to a second worker.
- Verify your clock synchronization. In a distributed environment, even a few seconds of drift between nodes can make a cron-based scheduler think it’s time to trigger a new cycle before the old one has even registered as ‘active’ in your state store.
- Trace the trigger, not just the execution. Don’t just ask why the code ran; ask what signaled it. I’ve spent hours debugging “duplicate” jobs only to find a legacy monitoring script was firing an API webhook every time it saw a slightly elevated CPU metric.
Lessons from the wreckage
Idempotency isn’t a luxury; it’s the only way to sleep at night when you can’t trust your infrastructure to run a task exactly once.
Stop assuming a “failed” job actually stopped; always check if a partial execution left your database in an inconsistent state before you trigger a retry.
If your architecture relies on a single trigger to maintain integrity, you haven’t built a system—you’ve built a countdown to a race condition.
The Cost of the Happy Path
Tutorials always show you the job succeeding on the first try, but they never show you the moment you realize your idempotency logic failed and you’re staring at a database full of duplicate transactions that shouldn’t exist.
Pernille Askeland
Beyond the Double Execution
At the end of the day, a job running twice isn’t just a nuisance; it is a symptom of a deeper architectural mismatch. Whether you are wrestling with a race condition that allows two workers to grab the same task, or a message queue that is being a bit too aggressive with its retry logic, the cause is rarely a single “broken” line of code. It is usually a gap between how you assume the system behaves and how it actually handles failure. You have to stop looking for a syntax error and start looking for the silent assumptions in your idempotency logic. If your database can’t tell the difference between a legitimate second attempt and a stray duplicate, you haven’t finished the implementation.
Maintenance isn’t about the thrill of the initial build; it’s about the quiet discipline of hardening the edges. Most tutorials will show you how to trigger a function, but they won’t show you how to build a system that survives its own unpredictable failures. As you move forward, try to write your code with the suspicion that the network will fail, the scheduler will glitch, and the message will arrive twice. If you can build for the broken path, you won’t just be a developer who writes features—you’ll be the one who builds systems that actually stay running.
Frequently Asked Questions
How do I tell if the duplicate execution is coming from the scheduler itself or if it's a side effect of a database transaction that didn't commit in time?
To figure this out, I look at the timestamps in the logs versus the database transaction logs. If the scheduler triggers twice with distinct IDs, the scheduler is the culprit. But if you see one execution start, a long gap, and then a second one starting while the first is still “active” in the DB, you’re likely looking at a transaction timeout. The scheduler thinks the first one died because the DB didn’t commit fast enough, so it sends another.
If I implement a distributed lock to prevent this, what happens to the job if the process holding the lock crashes halfway through?
That’s the classic trade-off. If the process crashes while holding the lock, you’ve just traded a duplicate execution for a dead job. Without a TTL (Time To Live) on that lock, the key stays in Redis or your database forever, and the scheduler will never touch that task again. You have to implement an expiration, but then you’re back to square one: if the job takes longer than the TTL, the lock expires and the ghost returns.
Is there a way to trace these duplicates back to a specific request ID or correlation ID when the logs look like two completely independent events?
That’s the hardest part. When the logs look like two distinct lives, you can’t rely on a single ID. I usually look for “fingerprints” in the metadata. If you aren’t already, you need to inject a correlation ID at the very edge of your system—the gateway or the ingestion point—and pass it through every layer. If the ID is missing in one of the duplicates, you’ve found your culprit: a retry triggered by a timeout.
