Half a Batch Succeeded. Now What?

I remember sitting in a cold server room at 3:00 AM, watching a monitor bleed red because a single malformed JSON object in a thousand-record set had brought our entire payment settlement to a grinding halt. Most tutorials make it look like a breeze: you loop through your array, you call the API, and if it returns a 200, you move on. They never show you the unraveling that happens when record 452 fails, but records 453 through 1,000 are still trying to commit. If you’re looking for a theoretical lecture on distributed systems, you’re in the wrong place; I want to talk about the actual mechanics of how to handle partial failure in a batch without losing your mind or your data integrity.

I’m not here to sell you on a fancy new orchestration framework or a complex saga pattern that requires a PhD to debug. Instead, I’m going to show you how I approach these messes when I’m staring at a legacy codebase that’s been running since before I finished my MSc. We are going to look at idempotency, dead-letter queues, and state tracking—the unglamorous, essential tools that keep your system from becoming a graveyard of half-finished transactions.

Atomic Batch Operations When All or Nothing Isnt an Option

In most textbooks, you’re taught that a batch is an atomic unit. You wrap the whole thing in a single transaction, and if record 4,999 of 5,000 fails, the database rolls everything back. It’s clean, it’s elegant, and in a high-volume payment environment, it is often completely impractical. If you are processing a million transactions and the last one triggers a constraint violation, you don’t want to throw away the 999,999 successful ones just to maintain theoretical purity.

The reality of maintaining legacy systems is that you frequently have to abandon the dream of absolute atomic batch operations. When you move away from “all or nothing,” you inherit a new set of headaches. You can’t just rely on the database to clean up your mess anymore. Instead, you have to implement manual checkpointing in distributed systems to keep track of exactly where the world ended and the failure began. If you don’t, you’ll find yourself running the same batch twice, doubling your entries and creating a reconciliation nightmare that no amount of coffee can fix.

Transactional Integrity in Batch Jobs the Cost of Broken Promises

We like to talk about transactional integrity in batch jobs as if it’s a binary state: either the transaction commits, or it rolls back. In a perfect world, that’s how it works. But when you’re dealing with legacy payment gateways or distributed databases, the “rollback” is rarely a clean, single command. You end up in a state of limbo where the database thinks the record is processed, but the downstream service timed out. If you haven’t designed for this, you aren’t just dealing with an error; you’re dealing with data corruption that looks like success.

This is where the cost of broken promises becomes visible in your logs. When a batch fails halfway through, you can’t simply restart the entire job without risking massive duplication. I’ve spent too many nights untangling why a customer was charged twice because a retry mechanism blindly re-ran a partially successful set. To survive this, you need more than just a try-catch block; you need robust dead letter queues for batch processing to isolate the poisoned records. You have to treat every failed entry as a distinct entity that needs its own lifecycle, rather than just a nuisance that stopped the clock.

Five ways to stop a partial failure from becoming a permanent headache

• Stop treating the batch as a single unit of work. If you’re wrapping 1,000 records in one massive transaction, you aren’t being safe; you’re just making sure that one bad row kills the entire afternoon’s work. Break it down into smaller, manageable chunks so a single malformed string doesn’t roll back a thousand successful updates.
• Implement a dead-letter queue for the failures. When a record hits a wall, don’t just log an error and move on—move the actual payload into a side table or a separate queue. You need to be able to inspect the exact state of the failed data without digging through gigabytes of text logs to find where the parser choked.
• Track state at the record level, not the job level. I’ve seen too many systems where the only way to know what happened is to re-run the entire batch and hope the idempotency logic holds. You should be able to query your database and see exactly which IDs are ‘pending’, ‘completed’, or ‘failed’ before you even touch the code.
• Design for idempotency from the start. You are going to retry things. It’s not a matter of if, but when. If your batch job crashes halfway through, your recovery logic needs to be able to pick up exactly where it left off without double-charging a customer or creating duplicate entries.
• Build ‘Circuit Breakers’ into your batch logic. If your error rate hits a certain threshold—say, 5% of the first hundred rows—kill the job immediately. If the first few records are failing, it’s usually not a data issue; it’s a connection issue, a schema mismatch, or a service outage. Continuing to hammer a broken downstream system just makes the cleanup harder later.

The Reality Check

Stop designing for the happy path; your system’s true quality isn’t measured by how fast it processes a perfect batch, but by how gracefully it handles the 5% that inevitably fails.

“All or nothing” is a luxury you rarely have in high-volume payment systems, so you need to build explicit state tracking to know exactly where a batch died and which records are stuck in limbo.

Idempotency isn’t an optional feature—it’s the only thing preventing you from double-charging a customer when you inevitably have to manually re-run a partially failed job.

## The Gap Between Theory and Production

“A tutorial will show you a perfect loop where every record processes in milliseconds, but my job begins when the tenth record in a thousand-item batch throws a database deadlock and leaves the remaining nine hundred and ninety stuck in a state of half-existence.”

Pernille Askeland

Beyond the Happy Path

At the end of the day, handling partial failures isn’t about finding a single magic library; it’s about accepting that entropy is a constant in any distributed system. We’ve looked at why the “all or nothing” approach fails when your dataset grows, why transactional integrity is a heavy tax on performance, and how to build the necessary scaffolding to track exactly which records survived the night. You can’t prevent every error, but you can stop them from becoming silent data corruption. If you aren’t explicitly planning for the middle ground between success and total catastrophe, you aren’t building a system—you’re just waiting for a production incident.

I spent years thinking a “good” developer was someone who wrote clean, elegant logic that never failed. I was wrong. A real developer is the one who stays calm when the logs start screaming at 3:00 AM because they already anticipated the partial failure state. Don’t aim for the perfection you see in the documentation; aim for resilience in the wreckage. Build your error handling with the same care you give your core logic, because the code that handles the mess is ultimately what keeps the system alive.

Frequently Asked Questions

How do I actually track which specific records failed without bloating my logs into something unreadable?

Don’t dump the whole object into your logs. If you’re processing ten thousand records, a single malformed payload shouldn’t turn your log aggregator into a landfill. Instead, I use a “Failure Manifest.” I capture the unique identifier, the specific error code, and the exact field that triggered the validation failure. Keep it structured—JSON is your friend here. You want to be able to query your logs for `error_type: validation` and see the exact IDs that need a retry.

If I implement a retry mechanism for the failed subset, how do I prevent the system from processing the successful ones a second time?

You need idempotency keys. If you just retry the whole batch, you’re asking for double-billing nightmares. I’ve seen it happen: a subset fails, the retry kicks in, and suddenly the successful records from the first attempt are processed again because the system doesn’t realize they’re already done. Track the state of every individual item in your batch. Before you execute any logic, check if that specific ID is already marked as ‘completed’ in your database.

At what point does a partial failure stop being a "retryable error" and start being a signal that the entire batch needs to be killed?

You stop retrying the moment the error stops being about the data and starts being about the system. If it’s a transient network hiccup or a single malformed record, retry it. But if you start seeing 401 Unauthorized, a database connection pool exhaustion, or a sudden spike in latency across the board, kill the batch. If the environment is failing, retrying is just paying for more wreckage.