It’s 3:00 AM, the Slack notifications are screaming, and you’re staring at a terminal where every command returns `No space left on device`. There is a specific, hollow feeling in your stomach when you realize the database is stalling because it can’t even write a single line to the error log. Most tutorials treat debugging a disk that filled up as a simple matter of running `rm -rf` on some old temp files, but they never mention the quiet panic of watching a production service choke to death because a hidden log rotation failed or a stray process is swallowing inodes like they’re free.
I’m not here to give you a sanitized checklist of “best practices” that only work when everything is going according to plan. I’ve spent twenty years maintaining legacy payment systems, and I know that the real work starts when the “happy path” disappears. In this post, I’m going to show you exactly how I trace the leak, from the initial `df -h` realization to finding that one rogue directory that nobody thought to monitor. We aren’t just clearing space; we are finding the root cause so you can actually sleep through the night.
Table of Contents
Hunting the Culprits Identifying Large Files and Directories
Once you’ve accepted that the drive is actually full, you have to stop guessing. I’ve spent too many late nights staring at a terminal, hoping a random service would just stop writing to `/var/log` on its own. You can’t do that. The first step in identifying large files and directories is to stop looking at the symptoms and start looking at the tree. I usually start with `du -ah / | sort -rh | head -n 20`. It’s a blunt instrument, but it cuts through the noise. It forces the filesystem to show you exactly where the weight is accumulating, whether it’s a runaway database WAL file or a rogue backup script that forgot how to rotate.
If you’re working on a production box, you’ll want to lean on more surgical linux disk usage analysis tools like `ncdu`. It’s much more intuitive for navigating deep, nested directory structures without getting lost in a sea of text. I prefer it because it lets me see the scale of the problem visually; seeing a single directory swell to 50GB in a matter of minutes tells a very different story than a slow, steady leak. You aren’t just looking for big files; you’re looking for the pattern of the growth.
The Toolbelt Linux Disk Usage Analysis Tools That Actually Work
I don’t bother with fancy GUI-based disk analyzers when I’m on a production server. They are too heavy, and frankly, they add another layer of abstraction I don’t need when a database is failing to write its WAL logs. Instead, I rely on the standard linux disk usage analysis tools that have been around since before I started my first job. `du` is my primary instrument, specifically `du -ah / | sort -rh | head -n 20`. It’s blunt, but it gets the job done by forcing the filesystem to reveal exactly where the bloat is hiding.
If I need to navigate the directory tree more visually without leaving the terminal, `ncdu` is my go-to. It’s much more intuitive for identifying large files and directories in real-time, allowing me to drill down into `/var/log` or `/tmp` with a few keystrokes. It’s a matter of efficiency; when the system is lagging because it’s struggling to manage metadata on a saturated partition, I can’t afford to waste time running heavy scripts. I just need to see the numbers and start deleting.
The "Don't Just Delete and Pray" Checklist
- Check your deleted files with `lsof +L1`. I’ve lost count of how many times I’ve cleared out a massive log file only to see the disk space stay at 100% because a running process still has a handle on it. You haven’t actually freed anything until the process is killed or restarted.
- Look for the “hidden” bloat in `/var/log`. It’s rarely the application code itself; it’s almost always a service that decided to start logging every single debug event to a file that nobody is rotating. If you see a 50GB `.log` file, don’t just `rm` it—find out why the rotation policy failed.
- Inspect your Docker volumes and unused images. If you’re running a containerized environment, `docker system prune` is your best friend, but be careful. I once spent three hours chasing a “disk full” error only to realize a defunct CI/CD runner was leaving orphaned volumes behind like digital trash.
- Don’t ignore the inode exhaustion. You can run `df -h` and see plenty of GBs left, but if you’ve got millions of tiny session files or cache entries, `df -i` will show you that you’re actually out of inodes. At that point, the filesystem is functionally dead, even if it’s technically “empty.”
- Check your database transaction logs. In my experience with payment systems, a stalled replication or a massive uncommitted transaction can cause the WAL (Write Ahead Log) to swell until it chokes the entire partition. If the DB is the culprit, deleting random files in `/tmp` won’t save you; you have to fix the state of the engine.
Lessons from the Trenches
Don’t trust the logs blindly; when a disk hits 100%, the very tools you need to diagnose the problem often fail to write new entries, leaving you blind exactly when you need sight.
Automation is a double-edged sword; the same cron job designed to clean up temp files can become the culprit if it fails silently or starts generating massive error logs of its own.
Prevention isn’t about more storage; it’s about implementing proactive monitoring and alerting on inode exhaustion, because a disk can be “full” even when `df -h` says you still have gigabytes left.
The Lesson in the Logs
A tutorial tells you how to scale a database; it never tells you how it feels to watch a production environment choke to death because a rogue log rotation script decided to take the afternoon off.
Pernille Askeland
Beyond the Immediate Fix
Once you’ve used `du` to find the rogue log file or cleared out the bloated `/var/lib/docker` volumes, the immediate panic subsides. You’ve identified the culprit, reclaimed your inodes, and the system is breathing again. But if you just delete the file and walk away, you haven’t actually solved the problem; you’ve just postponed the next outage. Whether it was a misconfigured rotation policy, a runaway debug log, or a database that outgrew its partition, the pattern will repeat. The goal isn’t just to find the large files, but to understand the underlying mechanism that allowed them to grow unchecked in the first place.
Maintenance is rarely glamorous, and it’s almost never as clean as the documentation suggests. Most of my career has been spent cleaning up the digital equivalent of a flooded basement, but there is a quiet satisfaction in turning a chaotic crash into a predictable, monitored process. Don’t just fix the disk; build the guardrails. Set up your alerts, tighten your log rotation, and write the monitoring script that catches the swell before it hits 90%. It’s the difference between being a firefighter and being an engineer.
Frequently Asked Questions
I've deleted the large log files, but `df -h` still shows the disk is full; why is the space not returning?
You’ve deleted the files, but the space is still gone. This is a classic. You likely have a process—probably a hung logging daemon or a database worker—that still holds an open file descriptor to those deleted files. The directory entry is gone, but the inode is still active because the process hasn’t let go. Run `lsof +L1` to find the ghosts. Kill the process, or restart the service, and the space will finally return.
How do I prevent a specific service from silently bloating the `/var/log` partition again next week?
Don’t just increase the partition size; that’s just delaying the inevitable. You need to force the service to play by the rules. First, check your `logrotate` configuration—ensure that specific service has its own dedicated config file with aggressive rotation and compression. If it’s still out of control, implement a hard limit via `systemd` journal settings or use a filesystem quota. If the service doesn’t respect rotation, you’ll be back here next Tuesday.
Is it safer to clear out old Docker volumes or should I be looking at database transaction logs first?
If you’re looking for a quick win to stop the bleeding, prune the Docker volumes. It’s low-risk and usually clears enough headroom to let you breathe. But don’t mistake a temporary fix for a solution. If your transaction logs are bloating, you have a fundamental configuration or architectural issue. Clean the volumes to stabilize the system, then immediately pivot to the logs. You can’t ignore a runaway WAL file just because Docker was easier to clean.
