The Disk Says It's Full. df Says It's Not.

It always happens at the worst possible moment. A deploy that ran fine yesterday hangs halfway through. The application starts throwing 500s. You SSH into the box, and there it is, sitting in the log like a shrug:

write error: No space left on device

So you run df -h, expecting to see a partition pinned at 100%. Instead you see 40% free. Plenty of room. The disk is not full, but the kernel insists it is.

A "full" disk on Linux has at least three different meanings, and only one of them is the obvious one.

This is one of those problems where the error message is technically honest and completely misleading at the same time. The good news is that the diagnosis takes about ninety seconds once you know the three things to check, and the fix is usually a single command. Here's the whole decision tree.

Start with the obvious case: the disk really is full

Before chasing exotic causes, rule out the boring one. Run df -h and read it carefully — the trap is looking at the wrong row.

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        79G   31G   48G  40% /
/dev/sda15      105M  6.1M   99M   6% /boot/efi
tmpfs           1.6G  1.6G     0 100% /run
overlay          20G   20G     0 100% /var/lib/docker/overlay2/9f3c...

The root filesystem has 48 GB free, which is what you saw the first time. But /run and an overlay mount are both at 100%. If your process is writing into one of those, it gets ENOSPC no matter how empty / looks. The error is per-filesystem, not per-machine.

So step one is figuring out which filesystem the failing write actually lands on. If you know the path, df -h /path/to/file tells you directly:

df -h /var/log/myapp/

Separate mounts for /var, /tmp, /home, and container overlays are extremely common on managed servers, and they're the single most frequent reason a "40% free" disk throws a full-disk error. Check the specific path before you check anything else.

If the right filesystem really is at 100%, you just need to find the weight. du is the workhorse here, but plain du -sh /* on a large server is slow and noisy. This narrows it faster:

sudo du -h --max-depth=1 / 2>/dev/null | sort -hr | head -15

Run it, pick the biggest directory, run it again one level deeper, repeat. Three or four rounds and you'll be standing on top of the culprit — usually a log file, a package cache, or an old backup somebody forgot about. If ncdu is installed (sudo apt install ncdu), sudo ncdu / gives you the same thing as a browsable interface, which is nicer at 2 a.m.

The disk has space but no inodes left

Every file, directory, and symlink on an ext4 filesystem consumes one inode — a small fixed-size record holding the file's metadata. The filesystem allocates a finite number of inodes when it's created, and that number doesn't grow. It's entirely possible to run out of inodes while 90% of the raw capacity sits unused.

Check with -i:

df -i

Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/root      10485760 10485760       0  100% /

IUse% at 100% with Avail still healthy is the signature. What causes it is always the same story: something is creating an enormous number of tiny files. A session-file directory that never gets garbage collected. A mail queue. A cache that writes one file per key and never expires. A crash loop dumping one file per restart, every second, for a week.

Finding the directory is the annoying part, because du measures bytes and you need counts. This counts files per top-level directory:

for d in /var/*/; do echo -n "$d "; sudo find "$d" -xdev -type f 2>/dev/null | wc -l; done | sort -k2 -nr | head

Once you've found it, delete carefully. A directory with two million files will choke rm * with an argument-list-too-long error, so use find with -delete and scope it by age:

# Preview first — always
sudo find /var/lib/myapp/sessions -type f -mtime +7 | head

# Then delete
sudo find /var/lib/myapp/sessions -type f -mtime +7 -delete

Deleting millions of small files is I/O-heavy and will take a while. Run it in a screen or tmux session so a dropped connection doesn't kill it halfway.

Worth knowing: XFS and Btrfs allocate inodes dynamically, so this failure mode is mostly an ext4 story. If you keep hitting it on a workload that's inherently file-count-heavy, the filesystem choice is part of the answer, not just the cleanup script.

The classic: deleted files that processes are still holding open

This is the one that produces the most confusion, and it's the most common cause of "I deleted 8 GB of logs and nothing changed."

On Linux, deleting a file removes its directory entry, not the data. The data lives until the last open file descriptor pointing to it is closed. If a running process had that file open when you deleted it, the process keeps writing into a file that no longer has a name — and the kernel keeps every byte of it allocated on disk. du can't see it, because du walks directory entries. df can, because df asks the filesystem. Hence the mismatch: df says 100% full, du adds up to far less.

The typical setup: an application writes to /var/log/app.log with no rotation, the file hits 30 GB, someone runs rm /var/log/app.log to reclaim space, the app never notices and keeps writing to the same descriptor. Nothing is freed, and now you can't even read the log.

lsof finds them. The +L1 flag lists open files whose link count is below 1 — meaning deleted but still held:

sudo lsof +L1

COMMAND   PID  USER   FD   TYPE DEVICE  SIZE/OFF NLINK   NODE NAME
nginx    1421  root    5w   REG  259,1 31984128     0 262541 /var/log/nginx/access.log (deleted)
python3  2087 appusr   3w   REG  259,1 8402931712    0 262998 /var/log/myapp/app.log (deleted)

There's your missing 8 GB, sitting in a file with NLINK 0. The SIZE/OFF column is the space you'll get back.

The clean fix is to make the process release the descriptor. Restarting the service does it:

sudo systemctl restart myapp

If you can't restart — the process is mid-job, or it's something you'd rather not bounce during business hours — you can truncate the file through its /proc handle. The descriptor stays open, but the data is released immediately:

# Using PID 2087 and FD 3 from the lsof output above
sudo truncate -s 0 /proc/2087/fd/3

Space comes back instantly. Do this only when you genuinely don't need the contents, because they're gone the moment you press enter. And treat it as a stopgap: the real fix is configuring logrotate with copytruncate (or making the app respond to SIGHUP and reopen its logs) so nobody has to rm a live log file again.

When it isn't the disk at all

A few cases produce the same message without any disk being involved, and they're worth recognizing so you don't spend an hour on the wrong problem.

Inotify watch limits. Development servers, file watchers, and anything running a bundler can exhaust the per-user inotify limit, and the kernel reports it as ENOSPC. If your error came from a file watcher rather than a write, check it:

cat /proc/sys/fs/inotify/max_user_watches

Raising it persistently:

echo "fs.inotify.max_user_watches=524288" | sudo tee /etc/sysctl.d/99-inotify.conf
sudo sysctl --system

Reserved blocks. ext4 reserves 5% of the filesystem for root by default. A non-root process starts failing at 95% while df shows 5% available. On a 500 GB data volume that's 25 GB held back for no good reason, and you can lower it:

sudo tune2fs -m 1 /dev/sda1

Docker. Containers have their own overlay filesystems and their own accumulation problem — dangling images, stopped containers, unused volumes, build cache. docker system df shows the breakdown, and docker system prune -a --volumes clears it. Read what that command is about to delete before you run it; --volumes removes unused volumes, which may contain data you wanted.

Make it not happen again

The fix took two minutes. Preventing the 3 a.m. repeat takes about twenty.

That last row is the one people skip. Almost every disk-space monitor watches bytes and ignores inodes, which means the inode failure mode arrives with zero warning every single time. Adding a second check costs one line in your monitoring config.

The ninety-second version

Next time you see it, work in this order:

1. df -h /the/actual/path — is it a different mount than you assumed?
2. df -i — inodes at 100%?
3. sudo lsof +L1 — deleted files still held open?
4. If none of those, is it inotify watches, reserved blocks, or Docker?

One of those four answers it nearly every time. The error message is vague on purpose — ENOSPC is a single return code covering several distinct resource limits — but the diagnosis isn't. You just have to ask the filesystem the right question.

And when you find it, take the extra twenty minutes for the prevention step. Future-you, woken by a pager, will be grateful.