The short answer: Because the filesystem ran out of inodes, not bytes. Every file needs one inode, the table is fixed at mkfs time on ext4, and when it fills the kernel returns the same ENOSPC error as a full disk. Run df -i to confirm: IUse% will be 100.
A runaway retry loop filled a mail spool with 65,000 tiny files. Total data written: about 2 megabytes on a 1 gigabyte volume. The volume died anyway, because every file needs an inode and the inode table filled while the disk stayed 99% empty. This replay shows the crash happening live, then the diagnosis and the fix.
The box is mail-01, a small mail relay with a 1 GB spool volume. A retry job has a bug: it re-queues the same messages forever and never cleans up after itself. Each retry is a tiny file, a few dozen bytes.
At the start of the replay the volume is healthy: 65,536 inodes, 15 in use. Then the loop starts. Under a running watch you can see IUse% climb to 100% in about 15 seconds. From that moment, every write on the volume fails.
The confusing part, and the reason this incident wastes hours in real shops, is the error text. The kernel says "No space left on device" while df -h reports the disk 1% used with 905M available. Both are true. They are talking about different budgets.
An ext4 filesystem has two separate budgets. The first is space: data blocks, 4,096 bytes each on this volume, the thing df -h shows. The second is inodes: the fixed-size table of file records. Every file, directory, and symlink consumes exactly one inode no matter how small the file is.
The inode count is decided when the filesystem is created. mkfs.ext4 gives a 1 GB volume 65,536 inodes by default, one per 16 KB of disk. That ratio assumes average files around 16 KB. Fill the volume with near-empty files and the inode table runs out while the data blocks sit idle.
When either budget hits 100%, writes fail with the same errno, ENOSPC, which programs print as "No space left on device". The error does not tell you which budget died. That is why the second command in any disk-full incident should be df -i.
df -h . shows the space budget: 974M size, 2.2M used, 1%. Healthy. df -i . shows the inode budget: 65536 total, 65536 used, 0 free, 100%. There is the corpse.
du --inodes -d 1 . counts directory entries instead of bytes and points straight at the culprit directory: 65,525 of the 65,527 inodes are under ./queue. ls queue | wc -l confirms the scale: 65,524 files. du -sh queue shows they hold about 2 megabytes. Size was never the problem.
root@mail-01:/var/spool/mailq# df -h . Filesystem Size Used Avail Use% Mounted on /dev/loop0 974M 2.2M 905M 1% /var/spool/mailq root@mail-01:/var/spool/mailq# df -i . Filesystem Inodes IUsed IFree IUse% Mounted on /dev/loop0 65536 65536 0 100% /var/spool/mailq
df -i /path: Inode budget: total, used, free, IUse%df -h /path: Space budget. Always read both in a disk-full incidentdu --inodes -d 1 /path | sort -rn: Which directory is eating the inodesfind DIR -name 'PATTERN' -delete: Mass delete that never hits the argv limittune2fs -l /dev/DEV | grep "Inode count": The fixed inode budget ext4 chose at mkfs timemkfs.ext4 -i 4096: Build future volumes with 4x inode densityWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.