The short answer: A file was deleted while a process still had it open. du only counts files reachable by name, but the deleted file's blocks stay allocated until the last file descriptor closes, and df counts them. Run lsof +L1 to list deleted-but-open files and see which process holds the space.
An app volume on app-01 fills to 100% in under a minute while du can only find 32 kilobytes of files. The space is inside a log file that was deleted while the application still had it open: on Linux, deleting a file removes its name, not its bytes. This replay shows the disk filling live, the du versus df contradiction, the lsof +L1 diagnosis, and the space coming back the moment the holder dies.
The box is app-01, an application server with a 1 GB data volume for logs and uploads. Weeks ago somebody noticed app.log getting huge and deleted it to free space. The application never reopened its log file, so it kept writing into the deleted file through its open file descriptor.
At the start of the replay the volume is already two thirds full and climbing. Under a running df loop you watch the last few hundred megabytes disappear in seconds. When it hits 100%, writes start failing with "No space left on device".
Then the trap: du, the tool everyone reaches for, totals 32 kilobytes on a volume df swears is full. Both are correct, and the difference between them is the whole lesson.
df asks the filesystem how many blocks are allocated. du walks the directory tree and adds up the files it can reach by name. Usually those agree. They stop agreeing the moment a file is deleted while a process still holds it open.
On Linux, rm removes a name from a directory. The file itself, the inode and every allocated block, survives until the last open file descriptor closes. A deleted-but-open file is invisible to du and ls, but its blocks are still allocated, so df keeps counting them. The kernel is not leaking space. It is keeping a promise to the process that still has the file open.
That is why "I deleted the huge log but the disk is still full" is one of the most common tickets in Linux operations. The deletion worked. The space just cannot come back yet.
du -sh . says 32K. df -h . says 100%. That gap is the signature: space allocated to files that no longer have names. One command finds them.
lsof +L1 lists open files with a link count of zero, meaning deleted but still held. In the replay it shows app.log, deleted, 1,003,884,544 bytes, held open on file descriptor 3. No lsof on the box? The /proc filesystem shows the same thing: ls -l /proc/PID/fd prints every descriptor, and the deleted one is labeled.
root@app-01:/srv/appdata# lsof +L1 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME bash 772 root 3w REG 7,0 1003884544 0 14 /srv/appdata/logs/app.log (deleted)
df -h /path: Block-level truth: how full the filesystem really isdu -sh /path: Name-level view: what is reachable through the treelsof +L1: Open files with zero links: deleted but still heldls -l /proc/PID/fd: The same reveal without lsof, works in containers: > /proc/PID/fd/N: Free a deleted file's space without a restart: > /var/log/app.log: Shrink a live log safely instead of deleting itWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.