The short answer: It is EMFILE: the process hit its file descriptor limit, which has nothing to do with disk space. Descriptors cover sockets and pipes as well as files. Count them with ls /proc/PID/fd | wc -l and compare against grep "open files" /proc/PID/limits.
A session service opens a file per request and never closes it. The descriptor count climbs to exactly 1024 and stops, and from that moment every open, socket, and log write inside that process fails with "Too many open files". The disk is nearly empty and has nothing to do with it. This replay shows the climb, the error, the two commands that diagnose it, and why restarting only buys time.
The box is api-02 running a session service. A file descriptor is what a process uses to refer to anything it has open: a file, a socket, a pipe. Each process gets a budget of them, and almost nobody watches that number until it runs out.
The service opens a session file per request and never closes it. Under a running count the descriptor total climbs steadily and then stops dead at 1024. Nothing else on the box moves: CPU is idle, memory is flat, the disk is nearly empty.
From that ceiling onward every open inside that process fails. The rest of the machine is completely healthy, which is what makes this look like an application bug rather than a resource problem.
The error is EMFILE, printed as "Too many open files". It says files, but a descriptor covers sockets and pipes too, so a service that hits this limit usually shows up first as refused or hanging connections rather than anything file related.
That is the trap. The message sends everyone to check disk space and inodes, and both are fine. The resource that ran out is a per-process table inside the kernel, and it has no relationship to storage at all.
There are two ceilings worth knowing apart. EMFILE is the per-process limit, which is what almost every incident is. ENFILE, "file table overflow", is the system wide limit in fs.file-max and is genuinely rare on a modern box.
Two commands settle it. Count what the process is using by listing its descriptors in /proc/PID/fd, and read the ceiling it was actually given from /proc/PID/limits. When the first number equals the second, the diagnosis is finished.
Read the limit from the running process rather than from a config file. Limits come from several places and the file you are reading may not be the one that applied. What /proc reports is what the kernel actually gave that process.
lsof on the process shows what those descriptors are, which is what points at the leak: thousands of handles to the same kind of file, all still held open long after the work finished.
root@api-02:~# ls /proc/$(pgrep -f session-api)/fd | wc -l 1024 root@api-02:~# grep 'open files' /proc/$(pgrep -f session-api)/limits Max open files 1024 1024 files
ls /proc/PID/fd | wc -l: How many descriptors a process currently holdsgrep "open files" /proc/PID/limits: The ceiling that process was actually givenlsof -p PID: What those descriptors are, which finds the leaksystemctl show UNIT -p LimitNOFILE: The limit systemd gave the servicecat /proc/sys/fs/file-max: System wide ceiling, for ruling out ENFILELimitNOFILE=65535: Unit setting, the only one that affects a serviceWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.