The short answer: fork failed with EAGAIN because the user hit their maximum process limit (RLIMIT_NPROC). It is not a permission or memory error: the user already owns as many processes as they are allowed. Check with ps -u USER | wc -l and compare against grep processes /proc/PID/limits.
A CI build step spawns a worker per source file and never waits for them. The process count climbs to exactly 200 and stops, and from that moment the CI user cannot start a single new process: not a shell, not ps, not ls. The box itself is fine and root works normally. This replay shows the climb, the EAGAIN failure in the build log, where the limit came from, and the recovery.
The box is build-01, a CI worker. A build step spawns one background worker per source file and then waits for all of them. Each worker is a sleep: no CPU, almost no memory, nothing that looks dangerous on a dashboard.
Under a running count you can watch processes owned by the CI user climb and then stop dead at 200. That flat line is not the build finishing. It is the kernel refusing to create process 201 for that user.
From that instant the CI user is locked out of the machine in the most complete way possible: every command they could type is itself a new process, so nothing they run can start.
Every new process on Linux starts with fork, and fork can fail. When it fails because the calling user has hit RLIMIT_NPROC, the kernel returns EAGAIN, which the shell prints as "Resource temporarily unavailable". Bash even retries a few times before giving up, which is why the log fills with retry lines.
The limit is per user, not per process and not per program. It counts every process that user owns across the whole system, so a runaway loop in one job exhausts the allowance for that user everywhere at once.
The word "temporarily" is doing real work in that message. Nothing is broken and nothing is corrupt. The moment processes exit, the same fork succeeds.
Read the limit from a live process instead of guessing: /proc/PID/limits shows the soft and hard ceiling that process actually got, which beats reading config files that may not be what got applied. Here it reads 200.
Then compare against the machine: kernel.pid_max is 4,194,304. The box has room for millions of processes. This was never a machine limit, and adding RAM or CPU would have changed nothing.
The config that produced it lives in /etc/security/limits.conf, applied by pam_limits when the session was created. Because the limit is applied at session setup, changing the file does not affect sessions that are already running.
root@build-01:~# grep processes /proc/$(pgrep -u ci -n)/limits Max processes 200 200 processes root@build-01:~# cat /proc/sys/kernel/pid_max 4194304
ps -u USER --no-headers | wc -l: How many processes a user currently ownsgrep processes /proc/PID/limits: The real limit a running process was givenulimit -u: The current shell's own process limitcat /proc/sys/kernel/pid_max: System wide ceiling, for ruling the box outpkill -9 -u USER: Kill every process owned by that usersystemctl set-property UNIT TasksMax=512: Cap a service tree with cgroups insteadWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.