Replay: watch real Linux incidents happen

Every replay is a real incident on a real server, recorded live and replayed byte for byte in a terminal player. Nothing is animated or mocked up: you watch the actual session, the actual errors, and the actual fix, with an explanation of why it happened and how to prevent it.

Current replays

The disk was 1% full and nothing could write: a runaway retry loop exhausts a mail spool's inodes in about 15 seconds while df -h still reports 1% used.

The disk was full and du found almost nothing: a volume fills to 100% while du sees 32K, because the space lives in a log file deleted while the app still held it open.

The build box stopped being able to start anything: a build spawns one worker per source file until the user hits their 200 process limit and can no longer run a single command.

Your service did not crash, the kernel killed it: a reporting job exhausts memory and the OOM killer takes the largest process, which is the innocent orders API.

Too many open files, and the disk has nothing to do with it: a service leaks one file descriptor per request until it hits 1024 and can open nothing at all.

The filesystem locked itself and it was trying to help: ext4 finds metadata it cannot trust and remounts the whole volume read-only, so every write fails at once.

Everything got slow and nothing was broken: a resolver stops answering and every lookup burns a five second timeout, with no error anywhere.

The port is in use and nothing is listening on it: a service cannot rebind its own port because sockets sit in TIME_WAIT, held by the kernel rather than any process.

Permission denied on a file whose permissions are correct: a directory loses its execute bit, so a path stops resolving even though the file at the end is mode 644 and correctly owned.

The backup script works, cron runs it, there are no backups: a script depends on PATH and variables that exist only in an interactive shell, and cron reads no .bashrc.

New incidents are recorded regularly. Zombie processes, TIME_WAIT exhaustion, clock skew, and more are on the way.

Do it yourself

Watching is step one. The break/fix challenges hand you a real broken server, an incident ticket, and a running clock, and the machine itself grades your fix. Fundamentals live in the training tracks.