The short answer: Check systemctl status for the unit: a service killed this way reports "Result: oom-kill". Then read dmesg for lines containing "Out of memory: Killed process", which name the victim and its memory footprint. A process that dies with no error in its own log is the classic signature, because SIGKILL cannot be caught or logged.
The orders API holds a 550MB working set, the way a cache or a database buffer pool does. A reporting job starts accumulating its result set in memory and available memory falls to zero. The kernel invokes the OOM killer, and it kills the orders API: the largest resident process, not the one that caused the shortage. This replay shows the memory drain, the dead service, and the kernel log that names the victim.
The box is cache-01 with 1GB of RAM. The orders API is running and holding about 550MB, which is normal for anything that keeps a cache or a buffer pool in memory. Memory is tight but there is nothing wrong.
A reporting job starts and accumulates its result set in memory, ten megabytes at a time. Every one of those allocations succeeds. Linux hands out memory right up until the moment it cannot, with no warning to anybody.
When there is nothing left to reclaim, the kernel invokes the out of memory killer. It does not return an error to the program asking for memory. It chooses a process and kills it outright, and the process it chose was the orders API.
The OOM killer does not know which process caused the shortage and does not try to find out. It scores every candidate by how much memory freeing it would recover, so the score is driven mostly by resident set size. Then it kills the highest score.
That logic is defensible: the kernel needs to recover the most memory with the fewest kills. But it means the victim is whatever is biggest, which on a real server is your database, your cache, or your application, and almost never the small script that pushed the box over.
This is why the incident is so disorienting. The service that died has no bug, no error in its own logs, and no crash. From its own perspective nothing happened at all: it received SIGKILL, which cannot be caught, logged, or cleaned up after.
systemd records the cause directly. The unit shows "Active: failed (Result: oom-kill)", which is conclusive and takes one command. Any service that dies with no error in its own log deserves this check before anything else.
The kernel log has the full story. dmesg holds the trigger, a table of every candidate process with its memory, and the final verdict naming the victim and the anon-rss that made it the biggest target. In this recording the processes are named orders-api and report-builder, so the log itself shows which one died and which one was responsible.
root@cache-01:~# systemctl status orders-api | sed -n 3p
Active: failed (Result: oom-kill) since Sat 2026-08-01 13:05:39 UTC
root@cache-01:~# dmesg | grep -i 'Out of memory'
Out of memory: Killed process 710 (orders-api) total-vm:935416kB, anon-rss:...
systemctl status UNIT: Result: oom-kill is conclusive proofdmesg | grep -i "out of memory": The kernel verdict naming the victimdmesg | grep -i oom-kill: The trigger and constraint for the killfree -m: Read the available column, not freesystemctl set-property UNIT MemoryMax=512M: Cap the greedy job in its own cgroupOOMScoreAdjust=-500: Unit setting that makes a service a smaller targetWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.