The short answer: A directory in the path is missing the execute bit for that user. The kernel resolves a path one component at a time and needs execute on every directory along the way, so the file at the end being readable is irrelevant. Run namei -l on the full path to find the component that is missing x.
A hardening change removes the execute bit from a config directory. The service immediately stops being able to read a file it owns, while the file's own mode and owner are unchanged and correct. Root can still read it, which sends everyone down the wrong path. This replay shows the break, why listing the directory still works, and the one command that walks a path the way the kernel does.
The box is app-05. A payments service reads a config file as its own user, which works fine at the start of the replay: the file is mode 644, owned by that service account, in a directory it owns.
Someone ships a hardening change that tightens permissions on the config directory. It is one line and it looks completely reasonable.
Immediately the service cannot read a file it owns, and nothing about the file changed. Root can still read it, which is the detail that wastes the most time: root bypasses permission checks entirely, so testing as root proves nothing at all.
To open a file, the kernel resolves its path one component at a time. For every directory along the way, the user needs execute permission on that directory, which is the right to traverse it. If any component in the chain denies that, the open fails no matter how permissive the file at the end is.
On a directory the bits mean something different from what they mean on a file. Read means you may list the names inside. Execute means you may actually use those names to reach what they point at. They are independent, which produces the strangest symptom in this incident: the service can list the directory and see the file exists, and still cannot open it.
That is why a directory almost always wants r and x together. A directory with read but no execute lets you see a catalogue of things you are not allowed to touch.
Stop looking at the file. When a permission error does not match the file's own mode, the problem is in the path, and the tool for that is namei -l, which walks every component and prints the mode and owner of each one.
One line in that output is missing an x for the user in question, and that line is the bug. It turns a guessing game into a single glance.
Test as the right user, never as root. su to the service account, or use sudo -u, because root ignores the permission bits that are causing the problem and will happily tell you everything is fine.
root@app-05:~# namei -l /opt/payments/config/gateway.conf f: /opt/payments/config/gateway.conf drwxr-xr-x root root / drwxr-xr-x root root opt drwxr-xr-x appsvc appsvc payments drw-r----- appsvc appsvc config -rw-r--r-- appsvc appsvc gateway.conf
namei -l /full/path/to/file: Mode and owner of every component in the pathsudo -u USER cat /path: Test as the affected user. Root proves nothingls -ld /path/to/dir: The directory mode. Note the d and the trailing bitsfind /tree -type d -exec chmod 755 {} +: Set directory modes without touching filesfind /tree -type f -exec chmod 644 {} +: Set file modes without touching directoriesstat -c "%A %U:%G %n" /path: Mode and ownership in one compact lineWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.