The short answer: ext4 detected metadata it could not trust and applied its errors=remount-ro behaviour, taking the mount read-only rather than writing more data to a device that is returning wrong results. Confirm with findmnt -no OPTIONS /path and read dmesg for the underlying ext4 or I/O error.
A database volume is healthy and writable. Metadata under the mount is corrupted, standing in for a failing drive, and the moment ext4 reads something that cannot be true it stops trusting the device and remounts the whole filesystem read-only. Every service writing there fails at the same instant. This replay shows the flip happening live, the kernel log that explains it, and the repair in the order that keeps the data.
The box is db-03 with an application database on its own volume. The tables read fine, writes work, and the mount options say rw. This is an ordinary healthy server.
Then metadata under the mounted volume is corrupted, which is what a failing drive does to you. Nothing announces itself: the damage is on the disk and the kernel has not read that block yet, so for a moment everything still looks normal.
The instant ext4 reads metadata that cannot be true, it stops trusting the device. Rather than keep writing onto a disk that is lying to it, it flips the entire mount to read-only. Every service on that volume starts failing at exactly the same moment, which is why this reads as a total outage rather than a disk fault.
ext4 is created with an errors behaviour, and on most distributions it is remount-ro. When the filesystem detects an inconsistency it cannot reconcile, it takes the whole mount read-only rather than continue.
This looks like the filesystem causing the outage, and it is the opposite. Continuing to write to a device that is returning wrong metadata is how a recoverable disk problem becomes unrecoverable data loss. The read-only flip is the filesystem refusing to make it worse.
The other options are worse in different ways. errors=continue keeps going and lets corruption spread. errors=panic takes the whole machine down immediately, which is sometimes right in a cluster where another node takes over, and rarely right anywhere else.
Read the mount options first. When they say ro on a filesystem that was mounted rw, you have the answer in one command, and it is the check most people skip while they are restarting services and blaming the application.
Then read the kernel log, because that is where the actual fault is recorded. dmesg holds the ext4 error, the inode or block it choked on, and the remount decision. Without it you are guessing about a disk, and guessing about disks loses data.
Do not skip to remounting read-write. The kernel has already told you it does not trust this device, and remounting without repairing puts you straight back where you started, this time with writes landing on known-bad metadata.
root@db-03:/var/lib/appdb/data# echo 'ORD-8805 pending' >> orders.tbl -bash: orders.tbl: Read-only file system root@db-03:/var/lib/appdb/data# findmnt -no OPTIONS /var/lib/appdb ro,relatime
findmnt -no OPTIONS /path: Whether the mount is rw or ro right nowdmesg | grep -i ext4: The actual filesystem error and remount decisionumount /path: Always before fsck. Never repair a mounted filesystemfsck.ext4 -y /dev/DEV: Repair, answering yes to each fixtune2fs -l /dev/DEV | grep -i error: The errors behaviour and error historysmartctl -a /dev/DEV: Whether the disk underneath is failingWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.