The short answer: A stalled DNS resolver is the most common cause. If requests take a round number of seconds, usually five or ten, that is a timeout rather than a fault. Compare time getent hosts NAME against a request straight to the IP address: if the IP is instant, only name resolution is broken.
A web server starts taking five seconds to do anything that touches a hostname. Nothing errors, nothing crashes, and every dashboard is green. The configured resolver stopped answering, and because its address is still routable the queries leave and simply never come back. This replay shows the five second stall appearing, the one comparison that isolates it, and the fix.
The box is web-04. At the start, resolving a hostname takes about ten milliseconds and an HTTP request is quick. Then the configured nameserver stops answering.
The critical detail is that the resolver's address is still routable. Queries leave the box and nothing comes back, so nothing reports an error. If the address were unreachable the resolver would fail instantly and this would be a five minute incident instead of a five hour one.
From then on every lookup burns the full resolver timeout before giving up. The number is suspiciously round, and a round number is almost always a timeout rather than a fault.
A broken thing announces itself. A slow thing gets absorbed: requests still succeed, health checks still pass, and the graphs show latency without showing a cause. Everyone starts looking at the application, because the application is what got slower.
DNS is uniquely good at this because almost everything resolves names, and the resolver library is deep enough in the stack that most tools never mention it. The database client, the HTTP client, the package manager, and the monitoring agent all pay the same five seconds without any of them saying the word DNS.
The timeout is configurable and defaults to five seconds per attempt with two attempts, so a single failed resolver can add ten seconds before a fallback is even tried. That is where the mysterious ten second page loads come from.
Make the same request twice: once by hostname, once by IP address. If the hostname is slow and the IP is instant, the network is fine, the remote service is fine, and only name resolution is sick. That single test cuts the problem in half in about ten seconds.
Then confirm which resolver is at fault. Query the configured nameserver directly and watch it time out, then query a known-good resolver and watch it answer immediately. Now you are not guessing, and you know exactly which box to page.
root@web-04:~# time getent hosts deb.debian.org real 0m5.012s root@web-04:~# dig +time=3 +tries=1 @10.255.255.1 deb.debian.org ;; no servers could be reached root@web-04:~# dig +short @1.1.1.1 deb.debian.org debian.map.fastlydns.net.
time getent hosts NAME: Resolution latency using the system resolver pathdig +time=3 +tries=1 @SERVER NAME: Query one specific nameserver with a short timeoutcat /etc/resolv.conf: Which resolvers and which timeout are configuredcurl -o /dev/null http://IP/: Same request by IP: isolates DNS from everything elseresolvectl status: What systemd-resolved is actually usingoptions timeout:1 attempts:2: resolv.conf line that makes failover fastWatch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.