How do I find what is using a port if no process shows up?

Use ss -tan rather than ss -tlnp. The l flag filters to LISTEN state and hides sockets in TIME_WAIT, CLOSE_WAIT, and every other state. If the port only appears without the l flag, the kernel is holding it and there is no process to kill.

How do I make a service restart without waiting for TIME_WAIT?

Set SO_REUSEADDR on the listening socket, which lets a new listener bind an address that has sockets in TIME_WAIT. For systemd services, socket activation avoids the problem completely because systemd holds the listening socket across restarts.

The port is in use and nothing is listening on it

The short answer: A socket from the previous connection is in TIME_WAIT, a kernel state that outlives the process by about a minute. ss -tlnp shows only listeners so it looks empty. Run ss -tan and the port appears in TIME-WAIT.

An edge proxy is stopped and restarted while a client connection is open. It dies immediately with "Address already in use", and checking for a listener on that port returns nothing at all. The socket is held in TIME_WAIT by the kernel, not by any process. This replay shows the failed restart, the empty listener check, the socket that is actually holding the port, and the one socket option that makes the whole problem disappear.

What you are watching

The box is edge-01 running a small proxy on port 8080. A client connects and stays connected, which is completely normal traffic. Then the service is restarted the way any restart happens.

It does not come back. The startup fails immediately with "Address already in use", which makes no sense: the thing that owned that port was just stopped, and the check for a listener on 8080 comes back empty.

This is the moment people start restarting the whole machine, because a port that is busy with nothing on it looks like the box has lost its mind.

Why a port stays busy with no process

When a TCP connection closes, the side that closes first keeps the socket in a state called TIME_WAIT for roughly a minute. The socket has no process attached to it any more. It is a kernel timer holding the address.

TCP does this on purpose. Packets from the old connection can still be in flight, and if a new connection reused that exact address pair immediately, those stray packets could be delivered into it. TIME_WAIT is the guarantee that cannot happen.

The critical detail is that a plain listener bind refuses to touch an address with any socket in TIME_WAIT, including one from a connection the old process had open. That is why a service with active connections at shutdown cannot restart, while the same service with an idle port restarts fine. It also explains the maddening intermittency: whether the restart works depends on whether anybody happened to be connected.

The diagnosis

The listener check is the trap. ss -tlnp only shows sockets in LISTEN state, so it comes back empty and appears to prove the port is free. Drop the l and the port is right there, held in TIME-WAIT.

That one flag is the entire diagnosis. A port that is empty in the listener view and present in the all-sockets view is being held by the kernel rather than by a process, and no amount of hunting for a process to kill will find anything.

root@edge-01:~# ss -tlnp | grep 8080
root@edge-01:~# pgrep -f edge-proxy
root@edge-01:~# ss -tan | grep 8080
TIME-WAIT  0  0  127.0.0.1:8080  127.0.0.1:44214

Quick reference

ss -tlnp | grep PORT: Listeners only. Empty here does not mean the port is free
ss -tan | grep PORT: All socket states, including TIME-WAIT. This finds it
ss -tanp state time-wait: Just the sockets in TIME-WAIT
journalctl -u UNIT -n 20: The real bind error the service printed
SO_REUSEADDR: The socket option that makes restarts work
systemctl list-sockets: Socket-activated units, where systemd owns the port

Watch the full replay, browse all recorded incidents, or fix a broken server yourself in the break/fix challenges.