TCP States

ss -t state - a hands-on Linux lab on a real virtual machine.

Every TCP connection moves through a series of states. ss shows you where each connection is in its lifecycle. This is how you diagnose broken connections.

A connection is not a thing. It is a place in a story

When you open a web page, your browser creates a TCP connection to the server. That connection does not just appear and disappear. It goes through a series of states, like stages in a conversation. It starts with a greeting (the three-way handshake), moves into the main conversation (data transfer), and ends with a goodbye (the four-way teardown).

The kernel tracks the state of every TCP connection on the system. The ss command lets you see those states. This matters because when something goes wrong with a network service, the TCP states tell you exactly where the conversation broke down.

There is a socket on this machine that will outlive the connection it belonged to by a full minute, and there is a state whose presence in quantity is always somebody's bug. You will meet both before the end of this lesson.

The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The real VM comes at the end, under Ready to practice. Your progress in the real lab is tracked automatically, so type commands naturally and watch the checkmarks appear.

What is a TCP state?

A TCP state describes where a connection is in its lifecycle. Think of it like a phone call. You dial the number (SYN-SENT). The phone rings on the other end (SYN-RECEIVED). They pick up (ESTABLISHED). You talk. One side says goodbye (FIN-WAIT). The other side hangs up (CLOSE-WAIT, LAST-ACK). The call is over (CLOSED).

The kernel uses these states internally to decide what packets to send next and what packets it expects to receive. The ss command reads these states directly from the kernel and shows them to you.

The TCP state diagram

Here is the full lifecycle of a normal TCP connection:

  Client                          Server
  ------                          ------
  CLOSED                          CLOSED
    |                               |
    |        (server listens)       |
    |                            LISTEN
    |                               |
    |-------- SYN ----------------->|
  SYN-SENT                      SYN-RECEIVED
    |                               |
    |<------- SYN+ACK -------------|
    |                               |
    |-------- ACK ----------------->|
  ESTABLISHED                   ESTABLISHED
    |                               |
    |    ... data flows both ways ...|
    |                               |
    |-------- FIN ----------------->|
  FIN-WAIT-1                    CLOSE-WAIT
    |                               |
    |<------- ACK -----------------|
  FIN-WAIT-2                        |
    |                               |
    |<------- FIN -----------------|
    |                            LAST-ACK
    |-------- ACK ----------------->|
  TIME-WAIT                     CLOSED
    |                               
    | (waits 2x MSL)               
  CLOSED

That diagram shows the client closing first, which is the textbook case. Either side is allowed to close first, and in this lab it is usually the server, because the little web server answers the request and hangs up immediately.

That is a lot of states. Do not memorize them all right now. The important ones to recognize are:

State	Printed by ss as	What it means
LISTEN	`LISTEN`	A server socket waiting for incoming connections
ESTABLISHED	`ESTAB`	Connection is open, data is flowing
TIME-WAIT	`TIME-WAIT`	Connection closed, kernel keeps the socket around briefly to handle late packets
CLOSE-WAIT	`CLOSE-WAIT`	The remote side closed, but the local application has not closed its end yet
SYN-SENT	`SYN-SENT`	A SYN went out, still waiting for the server to respond
SYN-RECEIVED	`SYN-RECV`	Server got a SYN, sent back SYN+ACK, waiting for the final ACK

The middle column matters. ss abbreviates some of the names, so the state you read about is not always spelled the way the tool prints it.

Why these names never changed

That diagram is not a teaching aid somebody drew for this lesson. It is in the specification. The 1981 document that defines TCP contains the state machine as a picture, with the same boxes and the same arrows, and every operating system since has implemented it.

Which is why the names survive. LISTEN, ESTABLISHED, TIME-WAIT and the rest are not Linux words, they are the spec's words, so the same state names appear on a BSD box, on a Windows server, in a switch, and in the output of a command written decades later. Learn them once here and you can read a socket table on any machine you will ever touch.

One name in that diagram is a promise about physics. TIME-WAIT lasts twice the Maximum Segment Lifetime, the longest anyone assumes a packet can survive while lost in the network. The spec picked a number, the network has to live inside it, and your socket table is where that assumption becomes visible: a socket sitting there doing nothing, waiting out the worst case.

Your lab network

This lab has a small network built inside your VM using network namespaces. A network namespace is a separate, isolated network stack inside one Linux machine, with its own interfaces, routing table and socket list. Think of each one as a separate computer.

  [client]  10.0.1.10  <-------->  10.0.1.20  [server]
  MAC: 02:00:0a:00:01:0a           MAC: 02:00:0a:00:01:14

The server runs a simple HTTP service on port 80. You will create connections to it and watch them move through states.

One consequence of the namespaces shows up immediately: ss only reports the sockets of the namespace it runs in. Typed at your normal prompt, it shows the sockets of the VM itself, not the lab network. To look inside a namespace, run the command through ip netns exec:

sudo ip netns exec server ss -ta

Read that as: "in the namespace called server, run ss -ta." Two helper commands generate the traffic: send-curl makes one HTTP request and lets it close, and hold-open opens a connection to port 80 and keeps it open for 60 seconds.

These commands are long and you will type them many times. Tab completion finishes command names anywhere on the line: send- then Tab twice lists both generators, hold- then Tab completes on its own. The shell also keeps your history, so the up arrow brings back the last ip netns exec line and you only have to change the flags on the end.

Seeing TCP states with ss

The ss command (socket statistics) shows the sockets the kernel is tracking. To see TCP sockets, use the -t flag. On its own, -t shows only connections. Add -a and you get listening sockets too.

Nothing else is going on in that namespace yet, so before you press Enter, commit to a number of rows and to what the first column of each will say.

sudo ip netns exec server ss -ta

prompt: student@states-lab:~$ answer: sudo ip netns exec server ss -ta ||| sudo ip netns exec server ss -at ||| sudo ip netns exec server ss -t -a output: State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 5 0.0.0.0:80 0.0.0.0:* hint: Inside the namespace, TCP sockets, and all of them rather than only the connected ones: sudo ip netns exec server ss -ta

One socket, because nothing has connected yet. The first column is the state. LISTEN is the HTTP server waiting for connections. 0.0.0.0:80 means it accepts connections on any address of that machine, port 80. On a LISTEN row, Send-Q is not a byte count, it is the accept backlog: how many completed connections the kernel will queue up for the program.

Now open a connection that stays open, and look again:

hold-open
sudo ip netns exec server ss -ta

A second row appears, this one with State ESTAB and a peer address, because there is now a real conversation with a machine on the other end. When hold-open lets go after 60 seconds, that row disappears. Watching the table change under you is the first of the five things your lab grades.

The ss command replaced the older netstat command. You will still see netstat in older documentation, but ss is faster and shows more detail. When in doubt, use ss.

Filtering by state

When a busy server has hundreds of connections, you do not want to read through all of them. ss can filter by state, with the keyword state followed by the name of one:

hold-open

With that connection open, ask for established connections and nothing else. Watch the header when it comes back, because one column is missing from it. The peer port below is written as a label because the kernel picks a fresh high number for every connection:

sudo ip netns exec server ss -t state established

prompt: student@states-lab:~$ answer: sudo ip netns exec server ss -t state established ||| sudo ip netns exec server ss -tn state established output: Recv-Q Send-Q Local Address:Port Peer Address:Port 0 0 10.0.1.20:80 10.0.1.10:<client port> hint: The TCP flag, then the keyword state, then the name of the state you want: sudo ip netns exec server ss -t state established

That shows connections in the ESTABLISHED state and nothing else. Notice the State column is gone from the output: you already know the state, so ss stops printing it.

You can filter by any state name:

sudo ip netns exec server ss -t state time-wait
sudo ip netns exec server ss -t state listening
sudo ip netns exec server ss -t state close-wait

Asking for one named state is the second of the five things your lab grades.

The names ss accepts here are its own, and they are not always what the output column prints. The listening state is listening, not listen. The established state is established, even though the column reads ESTAB. Get the name wrong and ss rejects the filter instead of matching nothing.

Commit: who is left holding the socket?

Now let a connection end on its own instead of holding it open. send-curl makes one request, gets the page, and the connection closes. Something is left behind for the next minute, in exactly one of the two namespaces.

[ ] The client, because it started the connection.
[x] The server, because it is the side that closed first here.
[ ] Both of them, because the connection belonged to both.

>>> The server. The side that sends the first FIN is the side that ends up in TIME-WAIT, and here that is normally the server, because this web server closes the connection as soon as it has sent the page. If you picked the client, you have the textbook diagram in mind, where the client closes first. Real servers close first all the time, and which one did it is a question you answer by looking, not by assuming. If you picked both, this is the misunderstanding worth killing early: TIME-WAIT is not a property of the connection, it is a property of one end of it.

Look at both ends

Make a request that closes on its own, then look for the leftovers on each side in turn:

send-curl
sudo ip netns exec server ss -t state time-wait
sudo ip netns exec client ss -t state time-wait

One of those two namespaces has a row and the other does not. The kernel keeps that socket for 60 seconds and then drops it.

Checking both sides is the habit worth taking away. When a problem lives at one end of a connection, looking only at the end you happen to be logged into will show you a perfectly healthy machine.

Commit: a pile of CLOSE-WAIT sockets

Here is a real morning on a real server. A service is misbehaving, and ss shows hundreds of sockets sitting in CLOSE-WAIT that never go away.

[ ] The network is dropping the packets that close those connections.
[x] The application received a close from the far side and never closed its own end.
[ ] The kernel is holding them deliberately, the same way it holds TIME-WAIT sockets.

>>> The application. If you see many connections stuck in CLOSE-WAIT, that usually means an application received a close from the remote side but never called close() on its own socket. This is an application bug, not a network problem. If you picked the network, notice that the machine already received the close successfully, which is precisely why the state changed to CLOSE-WAIT. If you picked the kernel, that is the confusion this gate exists to clear: TIME-WAIT is the kernel holding a socket on purpose and letting go on a timer, while CLOSE-WAIT is the kernel waiting for a program that is never going to call.

Seeing states on the wire with tshark

TCP states change because of specific flags in TCP packets. The three most important flags are:

SYN - "I want to start a connection." Sent in the opening handshake.
ACK - "I received your data." Sent on almost every packet after the handshake.
FIN - "I am done sending data." Sent when one side closes the connection.

You can watch these flags in a packet capture. The order of the two commands matters: tshark needs a second or two to open the interface, so schedule the traffic to fire a few seconds later and start the capture immediately.

(sleep 5; send-curl) &

That line means "wait 5 seconds, then run send-curl, in the background". If you fire the request first, the connection is over before the capture is listening and you sit staring at an empty screen.

The packet numbers, timestamps and client port will differ on your VM. Twenty lines will scroll past, and the screen below is trimmed to the three-way handshake at the beginning:

cap -c 20 -Y tcp.flags

prompt: student@states-lab:~$ answer: cap -c 20 -Y tcp.flags ||| cap -Y tcp.flags -c 20 ||| sudo ip netns exec client tshark -i veth-c -c 20 -Y tcp.flags output: 1 0.000000 10.0.1.10 -> 10.0.1.20 TCP [SYN] 43210 > 80 2 0.000123 10.0.1.20 -> 10.0.1.10 TCP [SYN, ACK] 80 > 43210 3 0.000234 10.0.1.10 -> 10.0.1.20 TCP [ACK] 43210 > 80 hint: The capture shortcut, a count of twenty, and a display filter naming the TCP flags field: cap -c 20 -Y tcp.flags

Packet 1: the client sends SYN (state moves to SYN-SENT). Packet 2: the server replies with SYN+ACK (state moves to SYN-RECEIVED, then ESTABLISHED). Packet 3: the client sends ACK (both sides are now ESTABLISHED).

After the data transfer, look for the FIN packets at the end of your own screen:

  ... data packets ...
  18  0.005678  10.0.1.10 -> 10.0.1.20  TCP  [FIN, ACK] 43210 > 80
  19  0.005789  10.0.1.20 -> 10.0.1.10  TCP  [FIN, ACK] 80 > 43210
  20  0.005890  10.0.1.10 -> 10.0.1.20  TCP  [ACK] 43210 > 80

The FIN packets are the goodbye. Each side sends one, and the connection moves through FIN-WAIT-1, FIN-WAIT-2, TIME-WAIT, and finally CLOSED.

This is the third of the five things your lab grades, and it is the moment the two views join up. The states you read in ss are the kernel's side of exactly these packets.

Milestone: two views of the same connection

Stop and take stock. You can list every socket a machine holds and name the stage each one is in. You can narrow that list to a single state on a busy machine. And you can watch the packets that cause the transitions, on the wire, as they happen.

That pairing is the skill. The socket table tells you where a connection is stuck; the capture tells you which packet did or did not arrive to move it along. Neither one alone settles an argument. Together they usually end it in a minute.

Challenge: counts, not a list

Scaffolding off. No command is printed from here on.

A server feels wrong and you have thirty seconds before somebody asks you what is happening. You do not want a list of sockets, you want the shape of the problem: how many sockets are there, and how many are in each state?

ss has a summary flag for exactly this, and it is -s. Your numbers will be different from anyone else's, because they depend on what is running at that moment:

prompt: student@states-lab:~$ answer: sudo ip netns exec server ss -s ||| sudo ip netns exec server ss --summary output: Total: 4 TCP: 2 (estab 1, closed 0, orphaned 0, timewait 0)

Transport Total IP IPv6 RAW 0 0 0 UDP 0 0 0 TCP 2 2 0 INET 2 2 0 FRAG 0 0 0 hint: The namespace form you have used all lesson, and the summary flag on its own with no other flags.

The output comes in two parts. First a summary line for TCP, then a table counting sockets per transport protocol and per address family.

The line starting with TCP: is the one to read. It gives the total and then the breakdown by state. This is the fastest health check there is for a server's connections. Thousands of TIME-WAIT sockets means the server is handling a lot of short-lived connections, which is normal. A CLOSE-WAIT count that keeps climbing means an application is not closing its sockets.

For the long version, list every socket in every state instead:

sudo ip netns exec server ss -t state all

all is a state name like any other, and it means "do not filter". You get every TCP socket, including TIME-WAIT and LISTEN. Either form is the fourth of the five things your lab grades.

Commit: can you make TIME-WAIT go away?

A colleague looks at a server holding thousands of TIME-WAIT sockets and wants them gone by lunchtime. Take a position before you agree to anything.

[ ] Lowering net.ipv4.tcp_fin_timeout, which is the TIME-WAIT knob.
[x] Nothing you can set. On Linux the value is compiled into the kernel.
[ ] Restarting the service, which drops its TIME-WAIT sockets immediately.

>>> Nothing you can set. On Linux, TIME-WAIT lasts 60 seconds and there is no sysctl to change it. The value is compiled into the kernel. tcp_fin_timeout is often mistaken for this knob, but it controls something else: how long a socket may sit in FIN-WAIT-2 waiting for the other side to close. If you picked the restart, notice who owns the socket. A TIME-WAIT socket has no process attached to it any more; the kernel is holding it, so restarting the program changes nothing. The right answer to your colleague is usually that these sockets are not the problem.

Challenge: watch the countdown

Still no command shown.

The best way to understand TCP states is to watch a connection die. Make a request that closes on its own with send-curl, then list the sockets with their timers showing. The flag that adds timer information to each row is -o, and you want it alongside the two flags you used in the very first listing.

The peer port is labelled below because it changes every run, and the seconds remaining will be whatever is left when you look:

prompt: student@states-lab:~$ answer: sudo ip netns exec server ss -tao ||| sudo ip netns exec server ss -taon ||| sudo ip netns exec server ss -t -a -o output: State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 5 0.0.0.0:80 0.0.0.0:* TIME-WAIT 0 0 10.0.1.20:80 10.0.1.10:<client port> timer:(timewait,52sec,0) hint: The listing from the first task with one more

Practice TCP States in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.