Ports and Sockets

ss -tlnp - a hands-on Linux lab on a real virtual machine.

A socket is IP plus port plus protocol. /proc/net/tcp is the kernel connection spreadsheet, in hex.

One machine, one cable, three services

Your server is running three services right now. HTTP on port 80. An SSH-like service on port 22. A custom echo service on port 9000. Each one listens on a different port number, and the kernel keeps them completely separate. A packet arriving on port 80 goes to the web server. A packet arriving on port 22 goes to the SSH service. A packet arriving on port 9000 goes to the echo service.

All three arrive on the same wire, in the same frames, addressed to the same IP address. Nothing about the cable knows the difference. Something inside the kernel does.

How does this work? The answer involves three concepts: ports, sockets, and the kernel data structures that tie them together. By the end of this lesson you will have listed those structures, read them in raw hexadecimal straight out of the kernel, and named the program behind each one.

The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The real VM comes at the end, under Ready to practice. Your progress in the real lab is tracked automatically, so type commands naturally and watch the checkmarks appear.

What is a port?

A port is a 16-bit number, ranging from 0 to 65535. It identifies a specific service on a machine. Think of it like an apartment number in a building. The IP address is the street address (which building). The port number is the apartment (which service inside that building).

Port numbers are divided into three ranges:

Range	Name	Purpose
0 - 1023	Well-known ports	Reserved for standard services (root required to bind)
1024 - 49151	Registered ports	Assigned to specific applications by IANA
49152 - 65535	Dynamic ports	Reserved by IANA for temporary use, never assigned to a service

Those ranges come from IANA, the body that hands out port numbers.

Commit: which port does a client actually get?

A server picks its port deliberately: the web server asked for 80 because that is where browsers look. A client does not care which port it uses, so it asks the kernel for any free one and takes whatever it is given.

You have already seen a few of those client ports in the capture lessons. Take a position on where they come from.

[ ] Something above 49152, because that is the dynamic range IANA reserved for exactly this.
[x] Something in the 30000s to 60000s, out of a range the kernel keeps for itself.
[ ] The next number after the last one used, counting up from 1024.

>>> Something in the 30000s to 60000s. Linux does not follow the IANA dynamic range. When a program asks the kernel for "any free port", the kernel picks from its own local port range, which on a default Linux system is 32768 to 60999. If you picked the IANA range, you have just met one of the most useful lessons in networking: the standards say one thing and the running system is the authority. Always check the machine. If you picked counting up from 1024, the kernel does move through the range rather than reusing a port immediately, but it starts inside its own range, and picking a low port would collide with the services that reserved them.

The range this kernel actually uses

You do not have to take that on faith. The range is a file:

cat /proc/sys/net/ipv4/ip_local_port_range

It prints two numbers, the low end and the high end of the range the kernel hands out. That is why the client ports you see in this lab are in the 30000s and 40000s rather than above 49152.

The well-known ports you will see most often:

Port	Protocol	Service
22	TCP	SSH (secure remote access)
53	TCP/UDP	DNS (name resolution)
80	TCP	HTTP (web, unencrypted)
443	TCP	HTTPS (web, encrypted)
25	TCP	SMTP (email)
3306	TCP	MySQL database
5432	TCP	PostgreSQL database

What is a socket?

A socket is the kernel data structure that represents one end of a network connection. It combines three pieces of information:

IP address (which machine)
Port number (which service)
Protocol (TCP or UDP)

Together, these three values uniquely identify one endpoint. A full connection is identified by a pair of sockets: the local socket and the remote socket. For example:

Local:  10.0.1.10:43218 (TCP)   <--->   Remote: 10.0.1.20:80 (TCP)

That is one TCP connection. The client is using port 43218, the server is using port 80. The kernel tracks this as a single socket pair.

Where the word came from

The socket is not a networking idea. It is a Unix idea about files that networking borrowed.

It arrived in 1983 with 4.2BSD, the Berkeley Unix release that first shipped TCP/IP to a wide audience. Its designers wanted network connections to behave like everything else in Unix, so they gave a connection a file descriptor: a small number a program uses to read and write, exactly as it would for a file on disk. The calls they invented, socket, bind, listen, accept, connect, are still the calls every operating system exposes today, including the ones with no Unix in them.

That design decision is why the Process column you will read later prints an fd= number next to each socket, and why the errors in this lesson look like ordinary file errors. To the program, a connection is just another open file. To the kernel, it is a socket structure with an address, a port and a protocol attached.

Your lab network

This lab has a server running three services on different ports:

  [client]  10.0.1.10  <-------->  10.0.1.20  [server]
  MAC: 02:00:0a:00:01:0a           MAC: 02:00:0a:00:01:14

  Server services:
    Port 22   - SSH mock (sends an SSH banner, then closes)
    Port 80   - HTTP (web server)
    Port 9000 - Custom service (sends "hello", then closes)

One helper command matters for this lesson: hold-open opens a TCP connection to port 80 and keeps it open for 60 seconds, so you can look at a live connection instead of chasing one that has already closed.

The client and the server are network namespaces: separate, isolated network stacks inside one Linux machine, each with its own interfaces, routing table and socket list.

This matters for every command below. ss and /proc/net/tcp only report the sockets of the namespace they run in. At your normal prompt you are in the VM's own namespace, and the three lab services are not there. To look inside the server namespace, run the command through ip netns exec:

sudo ip netns exec server ss -tln

Read that as: "in the namespace called server, run ss -tln."

Every command in this lesson starts with the same seven words. Type sudo ip netns exec s and press Tab: the shell has no idea what a namespace is, but it will finish ss for you further along the line, and the same Tab habit finishes /proc/net/tcp after /proc/net/t. Tab completion works on paths and command names anywhere in a line, not only at the start.

Listing listening sockets

The ss command shows every socket the kernel is tracking. To see which services are listening for connections, break the flags down:

-t shows TCP sockets
-l shows only listening sockets (servers waiting for connections)
-n shows numeric addresses (port numbers instead of service names)

Three services are running, so before you press Enter, commit to a number of rows and to what will be sitting in the Peer Address column of each one.

sudo ip netns exec server ss -tln

prompt: student@sockets-lab:~$ answer: sudo ip netns exec server ss -tln ||| sudo ip netns exec server ss -tnl ||| sudo ip netns exec server ss -t -l -n output: State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 5 0.0.0.0:22 0.0.0.0:* LISTEN 0 5 0.0.0.0:80 0.0.0.0:* LISTEN 0 5 0.0.0.0:9000 0.0.0.0:* hint: Inside the namespace, then TCP, listening and numeric: sudo ip netns exec server ss -tln

One row per service, three in total. Each socket is bound to a different port. 0.0.0.0 in the Local Address means "any address on this machine", so the service answers on every interface. 0.0.0.0:* in the Peer Address means no client is connected: the socket is only listening.

On a LISTEN row, Send-Q is not a byte count. It is the accept backlog: how many finished connections the kernel will hold in a queue while the program gets around to accepting them. All three of these programs asked for a backlog of 5.

This is the first of the five things your lab grades, and it is the single most typed diagnostic command in Linux networking. What is this machine offering to the network? That, in one line.

Commit: where does ss get all that?

ss is convenient, and convenience always hides something. Before you go under it, decide what you think it is doing.

[ ] It asks each running program which ports it opened.
[x] It reads the kernel's own accounting, the same data the file /proc/net/tcp exposes.
[ ] It sends a probe to every port from 0 to 65535 and notes which ones answer.

>>> It reads what the kernel already knows. Every socket on the machine belongs to the kernel, so the kernel has the complete list at all times and no asking around is needed. If you picked asking the programs, notice that a program in a tight loop, or wedged, or stopped, could not answer, and yet its sockets still appear. If you picked probing ports, that is what a port scanner does from the outside, and it is slow, noisy, and cannot see a thing about the process behind the port. The whole point of ss is that it is on the inside.

The kernel connection table: /proc/net/tcp

The ss command is convenient, but it is just a frontend. The real data lives in the kernel, and you can read it directly from /proc/net/tcp. Like ss, that file is per namespace, so read the server's copy.

The file starts with a header line naming the columns, then one line per socket. The screen below is trimmed: it shows the first three columns of those listening sockets, with the rest of each line cut off on the right.

sudo ip netns exec server cat /proc/net/tcp

prompt: student@sockets-lab:~$ answer: sudo ip netns exec server cat /proc/net/tcp output: sl local_address rem_address st 0: 00000000:0016 00000000:0000 0A 1: 00000000:0050 00000000:0000 0A 2: 00000000:2328 00000000:0000 0A hint: Same namespace form as before, then read the file with cat: sudo ip netns exec server cat /proc/net/tcp

This is the same information ss shows, but in raw hex, and the lines come out in whatever order the kernel keeps them in, not sorted by port. Let us decode one:

00000000:0016 is the local address. 00000000 is IP 0.0.0.0 (all interfaces). 0016 is hex for 22, the SSH port.
00000000:0000 is the remote address. All zeros means no client connected.
0A is the socket state. 0A is hex for 10, which means LISTEN.

Hex digits are place values of 16, the same way decimal digits are place values of 10. In a four-digit hex number the places are 4096, 256, 16 and 1, going left to right. So:

0016 = (0 * 4096) + (0 * 256) + (1 * 16) + 6      = 22
0050 = (0 * 4096) + (0 * 256) + (5 * 16) + 0      = 80
2328 = (2 * 4096) + (3 * 256) + (2 * 16) + 8      = 9000

Common socket states in hex:

Hex	Decimal	State
01	1	ESTABLISHED
02	2	SYN_SENT
06	6	TIME_WAIT
0A	10	LISTEN

Reading this file is the second of the five things your lab grades.

/proc/net/tcp holds IPv4 sockets only. IPv6 sockets live in a separate file, /proc/net/tcp6. UDP has the same pair: /proc/net/udp and /proc/net/udp6.

Reading /proc/net/tcp is not something you do every day. But knowing it exists helps you understand that ss is not magic. It is just reading from the kernel and formatting the output for humans.

Commit: drop one flag and see what happens

So far every ss command you have run carried -l. You are about to take it away and run ss -tn instead, with a live connection on the machine.

[ ] Nothing changes. -l is a display detail.
[x] The three listeners vanish, and connected sockets appear instead.
[ ] Listeners and connections, both at once.

>>> The listeners vanish and the conversations appear. -l does not add listening sockets to a list, it narrows the list to only those, and without it ss shows the sockets that are connected to something. That is why this lesson keeps telling you which flags to leave off: the flags are not decorations, each one is a filter, and the absence of a flag is as much a part of the question as its presence. If you wanted both at once there is a flag for that too, -a, and you will use it in the next lesson.

Connected sockets

Listening sockets wait for connections. Connected sockets are active conversations. The problem with looking at one is that a plain HTTP request is over in a millisecond, so hold-open exists: it opens a connection to port 80 and keeps it open for 60 seconds.

hold-open

Now there is something to look at. Drop the -l and ss stops showing listeners and shows conversations instead:

sudo ip netns exec server ss -tn

prompt: student@sockets-lab:~$ answer: sudo ip netns exec server ss -tn ||| sudo ip netns exec server ss -nt ||| sudo ip netns exec server ss -t -n output: State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 10.0.1.20:80 10.0.1.10:41234 hint: The same command as the listener listing with the listening flag removed: sudo ip netns exec server ss -tn

That is a connected socket, seen from the server's side. The state is ESTAB, short for established. Both ends are on the line: the server at 10.0.1.20 on port 80, and the client at 10.0.1.10 on a port the kernel picked out of the local port range. Your client port will be a different number.

A listening socket has a local address and no peer. A connected socket has both. That pair of addresses, four values in total, is what makes every connection on the machine unique.

Two machines can hold a thousand connections between them at once and never collide, because each one has a different client port, and that is the third of the five things your lab grades.

If you only see the column headers, the 60 seconds are up. Run hold-open again and look straight away.

Milestone: you can see what the kernel sees

Count what changed. You can list what a machine offers to the network, read the same information in the kernel's own hexadecimal, and watch a live conversation from the inside. Three views of one truth: a socket is an address, a port and a protocol, and the kernel keeps a row for every one of them.

What is missing is the interesting part. So far a socket is a row in a table. The next two steps make it a thing with a health report and an owner.

Socket details: TCP internal state

The -i flag adds the kernel's internal TCP bookkeeping under each connected socket. Run hold-open again first if the last connection has expired.

Under the socket row you get an indented block of name:value pairs. It is dense, and which pairs appear depends on your kernel version and on what the connection has actually done, so read your own output rather than expecting a fixed list. The second line below is written as a shape, with the values labelled rather than filled in, for exactly that reason:

sudo ip netns exec server ss -tin

prompt: student@sockets-lab:~$ answer: sudo ip netns exec server ss -tin ||| sudo ip netns exec server ss -tni ||| sudo ip netns exec server ss -t -i -n output: State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 10.0.1.20:80 10.0.1.10:41234 cubic rtt:<ms>/<ms> mss:<bytes> pmtu:1500 cwnd:<segments> bytes_sent:<n> bytes_received:<n> hint: The connected-socket command with the info flag added: sudo ip netns exec server ss -tin

These are the pairs worth finding in your own output:

Field	What it means
`cubic`	The congestion control algorithm in use. CUBIC is the Linux default.
`rtt:`	Round-trip time in milliseconds, as smoothed average and variation.
`cwnd:`	Congestion window, counted in segments. How much TCP will send before it waits for an acknowledgment.
`mss:`	Maximum segment size: the largest chunk of data this connection puts in one packet.
`pmtu:`	Path MTU, the largest frame the path will carry. On this veth link it is 1500.
`bytes_sent:` / `bytes_received:`	Totals for the life of the connection.
`retrans:`	Retransmissions, current and total. Anything above zero means packets were lost.

This is the kind of information that matters when you are debugging a slow service. A high rtt means latency on the path. A small cwnd means TCP is being cautious, either because the connection just started or because it saw loss. A non-zero retrans means the network is dropping packets.

Finding cwnd and rtt in that block is the fourth of the five things your lab grades.

The numbers here are shaped by the link. This lab is a virtual cable between two namespaces on the same machine, so the round-trip time is a fraction of a millisecond and nothing gets lost. Over the internet the same fields would show tens or hundreds of milliseconds.

Challenge: name the program behind the port

Scaffolding off. No command is printed from here on.

Here is the situation you will meet for real. A web server refuses to start because something is already sitting on port 80, and nobody knows what. Guessing is not a plan. Go back to the listing you ran first, and add the one flag that turns a port number into a program.

The flag is -p, for process. The pid and fd values below are labelled rather than printed, because they are different numbers on every boot:

prompt: student@sockets-lab:~$ answer: sudo ip netns exec server ss -tlnp ||| sudo ip netns exec server ss -tnlp ||| sudo ip netns exec server ss -t -l -n -p output: State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 5 0.0.0.0:22 0.0.0.0:* users:(("socat",pid=<pid>,fd=<fd>)) LISTEN 0 5 0.0.0.0:80 0.0.0.0:* users:(("python3",pid=<pid>,fd=<fd>)) LISTEN 0 5 0.0.0.0:9000 0.0.0.0:* users:(("socat",pid=<pid>,fd=<fd>)) hint: The listing command from the start of the lesson, with the process flag added to the end of the flags.

Same three listening sockets as the first task, plus a Process column on the end. Each entry tells you three things:

The program name, python3 for the web server and socat for the other two services
The process ID, pid=, which will be a different number on your VM
The file descriptor, fd=, which is the number that process uses internally for this socket

On a real server, this is how you find out what is holding a port. If your web server will not start because something is already on port 80, ss -tlnp names

Practice Ports and Sockets in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.