TCP Tuning

sysctl -a | grep ^net.ipv4.tcp - a hands-on Linux lab on a real virtual machine.

The kernel exposes dozens of TCP tuning knobs via sysctl. Buffer sizes, keepalive timers, backlog queues. This is what SREs adjust on production servers.

The settings nobody set

In the last lesson, you learned that TCP connections move through states. But the kernel does not just track states. It also manages buffers, timers, and algorithms that control how fast data flows, how long idle connections stay open, and how many new connections the server can accept at once.

Nobody on your team chose those values. They are defaults, sitting in the kernel, quietly deciding things. One of them means an idle connection can be dead for over two hours before Linux notices. Another decides how many callers can be queued at your front door before the kernel starts turning them away.

These settings are exposed through a system called sysctl. Every Linux machine has dozens of TCP tuning knobs you can read and adjust. This is what SREs (Site Reliability Engineers) look at when a production server is dropping connections or running slowly.

The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The real VM comes at the end, under Ready to practice. Your progress in the real lab is tracked automatically, so type commands naturally and watch the checkmarks appear.

What is sysctl?

sysctl is a command that reads and writes kernel parameters at runtime. The kernel stores these parameters in a virtual filesystem at /proc/sys/. The sysctl command is just a convenient way to read and write files in that directory.

For example, these two commands do the same thing:

sysctl net.ipv4.tcp_syn_retries
cat /proc/sys/net/ipv4/tcp_syn_retries

The dots in the sysctl name map to slashes in the file path. net.ipv4.tcp_syn_retries becomes /proc/sys/net/ipv4/tcp_syn_retries.

Why a kernel setting is a file

That mapping is not a convenience somebody bolted on. It is a design decision, and an old one.

Unix research systems in the 1980s introduced the idea that a running kernel should describe itself as ordinary files, and Linux took the idea further than anyone: under /proc/sys every tunable is a file you can read with cat and write with a redirect. No API, no special tool, no format to learn. The sysctl command, whose name comes from a BSD command of the same job, adds the dotted naming scheme and the ability to dump everything at once, and that is all it adds.

The practical consequence shows up the first time you have to explain a production change. A setting that is a file is a setting you can list, diff, put in configuration management and paste into a ticket. It is also why the permanent form of a tuning change is a file under /etc/sysctl.d/ rather than a command somebody typed once and forgot.

Your lab network

This lab has the same two-namespace network as previous lessons:

  [client]  10.0.1.10  <-------->  10.0.1.20  [server]
  MAC: 02:00:0a:00:01:0a           MAC: 02:00:0a:00:01:14

The server runs HTTP on port 80. You will inspect kernel parameters that control how TCP behaves.

Parameter names are long and easy to fumble. Use the file form and let the shell finish them: type cat /proc/sys/net/ipv4/tcp_keep and press Tab, and the shell completes the name, because it is a real path on a real filesystem. Tab twice after tcp_ and it lists every parameter starting that way. That is the fastest parameter browser on the machine, and it works before you know what you are looking for.

Commit: ask for the whole TCP family at once

There are dozens of TCP parameters and they all begin net.ipv4.tcp. So the obvious move is to ask sysctl for that prefix and see what comes back. Take a position on what it will do.

[ ] It prints every parameter that starts with that prefix.
[x] It fails, because that name is not a file and sysctl names are file paths.
[ ] It prints nothing at all and exits quietly.

>>> It fails. A sysctl name is the path to one file, and net.ipv4.tcp is not a file. It is only the first part of a lot of names, so there is nothing at that path to read and sysctl says so: sysctl: cannot stat /proc/sys/net/ipv4/tcp: No such file or directory. If you picked the prefix listing, that is a reasonable expectation from other tools, and the fix is one flag away. If you picked silence, remember the file model: asking for a file that does not exist is an error everywhere in Linux, not an empty answer.

Listing TCP parameters

To list a group of parameters, dump them all and filter. -a means "every parameter", and grep keeps the lines you want. The ^ means "only lines that start with this", so you get the TCP parameters and nothing else.

You will see a long list. The screen below is trimmed to ten representative lines out of it. These are typical defaults on a Debian system, and a few of them scale with how much memory the machine has:

sysctl -a | grep ^net.ipv4.tcp

prompt: student@tuning-lab:~$ answer: sysctl -a | grep ^net.ipv4.tcp ||| sudo sysctl -a | grep ^net.ipv4.tcp ||| /usr/sbin/sysctl -a | grep ^net.ipv4.tcp output: net.ipv4.tcp_abort_on_overflow = 0 net.ipv4.tcp_allowed_congestion_control = reno cubic net.ipv4.tcp_congestion_control = cubic net.ipv4.tcp_fin_timeout = 60 net.ipv4.tcp_keepalive_intvl = 75 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_tw_reuse = 2 net.ipv4.tcp_wmem = 4096 16384 4194304 hint: Dump every parameter with -a, then pipe into grep with ^ to keep the lines that start with the TCP prefix: sysctl -a | grep ^net.ipv4.tcp

That is a lot of knobs, and the real list is longer than this. Do not panic. Most of the defaults are fine for typical workloads. Understanding the important ones is what lets you diagnose a problem instead of guessing.

Read the shape of the list rather than the values for a moment. Some parameters are single numbers, some are words (cubic), and two of them are three numbers in a row. Those triples are the buffer settings, and they get a step of their own shortly. Listing this group is the first of the five things your lab grades.

You can also list a whole subtree by naming the directory it lives in: ls /proc/sys/net/ipv4/ prints one filename per parameter. Every sysctl name is just that path with the slashes turned into dots.

Keepalive timers

When a TCP connection is idle (no data flowing), how does the kernel know if the other side is still alive? It uses keepalive probes. Three parameters control this:

sysctl net.ipv4.tcp_keepalive_intvl
sysctl net.ipv4.tcp_keepalive_probes

Start with the third one, the timer that decides when the kernel first gets suspicious:

sysctl net.ipv4.tcp_keepalive_time

prompt: student@tuning-lab:~$ answer: sysctl net.ipv4.tcp_keepalive_time ||| sudo sysctl net.ipv4.tcp_keepalive_time ||| cat /proc/sys/net/ipv4/tcp_keepalive_time ||| /usr/sbin/sysctl net.ipv4.tcp_keepalive_time output: net.ipv4.tcp_keepalive_time = 7200 hint: sysctl followed by the full dotted name of the parameter: sysctl net.ipv4.tcp_keepalive_time

Run the other two and you have the full picture:

net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9

Here is what each one means:

Parameter	Default	Meaning
`tcp_keepalive_time`	7200	Seconds to wait before sending the first keepalive probe (2 hours)
`tcp_keepalive_intvl`	75	Seconds between subsequent keepalive probes
`tcp_keepalive_probes`	9	Number of unanswered probes before the kernel gives up

Reading any one of the three is the second of the five things your lab grades.

Keepalive only applies to connections where the application has enabled the SO_KEEPALIVE socket option. Not every application enables it. But when it is enabled, these kernel parameters control the timing.

Commit: how long until the kernel notices a dead peer?

Picture a connection that is open, idle, and quietly doomed: the machine on the other end lost power, so nothing will ever arrive from it again and nothing has been sent for a while. You have the three numbers. Do the arithmetic before you read on.

[ ] About 75 seconds, the interval between probes.
[x] Just over two hours, because the first probe does not go out for 7200 seconds.
[ ] Immediately. The kernel notices as soon as a packet goes unacknowledged.

>>> Just over two hours. With the defaults: after 2 hours of idle time, the kernel sends a keepalive probe. If it gets no response, it sends another probe every 75 seconds, up to 9 times. If all 9 fail, the kernel closes the connection. Total time to detect a dead connection with defaults: 7200 + (75 * 9) = 7875 seconds (about 2 hours 11 minutes). If you picked 75 seconds, that is the gap between probes and it only starts mattering after the first two hours have passed. If you picked immediately, notice the word idle: nothing is being sent, so there is nothing to go unacknowledged, and that is exactly why keepalive exists at all.

On a production server handling real-time connections, 2 hours is too long. SREs often reduce tcp_keepalive_time to 60-300 seconds so dead connections are detected faster.

Buffer sizes: tcp_rmem and tcp_wmem

TCP uses buffers to hold data before it is sent (write buffer) and after it is received but before the application reads it (read buffer). Two parameters control these, and each one answers with three numbers instead of one:

sysctl net.ipv4.tcp_wmem

Read the receive buffer first. Before you press Enter, ask yourself why a buffer setting would need three values rather than one:

sysctl net.ipv4.tcp_rmem

prompt: student@tuning-lab:~$ answer: sysctl net.ipv4.tcp_rmem ||| sudo sysctl net.ipv4.tcp_rmem ||| cat /proc/sys/net/ipv4/tcp_rmem ||| /usr/sbin/sysctl net.ipv4.tcp_rmem output: net.ipv4.tcp_rmem = 4096 131072 6291456 hint: sysctl followed by the full dotted name of the receive buffer parameter: sysctl net.ipv4.tcp_rmem

Three numbers, and the send buffer answers the same way:

net.ipv4.tcp_rmem = 4096    131072    6291456
net.ipv4.tcp_wmem = 4096    16384     4194304

The three numbers are:

Position	Name	Meaning
First	min	Minimum buffer size in bytes. The kernel always allocates at least this much.
Second	default	Default buffer size for new connections.
Third	max	Maximum buffer size. The kernel can grow the buffer up to this limit if needed.

For tcp_rmem (receive buffer): the default shows 4096 / 131072 / 6291456, which means 4KB minimum, 128KB default, 6MB maximum.

For tcp_wmem (write/send buffer): the default shows 4096 / 16384 / 4194304, which means 4KB minimum, 16KB default, 4MB maximum.

That is why there are three: the kernel auto-tunes these buffers based on available memory and network conditions, so the setting is a range it is allowed to move inside rather than a fixed size. On a server handling large file transfers or high-bandwidth connections, you might increase the max values. On a server handling millions of small connections, you might decrease the default to save memory. Reading either buffer parameter is the third of the five things your lab grades.

The kernel will not exceed the global maximums set by net.core.rmem_max and net.core.wmem_max. If you increase tcp_rmem or tcp_wmem max values, also check the core limits.

Milestone: you can read the machine's own settings

Stop and count. You can list a whole family of kernel parameters, read any single one by name, and complete the name with Tab when you only half remember it. You know the arithmetic behind a timer that most engineers have never added up, and you know why a buffer setting has three numbers in it.

Everything so far has been global: the values every connection on the machine starts from. Two more parameters decide what happens when a server gets busy, and then you will look at what one live connection actually settled on.

Commit: recycling TIME-WAIT sockets

Recall from the last lesson that TIME-WAIT sockets linger for about 60 seconds after a connection closes. A web server under heavy load has tens of thousands of them left over from inbound connections, and somebody has read that net.ipv4.tcp_tw_reuse recycles TIME-WAIT sockets.

[ ] Yes. It recycles TIME-WAIT sockets, which is exactly what is piling up.
[x] No. It only applies to connections this machine starts, never to inbound ones.
[ ] No, because the kernel ignores the setting on modern versions.

>>> No. It only affects outgoing connections, the ones this machine starts. It cannot recycle the TIME-WAIT sockets left behind by inbound connections to a server, which is the case people usually hope it will fix. If you picked yes, you are in large company, and this is one of the most copy-pasted wrong fixes in production tuning. If you picked the third answer, the setting is very much alive: it is genuinely useful on a machine that makes many outgoing connections, such as a proxy or an application server talking to a database, where running out of client ports is a real risk.

Connection reuse: tcp_tw_reuse

This parameter controls whether the kernel can reuse TIME-WAIT sockets for new outgoing connections. On a busy server making many outgoing connections, you can run out of available port numbers because they are all stuck in TIME-WAIT.

Read the value on this machine:

sysctl net.ipv4.tcp_tw_reuse

prompt: student@tuning-lab:~$ answer: sysctl net.ipv4.tcp_tw_reuse ||| sudo sysctl net.ipv4.tcp_tw_reuse ||| cat /proc/sys/net/ipv4/tcp_tw_reuse ||| /usr/sbin/sysctl net.ipv4.tcp_tw_reuse output: net.ipv4.tcp_tw_reuse = 2 hint: sysctl followed by the full dotted name of the reuse parameter: sysctl net.ipv4.tcp_tw_reuse

A 2, which is not a yes or a no. This is a three-value setting:

Value	Meaning
0	Disabled. TIME-WAIT sockets are never reused.
1	Enabled everywhere. The kernel may reuse a TIME-WAIT socket for a new outgoing connection when it is safe to do so.
2	Enabled for loopback traffic only. This is the default on current kernels.

So this machine reuses TIME-WAIT sockets for traffic that never leaves the box, and leaves everything else alone. Reading this parameter, or the backlog one coming next, is the fourth of the five things your lab grades.

Challenge: the queue at the front door

Scaffolding off. No command is printed from here on.

A busy web server is refusing connections under load even though the machine has plenty of CPU and memory left. The suspect is the listen backlog: the queue of connections that have completed the three-way handshake but have not yet been accepted by the application. If this queue fills up, new connections are dropped.

The kernel limit on that queue is called somaxconn, and it is not a TCP parameter. It lives in the net.core family, because the limit belongs to sockets in general rather than to TCP alone. Read it.

prompt: student@tuning-lab:~$ answer: sysctl net.core.somaxconn ||| sudo sysctl net.core.somaxconn ||| cat /proc/sys/net/core/somaxconn ||| /usr/sbin/sysctl net.core.somaxconn output: net.core.somaxconn = 4096 hint: The same lookup form as every parameter so far, with the family name in the middle changed from ipv4 to core.

The default was 128 on older kernels, which was a common bottleneck on busy web servers. Modern kernels default to 4096. High-traffic servers may increase this to 65535.

Note the family in the name. net.ipv4. parameters are TCP and IP specific; net.core. parameters apply to the socket layer underneath, which is also where rmem_max and wmem_max live. When a parameter you expect under net.ipv4 is not there, net.core is the next place to look.

Setting somaxconn higher than your application's listen backlog does nothing. The effective backlog is the smaller of somaxconn and the value passed to listen() in the application code.

Challenge: what one connection actually settled on

Still no command shown.

All the sysctl parameters you just looked at are global defaults. Each individual connection then keeps its own live numbers, and those are the ones that answer the question of why this connection, right now, is slow.

Two things to get right first. The lab network lives in network namespaces, separate network stacks inside this one VM, so ss only sees those sockets when you run it inside one. And the detail flag has nothing to show unless a connection is actually open, so run hold-open first, which opens a connection to port 80 and keeps it there for 60 seconds.

hold-open

Now list the TCP sockets inside the server namespace with the kernel's internal information attached. You used this exact combination in the Ports and Sockets lesson: the TCP flag, the numeric flag, and the flag that means info.

Under the socket row you get an indented block of name:value pairs. Which pairs appear depends on your kernel version and on what the connection has actually done, so the second line below is written as a shape with the values labelled, and you should read your own output rather than expecting a fixed list:

prompt: student@tuning-lab:~$ answer: sudo ip netns exec server ss -tin ||| sudo ip netns exec server ss -tni ||| sudo ip netns exec server ss -t -i -n output: State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 10.0.1.20:80 10.0.1.10:<client port> cubic wscale:<snd>,<rcv> rtt:<ms>/<ms> mss:<bytes> pmtu:1500 cwnd:<segments> bytes_sent:<n> bytes_received:<n> hint: Inside the server namespace, TCP sockets, numeric, and the info flag that adds the kernel internals.

These are the pairs worth finding in your own output:

Field	What it means
`cubic`	The congestion control algorithm in use. CUBIC is the Linux default, and it is the `tcp_congestion_control` value from the list above.
`rtt:`	Round-trip time in milliseconds, printed as smoothed average and variation.
`minrtt:`	The lowest round-trip time seen on this connection.
`cwnd:`	Congestion window, counted in segments. How much TCP will send before it waits for acknowledgments.
`mss:`	Maximum segment size: the largest chunk of data this connection puts in one packet.
`pmtu:`	Path MTU, the largest frame the path will carry. On this virtual link it is 1500.
`wscale:`	The window scale factors agreed during the handshake, send and receive.
`bytes_sent:` / `bytes_received:`	Totals for the life of the connection.
`retrans:`	Retransmissions, current and total. Anything above zero means packets were lost.

The congestion window is the one to understand. It controls how much data TCP is willing to have in flight before it waits for an acknowledgment. A small cwnd means the connection is being cautious, either because it has just started or because it saw loss. A large cwnd means it has ramped up and is pushing hard.

The round-trip time tells you how long a packe

Practice TCP Tuning in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.