Learn › The Network Stack › How Packets Find Their Way
sysctl - a hands-on Linux lab on a real virtual machine.
Flip one bit and your Linux box becomes a router. /proc/sys/net/ipv4/ip_forward controls everything.
This is the same network you walked through in the routing table lesson. The same client at 10.0.1.10, the same server at 10.0.2.20, the same box in the middle with one leg in each subnet.
Every routing table is correct. The client knows to hand cross-subnet traffic to 10.0.1.1. The server knows to answer through 10.0.2.1. Every interface is up. Every link is live.
And the client cannot reach the server. Not slowly, not partially. 100 percent packet loss, and not one error message anywhere.
The middle box is not broken. It is refusing. One kernel setting, one character long, decides whether a Linux machine is willing to carry other machines' traffic. By the end of this lesson you will have read that setting, watched it drop packets in silence, flipped it, and captured the exact moment a packet crosses from one subnet to the other.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The real machines come at the end. The lab user is student, the host is forward-lab, and your progress in the lab is tracked automatically, so type commands naturally.
[client] [router] [server]
10.0.1.10 <----> 10.0.1.1 10.0.2.1 <----> 10.0.2.20
Subnet A (10.0.1.0/24) Subnet B (10.0.2.0/24)Three machines, built as network namespaces: isolated copies of the Linux network stack, each with its own interfaces and its own routing table, all inside one VM. You have built these in earlier labs. They behave exactly like three physical boxes.
veth-rc faces the client, veth-rs faces the server.So every packet from client to server must pass through the router. That word, through, is the whole lesson.
Every machine running Linux is one of two things at any moment.
A host (an endpoint, like a laptop or a web server) accepts packets addressed to it and drops everything else. A packet arrives whose destination IP is not one of the machine's own addresses? Not my problem. Discarded.
A router does the opposite job: it accepts packets addressed to someone else, looks up the destination in its routing table, and sends them out another interface toward the next hop.
The switch between those two identities is a single kernel value: /proc/sys/net/ipv4/ip_forward. The /proc/sys directory is not files on disk. It is the kernel's live settings, dressed up as files: reading one asks the kernel a question, writing one changes its behavior immediately. The sysctl tool is the polite door to the same values: it turns a dotted name like net.ipv4.ip_forward into the path /proc/sys/net/ipv4/ip_forward and reads or writes it for you.
Zero means host. One means router. That is the entire mechanism.
The default was not always off. Early Unix workstations shipped with forwarding enabled, so any box that grew a second network interface silently became a router. Campus networks of the 1980s developed mystery paths through somebody's desk machine, and traffic died whenever that machine was switched off for the night.
The Host Requirements RFC, RFC 1122, ended that era in 1989: a machine acting as a host must not forward packets, full stop. A separate rulebook, RFC 1812, defines what a real router owes the network. The Linux kernel documentation still describes ip_forward in exactly those terms: the host defaults of RFC 1122 on one side of the switch, the router duties of RFC 1812 on the other.
So a fresh Linux install boots as a polite host. Your lab router is polite, and that is the entire outage.
First move in any forwarding problem: read the switch, never assume it. The helper check-forward runs the raw read for you inside the router namespace:
sudo ip netns exec router cat /proc/sys/net/ipv4/ip_forward
Before you press Enter, commit: this router is refusing to forward, so what single character will come back?
prompt: student@forward-lab:~$ answer: check-forward ||| sudo ip netns exec router cat /proc/sys/net/ipv4/ip_forward output: 0 hint: The helper is check-forward. The raw form is cat on /proc/sys/net/ipv4/ip_forward, inside the router namespace.
One character. That zero is not a status report generated by some service. It is the live value the kernel consults for every single packet that arrives addressed to someone else.
There is no file on disk behind it. Power the machine off and the value is gone, which will matter later in this lesson.
Long /proc/sys paths are a tab completion gift. Type /proc/sys/net/ipv4/ip_f and press Tab: the shell finishes ip_forward for you. Helper names complete too: check-f plus Tab. Engineers who tab complete do not typo sysctl paths.
Now the door engineers actually use day to day. sysctl takes the dotted name, and ip netns exec router runs the command inside the router's network stack, exactly as you did in the routing table lesson:
sudo ip netns exec router sysctl net.ipv4.ip_forward
prompt: student@forward-lab:~$ answer: sudo ip netns exec router sysctl net.ipv4.ip_forward output: net.ipv4.ip_forward = 0 hint: sysctl with the dotted key, run inside the router namespace: sudo ip netns exec router sysctl net.ipv4.ip_forward
Same value, friendlier frame. The dots in net.ipv4.ip_forward map one to one onto the slashes in /proc/sys/net/ipv4/ip_forward. That mapping is the entire relationship between sysctl and /proc/sys: there is exactly one set of values, with two doors in front of it.
Remember the mapping and you can navigate settings you have never seen. Any sysctl name you meet in a runbook is also a path you can cat.
The client is about to send a ping to 10.0.2.20. The packet will arrive at the router, which owns neither that address nor permission to forward. Take a position before you watch it happen.
>>> It drops the packet in silence. No ICMP error, no log line, nothing. The first answer is a fair guess because routers do send ICMP errors in other failures: no route to the destination earns a destination unreachable, and a dead TTL earns a time exceeded. But a machine with forwarding off is not a router failing to route. It is a host, and hosts treat transit packets as not their problem. The third answer describes nothing real: kernels buffer packets for milliseconds on a busy link, but no kernel parks traffic waiting for an administrator to change a setting.
Now watch that silence from the client's side. The helper client-ping runs:
sudo ip netns exec client ping -c 3 -W 2 10.0.2.20
Three pings, two seconds of patience each. You committed to silence, so decide: how many of the three come back?
prompt: student@forward-lab:~$ answer: client-ping ||| sudo ip netns exec client ping -c 3 -W 2 10.0.2.20 output: PING 10.0.2.20 (10.0.2.20) 56(84) bytes of data.
--- 10.0.2.20 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 4003ms hint: The helper client-ping sends three pings from the client to the server at 10.0.2.20.
Read the statistics line like an engineer. 3 packets transmitted means the client's kernel found a route and sent all three: the failure is not local. 0 received and 100% packet loss mean they died somewhere past the first hop. The run took about four seconds for three one-second pings, because after the last request ping still waits out the -W 2 timeout hoping for a reply. Waiting is what a lost packet costs you.
Now read what is not there. No Destination Host Unreachable. No Network is unreachable. No error of any kind, because the router sent none. From where the client sits, a disabled forwarder, a firewall DROP rule, and a cut cable beyond the first hop all look exactly like this. The silence is the signature.
One more reading before the fix, because it proves where the wall is. The helper client-ping-router pings the router's near address, 10.0.1.1, from the client. Same client, same wire, same router. The only difference is the destination address.
prompt: student@forward-lab:~$ answer: client-ping-router ||| sudo ip netns exec client ping -c 3 10.0.1.1 output: PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=0.035 ms 64 bytes from 10.0.1.1: icmp_seq=2 ttl=64 time=0.048 ms 64 bytes from 10.0.1.1: icmp_seq=3 ttl=64 time=0.041 ms
--- 10.0.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2041ms rtt min/avg/max/mdev = 0.035/0.041/0.048/0.005 ms hint: client-ping-router pings the router's near-side address, 10.0.1.1.
Perfect connectivity. These packets were addressed to the router, so they were delivered up the router's own stack and answered. ip_forward was never consulted, because nothing needed forwarding.
This is the to versus through distinction, and it is the diagnostic move to keep: when a gateway answers pings but nothing beyond it does, stop suspecting cables and start suspecting the gateway's willingness to forward.
File away one more detail: ttl=64. Hold that number for the next step.
TTL, time to live, is the hop counter you met in the ICMP lesson: every IPv4 packet starts with a number, and Linux starts it at 64. You just saw the router answer with ttl=64.
>>> 63. Every IPv4 router decrements TTL by exactly one before sending a packet onward. The server's reply leaves at 64, crosses one router, arrives at 63. If you picked 62, you were half right about the round trip: the request also paid a hop and reached the server at 63, but ping prints the TTL of the reply as it arrived, and the reply started fresh at 64. If you picked 64, TTL never waits for loss: it is decremented at every forward precisely so a looping packet dies at zero, at which point the router that killed it sends back an ICMP time exceeded. That death notice is the entire working principle of traceroute.
Time to turn the middle box into a router. The real command writes a 1 through the sysctl door:
sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1
The helper enable-forward runs exactly that. Use either.
prompt: student@forward-lab:~$ answer: enable-forward ||| sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1 ||| sudo ip netns exec router sysctl net.ipv4.ip_forward=1 output: net.ipv4.ip_forward = 1 hint: enable-forward, or sysctl -w net.ipv4.ip_forward=1 inside the router namespace.
sysctl -w echoes the new value as confirmation, and the change is already live: the very next packet to arrive gets forwarded. No service restarted, no daemon reloaded, nothing rebooted. You reached into a running kernel and changed its identity mid-flight.
One thing the kernel documentation flags about this exact key: it is special. Writing ip_forward does not set one value, it fans out, setting the forwarding flag on every interface at once. Finer control exists as net.ipv4.conf.<interface>.forwarding, one key per interface, for routers that should forward on some legs only.
You could also have written the raw file: echo 1 > /proc/sys/net/ipv4/ip_forward inside the namespace. Same value, same effect, no confirmation printed.
Prove that fan-out instead of trusting it. You never touched the client-facing interface's own key. Read it:
sudo ip netns exec router sysctl net.ipv4.conf.veth-rc.forwarding
prompt: student@forward-lab:~$ answer: sudo ip netns exec router sysctl net.ipv4.conf.veth-rc.forwarding output: net.ipv4.conf.veth-rc.forwarding = 1 hint: Same sysctl read, but the key is net.ipv4.conf.veth-rc.forwarding.
It reads 1, and you never wrote it. Your single global write propagated to every per-interface flag, which is why most engineers go years without meeting these keys.
They matter the day the design is asymmetric: a VPN box that should forward between tunnel and LAN but never toward the management interface, for example. And the fan-out cuts both ways: writing the global ip_forward again, in either direction, stomps every per-interface choice you made. On a box with deliberate per-interface settings, the global key is a loaded weapon.
Same command as before the flip. You committed to a TTL two steps ago. Collect your evidence.
prompt: student@forward-lab:~$ answer: client-ping ||| sudo ip netns exec client ping -c 3 -W 2 10.0.2.20 output: PING 10.0.2.20 (10.0.2.20) 56(84) bytes of data. 64 bytes from 10.0.2.20: icmp_seq=1 ttl=63 time=0.078 ms 64 bytes from 10.0.2.20: icmp_seq=2 ttl=63 time=0.092 ms 64 bytes from 10.0.2.20: icmp_seq=3 ttl=63 time=0.085 ms
--- 10.0.2.20 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.078/0.085/0.092/0.006 ms hint: The same cross-subnet ping as before: client-ping.
Hold this against the two earlier screens. The dead run: 100 percent loss. The router itself: ttl=64. This: ttl=63.
That 63 is a receipt. It proves the reply did not come from something on the client's own subnet: it was born at 64 one subnet away and paid exactly one router to get here. When you ping a machine and the TTL is a few notches below a round default (64, 128 for Windows, 255 for many network devices), the gap is the router count between you.
Stop and weigh the moment. With one written character, you changed a machine's species. Everything the routing table lesson taught, next hops, default routes, longest match, was already in place. This lesson supplied the missing ingredient: permission.
You have now performed, by hand, the exact move that infrastructure performs behind your back:
net.ipv4.ip_forward=1 when it starts, because containers live behind a bridge and their traffic must be forwarded to the world. Run the read on any container host and you will find a 1 someone else put there.kubeadm preflight checks fail if the value is 0, because pod traffic is forwarded traffic.So when a container or a VPN client can reach its own gateway but nothing beyond, you now check one thing first, and you know why.
You proved forwarding from the outside. Now watch it from the inside. From the traffic lesson you know tshark prints live packets, and the helper cap-router runs it inside the router namespace on all interfaces at once. The & runs the capture in the background so you can ping while it listens:
cap-router -c 4 & then client-ping
One ping needs exactly four captured frames. Work out why four before you look.
prompt: student@forward-lab:~$ answer: cap-router -c 4 ||| cap-router -c 4 & ||| sudo ip netns exec router tshark -i any -c 4 output: Running as user "root" and group "root". This could be dangerous. Capturing on 'any' 1 0.000000000 10.0.1.10 → 10.0.2.20 ICMP 104 Echo (ping) request id=0x4e21, seq=1/256, ttl=64 2 0.000021450 10.0.1.10 → 10.0.2.20 ICMP 104 Echo (ping) request id=0x4e21, seq=1/256, ttl=63 3 0.000063118 10.0.2.20 → 10.0.1.10 ICMP 104 Echo (ping) reply id=0x4e21, seq=1/256, ttl=64 4 0.000084029 10.0.2.20 → 10.0.1.10 ICMP 104 Echo (ping) reply id=0x4e21, seq=1/256, ttl=63 4 packets captured hint: cap-router -c 4 captures four frames inside the router namespace. Add & to run it in the background, then client-ping.
Ignore the scary first line: tshark warns whenever it runs as root, and in a lab namespace that is fine. The four frames are the payoff. Your timings and id will differ; the pattern will not.
One ping, four frames, because a capture on any sees the packet twice: once arriving on veth-rc, once leaving on veth-rs. Frames 1 and 2 are the same request, and read the TTL across the pair: in at 64, out at 63. Frames 3 and 4 are the reply making the return trip, in at 64, out at 63 again. You are not being told routers decrement TTL anymore. You are watching one do it.
Notice also what is absent: no process on the router touched any of this. Nothing was listening, no socket was involved, no application ran. Forwarding is pure kernel work between two interfaces, and a packet capture is the only witness it leaves.
Real outages are rarely this clean. On production machines, three other gatekeepers can drop transit packets while ip_forward reads a perfectly healthy 1. This is the checklist experienced engineers run, in order.
The FORWARD firewall chain. The to versus through distinction applies to firewalls too: rules guarding packets addressed to the box live in one chain (INPUT), and transit packets are judged by a separate FORWARD chain. A machine can accept your SSH session and silently drop everything through it. Docker is the famous case: when the daemon starts it sets the FORWARD policy to drop, which has broken countless VPN servers sharing a host with containers. The next lesson, packet filtering, is exactly this layer.
Reverse path filtering. With net.ipv4.conf.all.rp_filter=1, the kernel asks, for every arriving packet: would my route back to this source leave through the interface it arrived on? If not, the packet is dropped, silently. This kills spoofed traffic, and it also kills legitimate asymmetric routing, where requests and replies take different paths. Loose mode, rp_filter=2, only requires the source to be reachable somewhere.
Martians. A martian is a packet whose address cannot legitimately appear on the link it arrived on, named for packets that seem to come from Mars. Set net.ipv4.conf.all.log_martians=1 and the kernel confesses its silent drops to the kernel log, like IPv4: martian source 10.0.2.20 from 10.99.0.5, on dev veth-rc. Read that line carefully: the first address is the destination, the address after from is the offending source. Nearly everyone reverses it on first read.
One relative of forwarding worth knowing by name: when a
Practice IP Forwarding in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.