Learn › The Network Stack › How Packets Find Their Way
ip route - a hands-on Linux lab on a real virtual machine.
The kernel checks this table for every single packet. ip route get shows the exact decision for one destination.
On this VM, a machine called the client is about to send a packet to 10.0.2.20. Here is the problem. The client lives on the 10.0.1.0/24 network, and nothing on that network holds the address 10.0.2.20. No neighbor will answer for it. As far as the client's own subnet is concerned, the destination does not exist.
The packet will leave anyway, and it will arrive.
The thing that makes that possible is a table inside the kernel, and the client's copy of it is exactly two lines long. By the end of this lesson you will read that table line by line, ask the kernel to show its decision for any address before anything is sent, watch the packet cross two networks, and decode the same table in the raw hex the kernel keeps.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The screens show what this lab's own fixture produces, so what you read here is what the real VM will show you at the end. Your progress in the lab is tracked automatically, so type commands naturally.
A routing table is a list of rules the kernel keeps for outgoing traffic. Each rule, called a route, says: for addresses in this network, send packets out of this interface, and optionally hand them to this gateway first. A gateway is a machine that agrees to forward packets onward on your behalf. Gateway and router are two words for the same machine.
For every single packet this machine sends, the kernel consults the table and picks exactly one winning rule. There is no second mechanism. If no rule claims a destination, the packet is never sent at all, and you will see exactly what that looks like before this lesson ends.
Every operating system keeps a routing table. On Linux you read it with ip route show. The machines at the core of the internet play the same game at a different scale: a core router carries close to a million routes. The client in this lab carries two. Both answer every lookup with the same rule, which you are about to learn.
The first machine built for this job, the ARPANET's Interface Message Processor of 1969, was a refrigerator-sized cabinet, and the network it routed for had four computers on it. The job has not changed since: read the destination, look it up in a table, send it one hop closer.
[client] [router] [server]
10.0.1.10/24 --> 10.0.1.1 | 10.0.2.1 <-- 10.0.2.20/24
veth-c veth-rc veth-rs veth-sThree network namespaces are running on this VM. A network namespace is a private copy of the Linux network stack with its own interfaces and, the point of today, its own routing table. You have worked inside them since earlier labs in this track. Treat each one as a separate machine.
The client is on subnet 10.0.1.0/24. The server is on 10.0.2.0/24. Those are two different networks, so the client cannot deliver to the server directly. The router namespace has one leg in each subnet, 10.0.1.1 on the client side and 10.0.2.1 on the server side, and it forwards packets between them. The client's default gateway is 10.0.1.1. The server's is 10.0.2.1.
To run a command inside a namespace, wrap it the same way as in the ARP lab: sudo ip netns exec client ... runs one command as if you were sitting on the client.
Two habits save typing all lesson. Tab completion works on paths, so cat /proc/net/ro then Tab finishes the filename. And ip accepts any unambiguous abbreviation of its words: ip r means ip route, and sudo ip -n client route is the short spelling of sudo ip netns exec client ip route.
Start where the packet starts. Before you press Enter, commit to a number: the client has one interface and one gateway. How many lines do you expect its entire routing table to be?
sudo ip netns exec client ip route show
prompt: student@routing-lab:~$ answer: sudo ip netns exec client ip route show ||| sudo ip netns exec client ip route ||| sudo ip netns exec client ip r ||| sudo ip -n client route show output: default via 10.0.1.1 dev veth-c 10.0.1.0/24 dev veth-c proto kernel scope link src 10.0.1.10 hint: Wrap the route reader in the namespace runner: sudo ip netns exec client ip route show
Two lines. That is the client's complete knowledge of the world.
The second line appeared on its own the moment the address 10.0.1.10/24 was assigned to veth-c. A route born that way is called a connected route. The kernel reasons: if this machine holds an address inside 10.0.1.0/24, that whole subnet must be directly attached, so deliver to it without help.
The first line is the default route, the catch-all. This lab's setup added it, the same way DHCP quietly adds one on your laptop. Any destination no other line claims goes to 10.0.1.1 and becomes the router's problem.
The connected route first:
10.0.1.0/24 dev veth-c proto kernel scope link src 10.0.1.10| Field | Meaning |
|---|---|
10.0.1.0/24 | The destination network this rule claims, all 254 hosts of it |
dev veth-c | Send matching packets out of this interface |
proto kernel | Who installed the route: the kernel itself, during address assignment |
scope link | The destination is directly attached, one hop, no gateway needed |
src 10.0.1.10 | Stamp outgoing packets with this source address |
That last field is easy to skip past, and it quietly runs every connection you make. When a program sends traffic without picking a source address, which is almost always, the kernel copies the src value from the winning route onto the packet. On machines with several addresses, wrong-source mysteries almost always trace back to this one field.
Now the default route:
default via 10.0.1.1 dev veth-c| Field | Meaning |
|---|---|
default | Matches every destination. Internally it is spelled 0.0.0.0/0 |
via 10.0.1.1 | Hand the packet to this gateway |
dev veth-c | The interface through which that gateway is reachable |
proto is a signature. kernel means automatic, from address assignment. static means an admin typed it. dhcp means a lease granted it. When a route surprises you on a real machine, proto is your first clue about who put it there.
The destination 10.0.2.20 matches default, because default matches everything. Suppose the table also held a row for 10.0.2.0/24. Two rows, both matching one packet. Take a position on the tiebreak.
>>> The /24 wins. The rule is longest prefix match: when several rows claim a destination, the kernel uses the one with the most fixed bits, the narrowest network. If you picked the /8, that is backwards: covering more addresses makes a claim vaguer, not stronger. If you picked listing order, ip route show sorts its output for humans, and the kernel does not read the list top to bottom at all. This is also why default loses to everything: it is a /0, zero fixed bits, the vaguest possible claim, which is exactly what makes it a safe catch-all.
On a two-line table you can pick the winner by eye. On a real server's table you cannot, so ip route get exists. It asks the kernel to run the real lookup for one destination and print the verdict. Nothing is transmitted. It is the routing decision as a dry run.
Ask the client how it would reach the server:
sudo ip netns exec client ip route get 10.0.2.20
prompt: student@routing-lab:~$ answer: sudo ip netns exec client ip route get 10.0.2.20 ||| sudo ip netns exec client ip r get 10.0.2.20 ||| sudo ip -n client route get 10.0.2.20 output: 10.0.2.20 via 10.0.1.1 dev veth-c src 10.0.1.10 uid 0 cache hint: The dry-run reader is ip route get, aimed at 10.0.2.20, inside the client namespace.
Read it left to right: to reach 10.0.2.20, go via the gateway 10.0.1.1, out of veth-c, stamping source 10.0.1.10. The word via is the tell. The destination is not on a local subnet, and the packet's first stop will be the router.
uid 0 records which user the lookup ran for, root here because of sudo. It is printed because Linux can hand different users different routes. That is how a phone sends one app through a VPN while the rest of the system goes outside it.
The second line is a fossil. Until 2012 the kernel kept a cache of recent routing decisions, and this command showed you the cache entry. Attackers learned to flood it into uselessness, so kernel 3.6 removed the IPv4 route cache entirely, and every lookup since is computed fresh. The word cache stayed in the output. You are reading a label for a thing that no longer exists.
Now ask about the gateway's own address, 10.0.1.1, which sits inside the client's subnet. Before you run it, decide: will the answer say via?
sudo ip netns exec client ip route get 10.0.1.1
prompt: student@routing-lab:~$ answer: sudo ip netns exec client ip route get 10.0.1.1 ||| sudo ip netns exec client ip r get 10.0.1.1 ||| sudo ip -n client route get 10.0.1.1 output: 10.0.1.1 dev veth-c src 10.0.1.10 uid 0 cache hint: Same dry-run question, aimed at 10.0.1.1 this time.
No via. The connected route won, so delivery is direct.
That one missing word is the whole answer to the question the previous lesson kept asking: local or remote? ip route get settles it per address, straight from the kernel.
It also decides something you learned in the ARP lab. For a direct destination, the client sends an ARP request for the destination itself. For a via destination, the client sends the ARP request for the gateway instead, and the far machine's MAC address never appears on this wire at all. Routed packets are addressed, at the frame level, to the router.
The router forwards every packet that crosses between the two subnets. Before you read its table, take a position on what such a machine must know.
>>> The two connected routes are enough. Each subnet is directly attached to the router, so its table already claims both, and forwarding is a lookup in that same table. If you picked per-machine rows, that is not how routing scales: a row describes a network, and one /24 row covers all 254 hosts on it, which is the only reason core routers survive with about a million rows instead of billions. If you picked default routes, default means everything not otherwise matched, and it cannot point two directions at once. A router joining two subnets simply has no answer for the rest of the world, and shortly you will see what that costs it.
sudo ip netns exec router ip route show
prompt: student@routing-lab:~$ answer: sudo ip netns exec router ip route show ||| sudo ip netns exec router ip route ||| sudo ip netns exec router ip r ||| sudo ip -n router route show output: 10.0.1.0/24 dev veth-rc proto kernel scope link src 10.0.1.1 10.0.2.0/24 dev veth-rs proto kernel scope link src 10.0.2.1 hint: The same table reader, run inside the router namespace.
Two connected routes and nothing else. No default route. This router can reach 10.0.1.0/24, can reach 10.0.2.0/24, and cannot reach anything else on earth. For its job, that is complete knowledge.
Here is the part few people ever see stated plainly: Linux has no separate forwarding table. When a packet arrives addressed to someone else, the router's kernel looks it up in this exact table, the same one its own traffic uses, and sends it out the winning row's interface. Forwarding is ordinary routing performed on someone else's packet.
What separates a machine that silently drops such packets from one that forwards them is a single kernel switch, and that switch gets its own lesson (Lesson 13). This router has it on.
Stop and take stock. You can open the table any Linux machine routes by. You can name who installed each row from proto, and split every row into direct versus via. You can ask the kernel for its verdict on one address without sending a byte. And you know the tiebreak, longest prefix, that settles competing rows.
Three routing tables exist on this VM's namespaces and you have read two. The third belongs to the server, and from here the commands are no longer shown.
Read the server's routing table. It is the same form you have now used twice, aimed at the third namespace. Before you run it, predict which address its default route points at.
prompt: student@routing-lab:~$ answer: sudo ip netns exec server ip route show ||| sudo ip netns exec server ip route ||| sudo ip netns exec server ip r ||| sudo ip -n server route show output: default via 10.0.2.1 dev veth-s 10.0.2.0/24 dev veth-s proto kernel scope link src 10.0.2.20 hint: The namespace is called server. The reader is the one you used for client and router.
A mirror of the client: one connected route, one catch-all. The server's gateway is 10.0.2.1, the router's other leg. Both endpoints lean on the same middle machine, each from their own side.
When the server answers the client, the reply is a brand new packet, and the server's kernel routes it from scratch. Ask the server's kernel which route it would use to reach 10.0.1.10. No command shown. You have asked this exact kind of question before, in another namespace.
prompt: student@routing-lab:~$ answer: sudo ip netns exec server ip route get 10.0.1.10 ||| sudo ip netns exec server ip r get 10.0.1.10 ||| sudo ip -n server route get 10.0.1.10 output: 10.0.1.10 via 10.0.2.1 dev veth-s src 10.0.2.20 uid 0 cache hint: The dry-run route reader, run inside the server namespace, aimed at the client address 10.0.1.10.
via 10.0.2.1: the reply goes to the router too. That is what makes the round trip work.
Notice what this implies. Nothing links the two directions. The forward path was the client's decision, the return path is the server's, and each consulted only its own table. On the internet the two directions of one conversation often travel different roads entirely. Engineers call that asymmetric routing, and it is why a firewall watching only one side of a link can be blind to half of a conversation it swears it is inspecting.
On a broken machine you rarely want the whole table. You want one answer: where does traffic to everywhere else go? Add the word default to the reader and it prints only the catch-all.
sudo ip netns exec client ip route show default
prompt: student@routing-lab:~$ answer: sudo ip netns exec client ip route show default ||| sudo ip netns exec client ip r show default ||| sudo ip -n client route show default output: default via 10.0.1.1 dev veth-c hint: Add the word default after show.
One line: the client's only exit. On any real machine, this is the first read when something beyond the local subnet is unreachable.
The classic symptom: a machine can ping its neighbors but nothing farther away. Check the default route first. A missing default, or one pointing at a dead gateway, produces exactly that shape of failure. Your VM itself has a routing table too: run plain ip route show outside the namespaces and read it. Same grammar, different networks, because every namespace keeps its own table.
One address in the client's world appears in neither row of its table: the client's own 10.0.1.10. Ask the client's kernel for its route to its own address, and read every word that comes back. Two of them will be new.
prompt: student@routing-lab:~$ answer: sudo ip netns exec client ip route get 10.0.1.10 ||| sudo ip netns exec client ip r get 10.0.1.10 ||| sudo ip -n client route get 10.0.1.10 output: local 10.0.1.10 dev lo table local src 10.0.1.10 uid 0 cache <local> hint: The dry-run question one more time, aimed at the client's own address, 10.0.1.10.
Two surprises in one line. dev lo: a packet to your own address is delivered through the loopback device. It never reaches the network interface, never touches the wire. The machine hands it to itself internally.
And table local: this answer did not come from the table you have been reading. ip route show prints one table, called main. The kernel keeps others. Every address you assign gets bookkeeping rows in a hidden table called local, which the kernel writes itself and consults before main. That table is how a machine recognizes traffic addressed to it. You can read it any time with ip route show table local, and every local and broadcast row you find there was installed by the kernel, not by an admin.
The consultation order is set by policy rules, which ip rule prints: local first, then main, then a normally empty table called default. Policy routing goes much deeper than this track needs, so keep one sentence: ip route show is not the whole truth, it is the main chapter of it.
ip route is a formatter. The kernel also publishes the main table as a text file, /proc/net/route, and monitoring tools read it there. Look at the client's copy, and before the next step decodes it, try to spot the gateway 10.0.1.1 hiding in it.
sudo ip netns exec client cat /proc/net/route
prompt: student@routing-lab:~$ answer: sudo ip netns exec client cat /proc/net/route output: Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT veth-c 00000000 0101000A 0003 0 0 0 00000000 0 0 0 veth-c 0001000A 00000000 0001 0 0 0 00FFFFFF 0 0 0 hint: It is a file, so the reader is cat, run inside the client namespace: sudo ip netns exec client cat /proc/net/route
Same two routes, raw. The first data row is the default route, the second is the connected route, and every address is printed as eight hexadecimal digits, base sixteen, two digits per byte, with the bytes in reverse order. The gateway you were hunting is the 0101000A in the first row. The next step hands you the decoder.
Practice The Routing Table in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.