Learn › The Network Stack › How Packets Find Their Way
ip addr - a hands-on Linux lab on a real virtual machine.
One card, one MAC, two IP addresses. CIDR is the ruler, the kernel picks its identity per destination, and your own address is just a /32 route.
Somewhere inside this VM there is a small machine called client, and it is mid-move. Its network is being renumbered: the old office network 192.168.100.0/24 is being retired, and everything is migrating to 10.0.1.0/24.
During a cutover a machine cannot afford to vanish from either world. So right now the client's one network card carries two IP addresses at once:
[client] 10.0.1.10/24 <--wire--> 10.0.1.20/24 [server]
MAC 02:00:0a:00:01:0a
plus legacy 192.168.100.1/24 on the same cardOne card. One MAC address. Two IP addresses. Which raises the question this whole lesson answers: when the client sends a packet, which of its two names does it speak as? And how does the kernel decide, for every single packet, whether the destination is a neighbor or a stranger?
By the end you will read the kernel's verdict directly, catch it switching identities live, and find the exact place under /proc where it keeps its own addresses filed as routes.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The client and server are real, built inside your lab VM with fixed addresses, so each screen shows what the machine itself will show you. The real terminal comes at the end, under Ready to practice.
Progress in the lab is tracked automatically. Just type commands naturally, and let Tab do the typing where it can: /proc/net/fib_t plus the Tab key completes the whole path.
An IP address (Internet Protocol address) is the number that identifies one attachment point on a network, so machines that have never met can still exchange packets. The version this lesson works with is IPv4: one 32-bit number.
10.0.1.10 is that number dressed up for humans. Each of the four groups is an octet: 8 bits, holding a value from 0 to 255. Four octets make the 32 bits. The dots are pure formatting, called dotted decimal notation. The kernel never stores dots. It stores a single integer.
The proof is fun: ping 2130706433 works on most Linux boxes. That is 127.0.0.1 written as its raw 32-bit integer, and the resolver accepts it.
You already know this machine's Layer 2 name from the MAC lessons: burned into the card, meaningful only on the local wire. An IP address is the opposite on both counts. It is assigned by configuration, and it is meaningful across networks. A MAC says what hardware this is. An IP says where in the network it currently sits. Move a laptop from home to office and its IP changes; the MAC follows the hardware.
The 32-bit size was fixed in RFC 791 in 1981, when four billion addresses looked infinite. By 1993 the address registries were burning through them so fast that CIDR (Classless Inter-Domain Routing) was invented to slice the space finely instead of in three rigid class sizes. The slash you are about to meet everywhere is CIDR.
Vint Cerf picked 32 bits, and he has spent decades apologising for it in public. The address size was settled while TCP/IP was still a research experiment nobody expected the whole planet to run on, so four billion looked like a generous number for a test network. Jon Postel wrote it down as RFC 791 in September 1981, and the experiment never got switched off.
A network namespace is a private copy of the entire Linux network stack: its own interfaces, its own addresses, its own routes. The client and server here are two namespaces inside your VM, joined by a virtual cable. To run a command on the client machine, prefix it with sudo ip netns exec client. Read that as: on client, run this. Namespaces get a full lesson later in the track.
ip addr show lists addresses. Add dev veth-c to show a single interface, and veth-c is the client's end of the cable. Before you press Enter, commit to a number: how many inet lines should one card carry mid-migration?
sudo ip netns exec client ip addr show dev veth-c
prompt: student@ip-lab:~$ answer: sudo ip netns exec client ip addr show dev veth-c ||| sudo ip netns exec client ip a show dev veth-c ||| sudo ip netns exec client ip addr list dev veth-c output: 4: veth-c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:0a:00:01:0a brd ff:ff:ff:ff:ff:ff link-netns server inet 10.0.1.10/24 scope global veth-c valid_lft forever preferred_lft forever inet 192.168.100.1/24 scope global veth-c valid_lft forever preferred_lft forever inet6 fe80::aff:fe00:10a/64 scope link valid_lft forever preferred_lft forever hint: Run ip addr show dev veth-c on the client machine: sudo ip netns exec client ip addr show dev veth-c
Two inet lines. Each one is a full IPv4 address, both marked scope global, meaning valid for talking to other machines. Above them sits one link/ether line: the MAC you already know. One Layer 2 name, two Layer 3 names, and nothing about that is special. An interface can carry as many IP addresses as you care to add. An IP address is configuration, not identity.
valid_lft forever preferred_lft forever are lifetimes. Static addresses never expire. On a DHCP lease you would watch both count down in seconds. And there is a state between alive and gone: when only preferred_lft reaches zero the address is deprecated, still serving old connections, never chosen as the source for new ones. That is how networks retire an address without cutting live sessions.
The inet6 fe80::... line was configured by nobody. The kernel gives every up interface an IPv6 link-local address, scope link, valid only on this wire. Look closely at fe80::aff:fe00:10a and you can read the MAC inside it: there is 0a:00:01:0a with ff:fe stitched into the middle and the first byte's second bit flipped. The recipe is called EUI-64.
One subtlety worth carrying: these two IPv4 addresses sit in different subnets, so both are primary. Put two addresses in the same subnet and the kernel flags the second secondary. Delete the primary and every secondary goes with it, silently, unless promote_secondaries is enabled, which modern distributions ship on. Engineers have dropped whole service blocks learning that the hard way.
The header numbers 4: and @if3 are interface indexes: this card is interface 4 here, and the other end of the wire is interface number 3 over in the server machine. Index numbers are assigned in creation order, so they can differ on your VM, and the trailing link-netns server tag only prints when the kernel has a namespace id on file for the far side. The addresses read the same either way, and the addresses are the lesson.
Last thing, for later: when you want this card without the IPv6 noise, the address-family switch goes before the object: ip -4 addr show. File it away. You will need it on your own shortly.
Both addresses ended in /24. That slash number is CIDR notation, and it works like a ruler laid across the 32 bits: it measures how many bits, left to right, name the network. Whatever is left names the host inside that network.
10.0.1.10/24
network bits (24) host bits (8)
00001010.00000000.00000001 . 00001010
10 . 0 . 1 . 10Think street and house number. Every machine on 10.0.1.0/24 shares the street name 10.0.1, and the last octet is the house number. Two rules come free with the split. House number zero, here 10.0.1.0, names the street itself: the network address. The highest house number, 10.0.1.255, is the broadcast address: a packet sent there is addressed to everyone on the street. Neither can be a machine, so a /24 holds 254 usable hosts, not 256.
Before CIDR, the same ruler was written as a subnet mask: 255.255.255.0. Each 255 is eight ones in binary, marking network bits. /24 and 255.255.255.0 are the same statement in two spellings.
| CIDR | Subnet mask | Host bits | Usable hosts |
|---|---|---|---|
| /8 | 255.0.0.0 | 24 | 16,777,214 |
| /16 | 255.255.0.0 | 16 | 65,534 |
| /24 | 255.255.255.0 | 8 | 254 |
| /31 | 255.255.255.254 | 1 | 2 |
| /32 | 255.255.255.255 | 0 | 1 |
/31 looks illegal by the subtract-two rule, and it is legal anyway: RFC 3021 waives the network and broadcast addresses on point-to-point links, so a router-to-router wire burns two addresses instead of four. /32 is a single host, no subnet at all. You will meet /32 again in a few minutes, from the kernel's side.
>>> With /24, the first three octets are the street name. 10.0.1.200 shares 10.0.1 exactly, so it is a neighbor; a bigger house number changes nothing. The other two differ inside the first 24 bits, 10.0.2.x in the third octet and 10.1.1.x in the second, so both are different streets, reachable only through a router.
Every time any process sends a packet, the kernel answers one question before anything touches the wire: is the destination on one of my local subnets, or not?
You can make the kernel show this decision without sending anything. ip route get takes a destination and prints the exact verdict it would apply: which interface, and, crucially, which source address.
sudo ip netns exec client ip route get 10.0.1.20
prompt: student@ip-lab:~$ answer: sudo ip netns exec client ip route get 10.0.1.20 ||| sudo ip netns exec client ip r get 10.0.1.20 output: 10.0.1.20 dev veth-c src 10.0.1.10 uid 0 cache hint: The kernel answers routing questions with ip route get: sudo ip netns exec client ip route get 10.0.1.20
Read the verdict left to right: to reach 10.0.1.20, go out dev veth-c, and sign the packet src 10.0.1.10. There is no via anywhere. via is how a gateway appears, and its absence means direct delivery: same subnet, no router involved.
Where did that route come from? Nobody typed it. The moment an address like 10.0.1.10/24 lands on an interface, the kernel manufactures the matching connected route for 10.0.1.0/24 on its own. ip route marks these proto kernel. Addresses and their routes are born together, which also means they die together. Hold that thought.
The trailing uid 0 says whose routing question this was. Routing verdicts in Linux can legally differ per user: policy rules can steer one UID's traffic through a VPN while everyone else goes out the front door. So the kernel records who asked.
And cache is a fossil: the IPv4 route cache was removed from the kernel years ago, but the output label survived. It usually prints empty, exactly as here.
>>> The kernel chooses the source per route, not per interface. The winning route for 192.168.100.77 is the connected route for 192.168.100.0/24, and the address that gave birth to that route is 192.168.100.1, so that is the name the client speaks as. An application can overrule this by binding to a specific address before connecting, but most never do, and then the kernel decides. One card, two identities, chosen by destination.
Confirm it against the live kernel. Same question, legacy destination.
sudo ip netns exec client ip route get 192.168.100.77
prompt: student@ip-lab:~$ answer: sudo ip netns exec client ip route get 192.168.100.77 ||| sudo ip netns exec client ip r get 192.168.100.77 output: 192.168.100.77 dev veth-c src 192.168.100.1 uid 0 cache hint: Same command as the last step, different destination: sudo ip netns exec client ip route get 192.168.100.77
Same interface, different signature: src 192.168.100.1. Two screens ago the same machine signed itself 10.0.1.10. Nothing was reconfigured between the two answers. Only the destination changed.
This is source address selection, and it is the reason the migration works at all: old-world peers hear the old name, new-world peers hear the new one, and one card carries both conversations.
Milestone: you can now read the kernel's routing mind before a single packet exists. Most engineers meet source-address surprises in a packet capture, after the fact, at 2 a.m. You just got the same answer from one line, in advance.
Third destination: 8.8.8.8, which sits on neither of the client's subnets. Commit to an outcome before you run it. Does the packet go out veth-c and die quietly somewhere, or does something stop it earlier?
sudo ip netns exec client ip route get 8.8.8.8
prompt: student@ip-lab:~$ answer: sudo ip netns exec client ip route get 8.8.8.8 ||| sudo ip netns exec client ip r get 8.8.8.8 output: RTNETLINK answers: Network is unreachable hint: Same routing question, aimed at 8.8.8.8: sudo ip netns exec client ip route get 8.8.8.8
The client has no route covering 8.8.8.8 and no default gateway, so the kernel refuses at the routing step. No frame was built, no MAC was resolved, nothing touched the wire. The refusal is manufactured entirely inside the kernel, before Layer 2 is consulted at all.
Decode the message once and you own it forever. ip talks to the kernel over a socket protocol called rtnetlink; when the kernel says no, ip prints the kernel's raw error text after the RTNETLINK answers: prefix. Network is unreachable is the errno ENETUNREACH: the routing table's way of saying no entry matches.
Your own VM does have a default route, which is why it reaches the internet while the client cannot. The full tour of routing tables and gateways is the next lesson.
Where do all these verdicts come from? One structure. The kernel keeps IPv4 routes in a trie, a prefix tree built for longest prefix match: find the most specific route that covers a destination. The whole thing is mirrored as text under /proc at /proc/net/fib_trie. FIB stands for forwarding information base.
The file is long. Here is the slice around the client's addresses, trimmed to the part worth reading:
sudo ip netns exec client cat /proc/net/fib_trie
prompt: student@ip-lab:~$ answer: sudo ip netns exec client cat /proc/net/fib_trie output: Main: +-- 0.0.0.0/0 3 0 5 ... +-- 10.0.1.0/24 2 0 2
| -- 10.0.1.0 |
|---|
/24 link UNICAST
| -- 10.0.1.10 |
|---|
/32 host LOCAL
| -- 10.0.1.255 |
|---|
/32 link BROADCAST +-- 192.168.100.0/24 2 0 2
| -- 192.168.100.0 |
|---|
/24 link UNICAST
| -- 192.168.100.1 |
|---|
/32 host LOCAL
| -- 192.168.100.255 |
|---|
/32 link BROADCAST ... hint: Print the trie file on the client machine: sudo ip netns exec client cat /proc/net/fib_trie
Your full file is longer: a Local: section follows Main:, loopback entries surround these, and the +-- lines are internal tree bookkeeping whose numbers vary with the tree's shape. The leaves are the point.
Each subnet appears as /24 link UNICAST: the connected routes you have been using all lesson. And each of the machine's own addresses appears as /32 host LOCAL: a host route covering exactly one address, type LOCAL, meaning deliver upward into this machine instead of out a wire. The broadcast addresses are filed too, as /32 link BROADCAST.
Sit with that for a second. The kernel keeps no separate list of my addresses. The question is this packet for me is answered by the same longest-prefix lookup as every forwarding decision, and the /32 wins because no prefix is longer. Those LOCAL entries live in a separate local routing table that is consulted before the main one; ip route show table local prints it in readable form.
And the reason Main: shows them anyway is a real piece of kernel history: since Linux 4.0 the local and main tables share one merged trie, saving a table lookup on every packet, and this file prints views of that single tree.
>>> Routes born from an address die with the address. With 10.0.1.10/24 gone, the 10.0.1.0/24 connected route vanishes, 192.168.100.1 covers a different subnet, and there is no gateway. ip route get 10.0.1.20 would answer RTNETLINK answers: Network is unreachable, with the cable plugged in and healthy the whole time. Layer 1 up, Layer 2 up, Layer 3 gone: that exact combination eats hours in real outages.
Try it on the real machine if you are curious. The repair is the mirror image, and it is the same command that built this lab in the first place: sudo ip netns exec client ip addr add 10.0.1.10/24 dev veth-c. Adding the address resurrects the route.
Neither of the client's addresses would survive on the public internet. Three ranges are reserved by RFC 1918 for private networks, and internet routers drop them on sight:
| Range | CIDR | Addresses | Where you meet it |
|---|---|---|---|
| 10.0.0.0 to 10.255.255.255 | 10.0.0.0/8 | 16,777,216 | clouds and large fleets |
| 172.16.0.0 to 172.31.255.255 | 172.16.0.0/12 | 1,048,576 | the one everyone forgets exists |
| 192.168.0.0 to 192.168.255.255 | 192.168.0.0/16 | 65,536 | home and small office gear |
This lab uses one address from the first family and one from the third, both private. When a private machine needs the public internet, a router rewrites its source address to a public one on the way out. That rewrite is NAT (Network Address Translation), and it gets its own lesson.
One more reserved world hides inside every machine: 127.0.0.0/8, the loopback range on the lo interface. 127.0.0.1 is the machine talking to itself; the traffic turns around inside the kernel and never reaches any wire. On an address card it shows scope host, and that completes the scope set you have now seen in full: global for the world, link for this wire only, host for this machine only.
localhost is simply the name /etc/hosts gives 127.0.0.1. And the whole /8 loops back: ping 127.99.1.1 answers on any Linux box.
An address only matters because programs attach to it. A program waiting for connections holds a socket: an address plus a port, the numbered slot that separates services sharing one machine. ss -tln lists them: -t TCP, -l listening, -n nu
Practice IP Addresses and Subnets in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.