Learn › Linux networking › Route selection and next hops
Add a conflicting host route on three real Linux VMs, prove what it breaks with route lookups, the neighbor table and a tcpdump capture, then repair the path in both directions.
The problem: a client can reach most of a service network, but one service address stops responding after a more specific route appears. A VPN client can cause a similar one-destination incident when it installs a route for one service address. This lab does not install a VPN. It reproduces route precedence with a direct Ethernet host route. A real VPN route may point at a tunnel instead, so the ARP behavior below belongs to this particular lab fault.
You will explain an IPv4 address and prefix, predict which route Linux selects, inspect a live route lookup, neighbor state and packet observations, remove only the faulty route, then verify both directions. You will also test a different destination so the explanation cannot depend on memorizing one address.
Prerequisite: Chapter 1's three-machine route and forwarding exercise. Familiarity with a shell, ping and the difference between client and router helps. Section 2.1 rebuilds the path on fresh machines.
Success conditions: identify the selected prefix, output interface and next hop; distinguish an IP destination from the Ethernet receiver on the first link; cite what an ARP observation does and does not show; make the smallest justified repair; and state what the endpoint tests establish. The seven Check chapter checks report current machine state. They are formative feedback, not a grade or proof that you can explain the incident. Reading and command entry also do not establish independent mastery.
An IPv4 address has 32 bits. In 10.71.20.20/24, /24 says that the first 24 bits, here 10.71.20, identify the network. The remaining eight bits identify an address within it. The route 10.71.20.0/24 covers addresses from 10.71.20.0 through 10.71.20.255. The route 10.71.20.20/32 matches one address. It is a host route.
Suppose both ordinary routes are in the same routing table and no policy rule selects another table. Both match destination 10.71.20.20.
>>> The /32 matches fewer destinations. Linux selects the longest matching prefix in this case. The insertion order is not the deciding factor. You will inspect the actual choice on client before and after the fault.
The IP destination may be beyond the sender's local link. Client's eth1 and router's eth1 share 10.71.10.0/24; server's 10.71.20.20 is on another link. A route through 10.71.10.1 tells client to hand the packet to router first. That router is the next hop.
An Ethernet frame has its own receiver address, called a MAC or link-layer address. That receiver can change at each link while the packet's IP destination remains the server.
>>> Client needs router eth1's MAC address to send on client-net. The IPv4 destination remains 10.71.20.20. The neighbor table and packet observations will let you check the distinction.
ip -4 route get asks the Linux kernel which route it would select for the stated destination and inputs. It can report via (gateway or next hop), dev (output interface) and src (chosen source address). It does not send a packet.
>>> The lookup is a decision, not a delivery record. A bounded ping and actual observations at named interfaces test what happened when traffic was sent. Even those tests apply only to the observed time and traffic.
Start this chapter's practice network if it is not already running. Open the client, router and server shell tabs as you need them. The selected shell tab receives your command. Selecting a node in Topology shows its details but does not switch shells. Packets shows saved observations from the bounded recorder.
| Machine | Exercise interface | Assigned address | Link |
|---|---|---|---|
| client | eth1 | 10.71.10.10/24 | client-net |
| router | eth1 | 10.71.10.1/24 | client-net |
| router | eth2 | 10.71.20.1/24 | server-net |
| server | eth1 | 10.71.20.20/24 | server-net |
Each row gives an assigned address, not evidence that a packet has crossed the network. The diagram lets you locate the links. ip lets you check the guests. eth0 is reserved for management. Leave its address and default route alone. Route changes in this lesson affect only the exercise links and last until the machines restart or the lab ends.
On client, check the assigned address on the exercise interface. Then inspect router eth1 and eth2, and server eth1 in their own shell tabs. If an exercise address does not match the table, stop before changing routes and report the discrepancy. lo and eth0 are not part of this path.
Start on client with the command below. ip asks Linux for network state, -br gives a brief line, address show lists assigned addresses, and dev eth1 limits the view to this interface. On router, substitute eth1 and then eth2; on server, use eth1.
ip -br address show dev eth1machine: client prompt: admin@client:~$ answer: ip -br address show dev eth1 output: eth1 UP 10.71.10.10/24 hint: On client, request the brief address view for eth1.
Find 10.71.10.10/24 on eth1. Your live line may also include an IPv6 address. Check router and server in their own tabs with the same ip -br address show dev ... form and the interfaces in the table. An assigned address confirms a configured interface, not a working remote path.
The initial exercise state has no explicit routes between the two subnets and router forwarding is off. Check chapter runs seven state checks and includes endpoint probes. Use it after a recording closes, because its probes can otherwise appear among your observations. During the controlled fault, some checks should fail; that is expected feedback.
Client can reach 10.71.10.0/24 directly. Server is in 10.71.20.0/24, so client needs a route through router's address on client-net. labnet applies only reviewed exercise changes with administrator rights. It has no general root shell. On a machine you administer outside this lab, equivalent route changes normally require privileged access.
On client, run these lines one at a time. replace adds the route if absent or updates it if present. The first line gives the 10.71.20.0/24 destination a gateway on client-net. The second lists the main IPv4 table so you can see what changed. -4 means IPv4, via names the next hop, and dev names the outgoing interface.
labnet ip route replace 10.71.20.0/24 via 10.71.10.1 dev eth1
ip -4 route show table mainmachine: client prompt: admin@client:~$ answer: labnet ip route replace 10.71.20.0/24 via 10.71.10.1 dev eth1 output:
hint: Route the server subnet through router 10.71.10.1 on client eth1.
machine: client prompt: admin@client:~$ answer: ip -4 route show table main output: 10.71.20.0/24 via 10.71.10.1 dev eth1 hint: List the main IPv4 routing table on client.
Find a route beginning 10.71.20.0/24 via 10.71.10.1 dev eth1 among the live table's other entries. The command's silence on success is not proof of delivery. Keep the management default route unchanged.
The reply is a separate packet. Server must select its own route back to 10.71.10.10; a route on client does not configure server. On server, add the route to client-net through router's address on server-net, then inspect the table.
In the server shell, the destination is client-net and the gateway is router's eth2 address on server-net. Inspect the table after the change.
labnet ip route replace 10.71.10.0/24 via 10.71.20.1 dev eth1
ip -4 route show table mainmachine: server prompt: admin@server:~$ answer: labnet ip route replace 10.71.10.0/24 via 10.71.20.1 dev eth1 output:
hint: On server, route client-net through 10.71.20.1 on eth1.
Use ip -4 route show table main on server and find 10.71.10.0/24 via 10.71.20.1 dev eth1. This is the return path. Do not alter the management default route.
Router has an interface on each exercise link, but Linux must be allowed to forward IPv4 packets between them. Proxy ARP is a separate feature: a router can answer an ARP question for an address it does not own. That could hide the fault this lesson is meant to expose, so check that it is disabled on router's client-facing link.
In the router shell, the first command enables forwarding through the reviewed labnet operation. The second reads the live kernel setting. Run them separately so the read checks the result of the write.
labnet sysctl -w net.ipv4.ip_forward=1
/usr/sbin/sysctl net.ipv4.ip_forwardmachine: router prompt: admin@router:~$ answer: labnet sysctl -w net.ipv4.ip_forward=1 output: net.ipv4.ip_forward = 1 hint: On router, enable the reviewed IPv4 forwarding setting.
machine: router prompt: admin@router:~$ answer: /usr/sbin/sysctl net.ipv4.ip_forward output: net.ipv4.ip_forward = 1 hint: Read the router forwarding setting after writing it.
On router, read /usr/sbin/sysctl net.ipv4.conf.all.proxy_arp net.ipv4.conf.eth1.proxy_arp. Both values must be 0. If either is 1, stop and report it rather than changing a setting outside the reviewed lab operations. Forwarding does not add endpoint routes, and a changed ip_forward value can reset some related kernel parameters.
Before adding a fault, test the working baseline. Predict that client and server will each receive three replies because both endpoint subnet routes and router forwarding are in place. ping -n uses numeric addresses, -I eth1 selects the exercise interface, -c 3 sends three requests, and -W 1 bounds each reply wait.
Run this in client shell:
ping -n -I eth1 -c 3 -W 1 10.71.20.20Then switch to server shell and run:
ping -n -I eth1 -c 3 -W 1 10.71.10.10Read each live packets transmitted and received summary before continuing.
machine: client prompt: admin@client:~$ answer: ping -n -I eth1 -c 3 -W 1 10.71.20.20 output: 3 packets transmitted, 3 received, 0% packet loss hint: From client eth1, send three numeric pings to server.
machine: server prompt: admin@server:~$ answer: ping -n -I eth1 -c 3 -W 1 10.71.10.10 output: 3 packets transmitted, 3 received, 0% packet loss hint: From server eth1, send three numeric pings to client.
Read the actual summaries from your shells. Times vary. Both tests should say three transmitted and three received. A reply with TTL 63 is consistent with one router on these Linux images, but TTL alone is not a complete path trace. If either test fails, inspect the two endpoint route lookups and router forwarding, correct only the known wrong setup value, and repeat both pings. Do not inject the fault into a broken baseline.
The routing table lists candidate routes. A lookup resolves the choice for one destination. In this exercise, the default rules select the main table for ordinary remote destinations. A later chapter covers policy rules that can select another table.
On client, predict via 10.71.10.1 dev eth1 src 10.71.10.10, then inspect the live result. Save it as the baseline for comparison.
In client shell, ask for the route to server's exact address. get reports the kernel's selected route for this lookup; it does not send traffic.
ip -4 route get 10.71.20.20machine: client prompt: admin@client:~$ answer: ip -4 route get 10.71.20.20 output: 10.71.20.20 via 10.71.10.1 dev eth1 src 10.71.10.10 hint: Ask client which IPv4 route it selects for server's address.
The live output can include uid and cache lines. via names the next hop, dev the output interface and src the proposed source address. This is the decision for the queried inputs. It sends no packet and cannot prove a reply. If you see a different next hop, inspect ip -4 rule show and ip -4 route show table main before proceeding.
Client must learn router eth1's MAC address before it can send a frame to that next hop. ARP, the Address Resolution Protocol, asks on a local IPv4 Ethernet link, “Who has this IP address?” The neighbor table records learned IP-to-MAC mappings and their recent state.
After the working ping, inspect client eth1. Find 10.71.10.1, the local next hop, rather than 10.71.20.20, the remote IP destination.
In client shell, neigh show lists learned neighbors and dev eth1 confines the list to the local exercise link.
ip -4 neigh show dev eth1machine: client prompt: admin@client:~$ answer: ip -4 neigh show dev eth1 output: 10.71.10.1 lladdr 90:de:01:d4:f6:96 STALE hint: List IPv4 neighbors learned on client eth1.
The MAC and state vary. lladdr is the learned MAC. REACHABLE means recently confirmed; STALE means it has not been confirmed recently; DELAY or PROBE means Linux is checking. An absent entry by itself does not prove router is down. The address being resolved is the useful comparison.
Select the client shell, then open Topology and choose Ping & record. The From label must show client. The recorder starts at client eth1, router eth1 and eth2, and server eth1, then sends two pings from the selected endpoint. Open Packets to inspect each observation's capture point, Ethernet addresses, IP addresses, protocol and original packet bytes. The recorder includes ARP and IPv4 ICMP on the exercise links. It excludes management eth0.
The recording stops after ten seconds or 50 observations per interface, with 200 observations total. The newest eight recordings remain available for up to 24 hours. Save observations you need in your own notes before ending the lab. For Record next ping, keep Topology open with the client console visible. Type the ping in that console without pressing Enter. Click Record next ping, wait until the status reads Recording rather than Preparing, then return focus to the client console and press Enter. Preparing the command first leaves the ten-second window for packets. Stop recording ends the window early. If it expires before the ping runs, start a fresh recording; an empty capture from that window does not establish that no packets were sent.
Find an ICMP echo request at client eth1 and router eth1, and look for matching request and reply observations farther along the path. For the first link, compare the frame's receiver MAC with client neighbor 10.71.10.1. The frame should target router's MAC while its IP destination is still 10.71.20.20. Router sends a new frame on server-net with link addresses appropriate to that link. ARP observations are questions and replies on one local link; they are not evidence that the ARP message traveled through router. Capture points and packet identifiers matter more than animation order because guest clocks are not assumed synchronized.
A captured request at one interface proves that interface saw it. A missing observation can reflect capture limits, timing or filtering. A matching reply plus successful endpoint ping is stronger evidence for this tested exchange. Stop recording before Check chapter, which sends its own probes.
The controlled fault adds 10.71.20.20/32 dev eth1 on client, without via. The correct /24 route remains. A VPN route could create a similar one-destination override, but this lab adds the route directly and runs no VPN client.
>>> Within this table, the longest matching prefix wins. dev eth1 with no gateway tells client to try delivery on that local link. It should ask by ARP for 10.71.20.20 itself. Server owns that address on the separate server-net, so no device on client-net should answer in this lab.
On client, add only the host route. The operation is intentionally restricted to the exercise interface. Inspect the whole main table and look up the destination again before sending traffic. Record the baseline and fault lookups side by side.
Run the two lines in client shell, one at a time. The /32 matches only server's address. With no via, it tells Linux to treat that address as directly reachable on eth1. Compare the resulting get output with the earlier lookup.
labnet ip route replace 10.71.20.20/32 dev eth1
ip -4 route get 10.71.20.20machine: client prompt: admin@client:~$ answer: labnet ip route replace 10.71.20.20/32 dev eth1 output:
hint: Add only the reviewed direct host route for server's address on client eth1.
machine: client prompt: admin@client:~$ answer: ip -4 route get 10.71.20.20 output: 10.71.20.20 dev eth1 src 10.71.10.10 hint: Ask client which route it now selects for 10.71.20.20.
ip -4 route show table main should still show 10.71.20.0/24 via 10.71.10.1 dev eth1 and a more specific 10.71.20.20 dev eth1 without via. The table may omit /32 when printing a host route. The lookup has lost via 10.71.10.1. That identifies a changed decision, but delivery still needs a separate test. Do not flush routes or touch the management route.
Keep Topology open and select the client shell. Type the command below in the client console without pressing Enter. Click Record next ping, wait for Recording rather than Preparing, then focus the client console and press Enter. Keep this recording separate from Check chapter. If the ten-second window expires before the ping runs, prepare the command and start another recording. After it closes, open Packets and compare it with the working recording.
The destination stays the same as in the working baseline, so a changed result can be compared with the new route lookup and ARP observations.
ping -n -I eth1 -c 3 -W 1 10.71.20.20machine: client prompt: admin@client:~$ answer: ping -n -I eth1 -c 3 -W 1 10.71.20.20 output: 3 packets transmitted, 0 received, 100% packet loss hint: While Record next ping is active, send three pings from client eth1 to server.
Read your live ping summary. In the verified original lab, the client received no replies with
Practice Chapter 2. Route selection and next hops in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.