Learn › Linux networking › Persistence and recovery
A real three-machine networking investigation with chapter-specific checks.
A network can work now and fail after a reboot. A command can change the running Linux kernel without leaving instructions for the next boot. In this chapter you will remove one boot instruction, observe what survives, and repair the existing configuration so the path returns after a real reboot.
Preparation: complete Linux Foundations and Networking Chapters 1 to 6 in order. Chapters 1 and 2 taught exercise addresses, gateways, route lookups and return paths; Chapter 3 distinguished forwarding from firewall permission. Here you will learn the service and boot concepts needed for persistence.
This fresh session has three machines. Use the named Client, Router and Server shells, and check the prompt before entering a command. Client's eth1 is 10.71.10.10/24. Router joins the two exercise networks through eth1 at 10.71.10.1/24 and eth2 at 10.71.20.1/24. Server's eth1 is 10.71.20.20/24. The eth0 interface carries the management connection to each machine. Keep its address, routing rules and default route unchanged.
The workspace checks machine state automatically. They can all pass when the healthy session opens. That is useful feedback about the starting state, not evidence that you diagnosed a fault or tested a reboot. Keep your predictions and actual before, fault, repair and post-boot observations separately.
The kernel is the running part of Linux that uses addresses, routes and forwarding settings to handle packets. ip route replace changes its current route table. A reboot starts another kernel, so something must recreate the route.
systemd starts services during boot. A unit is its description of a service, including the command to run and dependencies on other services. The lab supplies an address service named lc-network-interfaces.service. On Client and Server, a separate route unit adds the route to the other exercise subnet after that address service has run.
These route units are oneshot services: each runs one command, then exits. RemainAfterExit=yes lets systemd continue to report the unit as active after that successful command. It does not make systemd watch the route and replace it whenever someone deletes it.
Each observation answers a different question:
enabled means it is connected to the boot sequence. It still needs to run successfully.active means systemd considers the unit active. For this oneshot service, its command has finished.Router uses another mechanism. A sysctl is a kernel setting. net.ipv4.ip_forward controls whether Router can pass IPv4 packets between interfaces. A running value of 1 permits forwarding; 0 disables it. The lab's file /etc/sysctl.d/90-linuxcamp-network.conf supplies the value to load during boot. The file and the running value are separate observations.
Some changes require administrator privileges. labnet is the lab's helper: it checks that a request stays within this exercise, then runs the real Linux command on the selected machine. It is not a standard Linux command or a simulation. Ordinary reads such as cat, ip and systemctl cat do not need it. A refused command is a reason to inspect the request, not to add sudo or change another machine.
On Client, print the session settings, called the descriptor. JSON represents each setting as a name paired with a value.
python3 -m json.tool /etc/lab/chapter.jsonFind routeUnit, routeUnitPath, addressUnit and forwardingFile. routeUnit has a name beginning lc9- and ending .service, with a session-specific identifier between them. chapterId: 9 is the preserved internal identity for this displayed Chapter 7. The address unit should be lc-network-interfaces.service, and the forwarding file should be /etc/sysctl.d/90-linuxcamp-network.conf. Do not copy a unit name from a previous session.
Run this read block separately on Client, Router and Server. Keep the output in your notes outside the guest before any reboot.
hostname
cat /proc/sys/kernel/random/boot_id
ip -br -4 address
ip -4 route show table main
ip -4 rule showhostname identifies the machine. The boot ID is a long identifier that changes when that machine boots again. Record one per machine. In the address output, record eth0 and the exercise interfaces. In the route output, save the default entry, including its via gateway and dev interface. Routing rules choose which route table to consult; record them for comparison, without changing them.
Use these saved results as the comparison for every reboot. You do not need to copy the same unchanged output into a new report each time. Record the new boot ID, note which observations match, and save any difference.
Recovery checklist, used after each reboot:
1. On the selected machine, run hostname and cat /proc/sys/kernel/random/boot_id. The name must be the same and the boot ID must differ from its saved value. Read the other two machines' boot IDs to confirm they did not reboot. 2. Repeat ip -br -4 address, ip -4 route show table main and ip -4 rule show on the selected machine. Compare its exercise addresses, eth0, default route and rules with your baseline. A shell reconnect by itself does not establish that all these properties survived. 3. On a rebooted endpoint, reload route_unit with the assignment taught in Step 7.2, then read its enabled state, active state and current-boot journal. On a rebooted Router, read the saved forwarding file and /proc/sys/net/ipv4/ip_forward instead. These identify what boot setup restored. 4. Repeat the two endpoint lookups and bounded pings from Step 7.3. A boot owner can look correct while traffic still fails, so keep both-direction traffic evidence for each reboot.
The expected baseline already has working exercise routes and persistent forwarding. If the descriptor, machine identity or exercise addresses disagree with this lesson, keep the evidence and report the setup mismatch before introducing a fault.
machine: client prompt: admin@client:~$ answer: python3 -m json.tool /etc/lab/chapter.json output:
hint: Find this session's configuration owners. Use the actual output to check your explanation.
On Client, then separately on Server, load the exact route-unit name and inspect its saved instruction and present state.
route_unit=$(python3 -c 'import json; print(json.load(open("/etc/lab/chapter.json"))["routeUnit"])')
systemctl cat "$route_unit"
systemctl is-enabled "$route_unit"
systemctl is-active "$route_unit"
labnet journalctl -u "$route_unit" -b --no-pager -n 20The first command reads the routeUnit value with Python and stores it in the shell variable route_unit. The later commands use that value. A new shell after reboot will need this assignment again. systemctl cat prints the unit file; it does not change it. journalctl reads the journal. -u selects this unit, -b selects this boot, -n 20 limits the output to twenty entries, and --no-pager prints without opening a scrolling viewer.
Find Requires=lc-network-interfaces.service and After=lc-network-interfaces.service. The first declares the dependency; the second orders the route action after the address service. The route needs its exercise interface configured before it can use the local gateway. Inspect ExecStart, the command the unit runs. Client's command adds 10.71.20.0/24 via 10.71.10.1 dev eth1; Server's adds 10.71.10.0/24 via 10.71.20.1 dev eth1. Neither changes a default route. Both units should report enabled and active. Read any failure message rather than treating the word active alone as traffic evidence.
machine: client prompt: admin@client:~$ answer: route_unit=$(python3 -c 'import json; print(json.load(open("/etc/lab/chapter.json"))["routeUnit"])') output:
hint: Load this endpoint's unit name. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: systemctl cat "$route_unit" output:
hint: Inspect the endpoint boot instruction. Use the actual output to check your explanation.
machine: server prompt: admin@server:~$ answer: route_unit=$(python3 -c 'import json; print(json.load(open("/etc/lab/chapter.json"))["routeUnit"])') output:
hint: Load this endpoint's unit name. Use the actual output to check your explanation.
machine: server prompt: admin@server:~$ answer: systemctl cat "$route_unit" output:
hint: Inspect the endpoint boot instruction. Use the actual output to check your explanation.
>>> The kernel accepted a runtime change. A persistent repair also needs the right saved instruction, successful execution after boot, and fresh traffic tests.
On Router, compare the saved forwarding instruction and running setting.
cat /etc/sysctl.d/90-linuxcamp-network.conf
cat /proc/sys/net/ipv4/ip_forwardThe file should contain the session-owner comment and net.ipv4.ip_forward = 1; the second read should print 1. No change is justified when both already match the intended baseline.
Before sending traffic, predict Client's next hop and Server's return next hop. On Client, run:
ip -4 route get 10.71.20.20
ping -n -I eth1 -c 2 -W 1 10.71.20.20On Server, run:
ip -4 route get 10.71.10.10
ping -n -I eth1 -c 2 -W 1 10.71.10.10Client's lookup should select via 10.71.10.1 dev eth1; Server's should select via 10.71.20.1 dev eth1. These are the fields to compare in your actual output, not recorded results from your machines. Ping sends two requests (-c 2) using eth1 (-I eth1), with a bounded reply wait (-W 1), and prints numeric addresses (-n). Read the transmitted and received counts. Two replies support current reachability; missing replies require investigation rather than an assumed cause.
Make an ownership table for Client's remote route, Server's remote route and Router's forwarding setting. Record the running observation, saved unit or file, boot action and any uncertainty.
machine: router prompt: admin@router:~$ answer: cat /etc/sysctl.d/90-linuxcamp-network.conf output:
hint: Read the saved forwarding value. Use the actual output to check your explanation.
machine: router prompt: admin@router:~$ answer: cat /proc/sys/net/ipv4/ip_forward output:
hint: Compare the running forwarding value. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: ip -4 route get 10.71.20.20 output:
hint: Read Client's selected next hop. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: ping -n -I eth1 -c 2 -W 1 10.71.20.20 output:
hint: Test Client to Server. Use the actual output to check your explanation.
machine: server prompt: admin@server:~$ answer: ip -4 route get 10.71.10.10 output:
hint: Read Server's selected return hop. Use the actual output to check your explanation.
machine: server prompt: admin@server:~$ answer: ping -n -I eth1 -c 2 -W 1 10.71.10.10 output:
hint: Test Server to Client. Use the actual output to check your explanation.
Disabling a unit removes its boot instruction without stopping this running oneshot service or deleting the route it already added. Before changing Client, write two predictions: will its route and ping still work immediately, and will that route return after Client reboots? Explain why the answers may differ.
On Client, run:
labnet systemctl disable "$route_unit"
systemctl is-enabled "$route_unit"
systemctl is-active "$route_unit"
ip -4 route get 10.71.20.20
ping -n -I eth1 -c 2 -W 1 10.71.20.20The enabled-state read should print disabled and return a nonzero status. That is expected evidence of this deliberate change. active can remain, and the current path can still work. No error from the change command alone establishes either result. Read the actual unit state, selected route and replies.
The automatic Client persistence check should now fail even while its path check passes. This is a useful partial state: present connectivity and boot configuration disagree.
machine: client prompt: admin@client:~$ answer: labnet systemctl disable "$route_unit" output:
hint: Remove only the Client boot instruction. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: systemctl is-enabled "$route_unit" output:
hint: Observe the expected disabled state. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: ping -n -I eth1 -c 2 -W 1 10.71.20.20 output:
hint: Test the path before reboot. Use the actual output to check your explanation.
Save the current observations and all three boot IDs first. Rebooting disconnects the selected terminal while the guest starts again. On Client only, run:
labnet systemctl rebootA reboot deliberately disconnects the shell. The disconnect does not prove that the machine has restarted successfully. Compare its identity and boot ID, then inspect the recovered network as described below.
Wait for Client to return, then open its shell again. Do not interpret the disconnect itself as a failure or success. If the same guest does not become available, record the failure and stop this test. Ending the session and starting another creates a fresh baseline; it cannot prove that the original guest recovered.
On Client, follow the recovery checklist from Step 7.1. Its boot ID must change while Router and Server keep theirs. Compare management before investigating the exercise path.
Reload route_unit using Step 7.2's assignment, then read the unit's enabled and active states and current-boot journal. Repeat the Client lookup and the two endpoint pings. The exercise address service should restore Client's eth1 address, but the disabled route unit should not restore its remote-subnet route. A lookup may fall back to another route, including the default route; look for the required exercise next hop and eth1, rather than accepting any successful lookup. A missing current-boot journal entry is consistent with a disabled unit not running. It is not, alone, proof of the cause.
Use the combined evidence to explain the failure: the exercise address returned, the specific remote route did not, and the boot instruction is disabled. A failed ping by itself would not distinguish this from a forwarding or firewall problem.
machine: client prompt: admin@client:~$ answer: cat /proc/sys/kernel/random/boot_id output:
hint: Compare Client's boot ID after reconnect. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: ip -4 route get 10.71.20.20 output:
hint: Find the route selected after reboot. Use the actual output to check your explanation.
Reload and inspect the existing Client unit with the assignment and reads from Step 7.2. Inspect Server's unchanged owner for comparison. The disabled boot instruction explains why the route did not return. Enable the existing Client route unit, then run it now. enable arranges future boot execution; restart executes this oneshot command in the current boot. Do not create a second unit or alter Server's healthy route.
On Client, after confirming route_unit still names this session's unit, run:
labnet systemctl enable "$route_unit"
labnet systemctl restart "$route_unit"
systemctl is-enabled "$route_unit"
systemctl is-active "$route_unit"
labnet journalctl -u "$route_unit" -b --no-pager -n 20
ip -4 route get 10.71.20.20
ping -n -I eth1 -c 2 -W 1 10.71.20.20On Server, repeat its lookup and bounded ping from Step 7.3. Verify the specific exercise next hops, replies in both directions and preserved management observations. The automatic checks should return to their healthy state. This establishes a working repair now.
machine: client prompt: admin@client:~$ answer: route_unit=$(python3 -c 'import json; print(json.load(open("/etc/lab/chapter.json"))["routeUnit"])') output:
hint: Reload the unit name in the recovered shell. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: labnet systemctl enable "$route_unit" output:
hint: Restore the Client boot instruction. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: labnet systemctl restart "$route_unit" output:
hint: Apply the Client route now. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: labnet journalctl -u "$route_unit" -b --no-pager -n 20 output:
hint: Read this boot's route-service evidence. Use the actual output to check your explanation.
machine: client prompt: admin@client:~$ answer: ping -n -I eth1 -c 2 -W 1 10.71.20.20 output:
hint: Test the repaired outward path. Use the actual output to check your explanation.
machine: server prompt: admin@server:~$ answer: ping -n -I eth1 -c 2 -W 1 10.71.10.10 output:
hint: Test the repaired return path. Use the actual output to check your explanation.
Predict what will happen when Client reboots again. Save its current boot ID, run the same Client reboot command from the runtime-only reboot step, and repeat that step's recovery reads. This time require another new boot ID, enabled, a successful current-boot route service, the correct exercise route and fresh replies in both directions. Those observations test persistence through this reboot.
Practice Chapter 7. Persistence and recovery in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.