Learn › Linux networking › Forwarding policy with nftables
Read a router's nftables firewall table (chains, hooks, rule order and counters), trace a dropped web request with rule counters and packet captures, and delete only the faulty rule by its handle while the rule that blocks a second port stays in place.
The problem: client can ping server, and server answers its own web request, but client cannot reach the service on TCP port 8080. Which packet-path decision changed?
This chapter uses the three assigned Linux machines from Chapters 1 and 2. Setup supplies the endpoint routes, router forwarding, two small services bound to server 10.71.20.20 on ports 8080 and 8081, and a chapter-owned nftables table on router. Port 8080 should answer client. Port 8081 should answer on server itself but remain blocked from client. That denied port is a negative control: preserving its failure is part of a correct repair.
You will explain local delivery and forwarding, predict which rule sees a request, inspect live service, route and counter evidence, remove one justified fault, verify the allowed and denied paths, and revisit the mechanism with a server-input fault later. Check chapter reports seven current-state checks. Reading, typing a command, or passing those checks alone does not establish independent diagnosis.
Use the named client, router and server shell tabs. A selected Topology node does not change the shell that receives a command. Before each exercise, confirm its named shell and prompt; output from another machine answers a different question. eth0 carries management traffic. Leave its addresses, default routes, SSH access and the base lab_access firewall table unchanged.
labnet is a Linux Camp helper installed in this lab, not a standard Linux command or an alias for nft. Your shell sends the command words after labnet to a root-owned service. That service checks who sent them, this session's chapter descriptor and the permitted command form. It runs the real Linux program and returns its output and exit status. For changes, these checks keep the exercise inside its own table and prevent accidental changes to the management firewall that keeps your shell connected. For example, after you read this session's table name below, labnet nft -a list chain inet "$LC_TABLE" transit asks the service to run nft against that table's transit chain.
On a Linux machine you administer outside this lab, if the same table exists and you have set LC_TABLE to its name, the corresponding privileged command would be sudo nft -a list chain inet "$LC_TABLE" transit, provided your account and local policy permit it. This lab does not grant general sudo access. Use labnet for the approved privileged operations shown in this chapter. Run ordinary observations such as ip -4 route get, ss and curl directly. If labnet refuses a request, the session may not be ready or the request may be outside its allowed scope; do not work around the refusal by changing the management firewall.
The assigned exercise links are client eth1 at 10.71.10.10/24, router eth1 at 10.71.10.1/24, router eth2 at 10.71.20.1/24, and server eth1 at 10.71.20.20/24. In Chapter 2, client's route lookup chose router 10.71.10.1 as the next hop for server's address. That choice tells client where to send first; it does not prove that a service answers. This chapter adds a TCP port, a number selecting a service at an IP address, and a firewall decision along the routed path. A web request to 10.71.20.20:8080 must be routed through router; a request from server to its own 10.71.20.20:8081 stays on server.
On each named machine, ip -br -4 address gives brief IPv4 interface lines. Compare eth1 and eth2 with the assignments before changing policy. If they differ, stop and report the discrepancy. On client, also ask which route would be selected for server; via should name router 10.71.10.1 and dev should name eth1.
ip -br -4 address
ip -4 route get 10.71.20.20machine: client prompt: admin@client:~$ answer: ip -br -4 address output:
hint: Read client IPv4 interfaces before judging its route.
machine: client prompt: admin@client:~$ answer: ip -4 route get 10.71.20.20 output:
hint: Ask client for the route to server's address; find via and dev.
The chapter descriptor is a root-owned JSON file at /etc/lab/chapter.json. On router, read it and require chapterId 3. Its table value starts with lc3_ and identifies this session's only editable firewall table. The exact suffix varies. Save the table name in the current router shell; if you open a new shell later, run the assignment again.
python3 -m json.tool /etc/lab/chapter.json
LC_TABLE=$(python3 -c 'import json; print(json.load(open("/etc/lab/chapter.json"))["table"])')
printf '%s\n' "$LC_TABLE"
labnet nft -a list chain inet "$LC_TABLE" transitmachine: router prompt: admin@router:~$ answer: LC_TABLE=$(python3 -c 'import json; print(json.load(open("/etc/lab/chapter.json"))["table"])') output:
hint: In router shell, assign LC_TABLE from the root-owned chapter descriptor.
machine: router prompt: admin@router:~$ answer: labnet nft -a list chain inet "$LC_TABLE" transit output:
hint: List this session's transit chain with handles and counters.
-a shows nftables rule handles. Confirm the table contains transit, a chain attached to hook forward, with a port 8081 drop labeled lc-control-deny before a port 8080 accept labeled lc-service-allow. Do not guess a handle or table name from an example. If the descriptor or table is absent, stop and restart with a new lab session rather than creating policy outside setup.
First predict the two client outcomes. The route and router forwarding can be healthy while one TCP port is denied. On server, ss shows whether a program is listening on each assigned port. A listener is a program waiting for connections; -l selects listeners, -n numeric ports and -t TCP. The service must bind only to 10.71.20.20, the exercise address.
>>> A listening socket is necessary for an answer, but the path to it can still be blocked. Router's rule matches client-to-server TCP destination port 8081, not port 8080.
On server, inspect the two listeners and request the denied-from-client port locally. The web endpoint is /health.txt. A successful live response should contain linux-camp-networking-ok; the lesson does not supply a captured result.
ss -H -lnt '( sport = :8080 or sport = :8081 )'
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txtmachine: server prompt: admin@server:~$ answer: ss -H -lnt '( sport = :8080 or sport = :8081 )' output:
hint: Confirm both TCP listeners use server's exercise address.
machine: server prompt: admin@server:~$ answer: curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txt output:
hint: From server itself, ask the 8081 service for health.txt.
On client, test both ports with the same bounded request. --noproxy '*' connects directly, and the timeouts prevent a blocked connection from holding the shell indefinitely. Read the live exit and message. A curl timeout alone cannot identify the firewall; pair it with service and policy evidence.
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8080/health.txt
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txtmachine: client prompt: admin@client:~$ answer: curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8080/health.txt output:
hint: From client, request the allowed 8080 health endpoint.
machine: client prompt: admin@client:~$ answer: curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txt output:
hint: From client, request the same server's restricted port and inspect the timeout.
The input hook sees a packet addressed to the machine that owns that hook. The forward hook sees a packet the machine routes onward. Client's request enters router on eth1, but its IP destination is server, so router's forward hook sees it. Server's input hook can inspect the packet after it reaches server. A ping to router's own 10.71.10.1 tests router input, not the web request's forward path.
>>> Router did not create the request, and the destination is not router. The packet is transit traffic. A successful ping to router does not prove this TCP request can pass the forward chain.
An nftables table contains chains; a chain contains ordered rules. A match can specify incoming interface, outgoing interface, source IP, destination IP and TCP destination port. Read the existing 8081 rule in this order: iifname eth1 means it arrived on router's client link; oifname eth2 means routing selected router's server link; ip saddr 10.71.10.10 and ip daddr 10.71.20.20 name the two endpoints; tcp dport 8081 chooses the restricted service; counter drop counts a match and discards that packet. Its comment lc-control-deny is an identifying label, not another match. The first matching rule in this chain gives a verdict. An accept completes this chain's decision but cannot override a drop in another chain. The chapter's transit chain has policy accept for packets neither rule matches. Its port 8080 accept mainly provides a useful counter because this chain would otherwise accept that traffic.
Measure a rule counter around one bounded request. On router, read the packets value beside lc-control-deny. Then send one client request to 8081 and read the router chain again. Do not run Check chapter between the readings; it sends its own probes and changes counters. The delta may exceed one because TCP can retry a connection. The important result is a positive delta on the matching deny rule, with the service still healthy locally.
# router, before and after the client request
labnet nft -a list chain inet "$LC_TABLE" transit
# client, between those readings
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txtRead commands through labnet are allowed for this table; rule changes through labnet are restricted to its exact table. The root-owned lab_access table still protects management access and is outside this exercise.
The controlled fault inserts one port 8080 drop at the top of router's transit chain. Do this only after the healthy 8080 and denied 8081 baseline agrees with the setup. On router, use the table name read from this session's descriptor. Each match narrows the rule to client eth1 to router eth2, client IP, server IP and TCP destination port 8080. counter records matches. Insert once, then list the chain to see order and the new rule's own handle.
labnet nft insert rule inet "$LC_TABLE" transit iifname eth1 oifname eth2 ip saddr 10.71.10.10 ip daddr 10.71.20.20 tcp dport 8080 counter drop comment lc-injected-fault
labnet nft -a list chain inet "$LC_TABLE" transitmachine: router prompt: admin@router:~$ answer: labnet nft insert rule inet "$LC_TABLE" transit iifname eth1 oifname eth2 ip saddr 10.71.10.10 ip daddr 10.71.20.20 tcp dport 8080 counter drop comment lc-injected-fault output:
hint: On router, insert only the chapter-owned client-to-server port 8080 drop.
machine: router prompt: admin@router:~$ answer: labnet nft -a list chain inet "$LC_TABLE" transit output:
hint: Check that the injected drop precedes the 8080 accept and record its handle.
The exercise's machine check may send its own web probes, so the new rule's counter need not begin at zero. Use a fresh before reading when you measure a later request.
Before another probe, predict: client should still ping server, server should still answer itself on 8080, but client 8080 should fail. Client 8081 should remain denied. These different outcomes separate address reachability, a working listener and port-specific policy. Test them on the named machines. Use the same curl syntax shown in Section 3.2, changing only the port or machine where instructed.
# client
ping -n -I eth1 -c 2 -W 1 10.71.20.20
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8080/health.txt
# server
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8080/health.txtmachine: client prompt: admin@client:~$ answer: curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8080/health.txt output:
hint: From client, request 8080 after the router fault and inspect the failure.
After Exercise 3-11 and its machine check settle, make a separate counter comparison. Read router's chain and record the lc-injected-fault and lc-service-allow packet counts. Run one more client 8080 curl, then read the same chain again. Do not run Check chapter or another exercise between these two readings: chapter checks send their own web probes. If the new drop counter rises while the allow counter does not, the earlier drop matched first. TCP retries can make the increase larger than one. A counter proves a rule matched traffic; it does not by itself prove what appeared on every interface.
The combined observations support this bounded explanation: client can route and ping to server, server's service answers locally, the client TCP request fails, and the specific router forward-drop rule counts the test. They do not constitute an interface-by-interface TCP capture. The bounded Packets recorder in this chapter can show ICMP path tests, but it does not record this TCP web request. Do not use an empty Packets view as proof that the request never crossed a link. Even a future bounded TCP capture would need a known listening window, capture point and matching request before absence could be interpreted. Stop and inspect the actual route and rule order if evidence contradicts the prediction. Run Check chapter after your counter readings. Its seven checks should identify the permitted application as failing while the negative control and network invariants remain intact.
Use the next hint only if your own evidence has not yet explained the result.
1. Ask whether client chose router as the next hop and whether server answers its own 8080 request. 2. Compare the first matching 8080 rule with the rule below it. Which counter rises when one client request runs? 3. The measured counter rose on which exact rule? What does that establish, and what packet-path detail remains unobserved without a TCP capture? 4. Read the faulty rule's handle from this session's chain listing. Identify the smallest deletion that leaves lc-control-deny in place.
If the route through router is intact, server answers itself on 8080, the client request fails and the new drop counter rises during that request, the evidence points to the injected rule. This does not trace every packet across router. Preserve the healthy service and the required 8081 denial. Do not flush a table or modify lab_access.
On router, list the chain with -a and find the handle on the line labeled lc-injected-fault. The number is assigned in this session. Replace NUMBER in the next form with that observed number. If the fault was inserted twice, each copy has its own handle and must be identified separately before deletion.
labnet nft -a list chain inet "$LC_TABLE" transit
labnet nft delete rule inet "$LC_TABLE" transit handle NUMBERThe deletion should print no success text. Verify the decision and the service separately. Client 8080 must answer again; client 8081 must remain denied; server 8081 must answer locally; client and server must each ping the other. Re-list the chain to confirm lc-control-deny and lc-service-allow remain and lc-injected-fault is gone. Do not compare counters across a Check chapter run as if no traffic had occurred.
# client
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8080/health.txt
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txt
ping -n -I eth1 -c 2 -W 1 10.71.20.20
# server
curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txt
ping -n -I eth1 -c 2 -W 1 10.71.10.10
# router
labnet nft -a list chain inet "$LC_TABLE" transitmachine: client prompt: admin@client:~$ answer: curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8080/health.txt output:
hint: After deleting the observed fault handle, request 8080 from client again.
machine: client prompt: admin@client:~$ answer: curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txt output:
hint: The client-to-server 8081 control must still time out after repair.
machine: server prompt: admin@server:~$ answer: curl --noproxy '*' --fail --connect-timeout 2 --max-time 4 http://10.71.20.20:8081/health.txt output:
hint: From server itself, verify 8081 still answers locally.
After the last counter reading, run Check chapter. Its seven results test the allowed response, client-side 8081 denial, local 8081 health, deny-rule counter growth, both ICMP directions, listener binding and unchanged unrelated network state. A green result describes this moment's machine state. It does not read your diagnosis or prove sustained service reliability.
Apply the mechanism to a changed case without changing this lab. On a later attempt, suppose client can still ping server but its port 8080 request times out. A current route lookup still chooses router. Server's listener still binds 10.71.20.20:8080. Router's lc-service-allow counter rises during the client request, and a separate, named drop rule in a chain at server input also rises. Use the same route, listener, request and before/after counter method to explain which gate now blocks the request. Predict why deleting a router rule would leave this failure in place. A local request from server to its own service may take a different path from a remote arrival, so do not use it alone to exclude server input policy. These are stated conditions for a changed reasoning case, not live output from this session.
Write a diagnosis from your own run: cite the before and after client 8080 results, local service result, route, rule order and counter readings. Name the one handle you removed and show that the 8081 control stayed blocked from client but healthy on server. State one
Practice Chapter 3. Forwarding policy with nftables in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.