Learn › The Network Stack › What Moves on the Wire

ARP

ip neigh - a hands-on Linux lab on a real virtual machine.

ARP answers one question: "I know your IP, what is your MAC?" Watch it happen from scratch. See every field. Flush the cache and trigger it again.

Your machine is about to shout at everybody

Every Ethernet frame needs a destination MAC address. You proved that two lessons ago by reading one out of the raw bytes.

Now look at the problem that creates. When your machine wants to send a packet to 10.0.1.20, the only address it has is that IP. It does not know the MAC address. It cannot build the frame. And there is no directory it can look this up in, no server to ask, nothing on disk that knows the answer.

So it asks. Out loud. To every machine on the local network at once.

In this lab you get to watch that happen from nothing, because the client's memory of the server has been wiped before you arrived.

The black boxes below are a practice terminal: a safe sandbox that checks only the one command each step teaches. The real VM waits at the end. There you are student on the machine arp-lab, and inside it live two namespaces joined by a virtual cable. A namespace is a private copy of the network stack, a separate computer as far as networking is concerned. Your progress is tracked automatically, so type commands naturally. Timestamps and id numbers in your own captures will differ from the printed screens. Every address and field this lesson is about will match.

The question ARP exists to answer

That shout has a name. ARP, the Address Resolution Protocol, answers exactly one question: "I know your IP address. What is your MAC address?"

It is the bridge between Layer 3 and Layer 2. Layer 3 is where IP addresses live, the addressing that works across the whole internet. Layer 2 is where MAC addresses live, the addressing that works on one cable segment.

Without ARP your machine could build IP packets all day long and never send one. It would have no way to wrap them in Ethernet frames, because the destination field on the envelope would be a blank it could not fill.

Two steps: one broadcast, one answer

The whole protocol is two messages.

1. ARP Request (broadcast). Your machine sends a frame to ff:ff:ff:ff:ff:ff, the broadcast MAC address. Every device on the local network receives it. Inside is a simple message: "Who has IP 10.0.1.20? Tell 10.0.1.10."

2. ARP Reply (unicast). The machine that owns 10.0.1.20 responds directly, not broadcast, with: "10.0.1.20 is at 02:00:0a:00:01:14." Only the original requester receives this reply.

After the reply arrives, the kernel writes the IP-to-MAC mapping into a local cache called the ARP table, also called the neighbor table. The next time it needs to send to 10.0.1.20 it already knows the MAC and skips the exchange entirely.

That cache is why you have never noticed ARP happening. It runs once and then stays quiet for minutes.

Where this protocol came from

ARP is old. David C. Plummer wrote it up at MIT, and it was published as RFC 826 in November 1982. That is before the web existed, and before almost every machine running it today was designed.

What happened next is the remarkable part: nothing. The packet you are about to capture carries the same fields, in the same order, that the 1982 document specified. There is no version number in an ARP message, no negotiation and no extension in common use. It was small enough and general enough that it never needed a second draft.

That generality is visible in the packet itself. Nothing in the format hard-codes Ethernet or IPv4. Both are named by number inside the message, which is how one protocol from 1982 kept working on hardware nobody had invented yet.

The price of that simplicity is the part that never got fixed. An ARP message has no field for proving the sender is who it says it is, because in 1982 nobody on the network was a stranger. You will meet the consequence of that near the end of this lesson.

Your lab network

This lab uses the same two-namespace network as the previous lessons:

  [client]  10.0.1.10  <-------->  10.0.1.20  [server]
  MAC: 02:00:0a:00:01:0a           MAC: 02:00:0a:00:01:14

One thing is different here: the ARP cache starts empty. The client does not know the server's MAC address yet. When you send the first ping, you watch ARP from the very first broadcast.

One more thing to know before you type anything. Your shell is not inside the client. Client and server are network namespaces, separate network stacks inside this one machine, each with its own interfaces and its own ARP table. To run a command inside one, you prefix it with sudo ip netns exec client. The helpers do that for you: send-ping, cap and flush-arp all run inside the client namespace. When you want something the helpers do not cover, write the prefix yourself.

Those helper names tab-complete. Type flu and press Tab and the shell finishes flush-arp. Type send-p and Tab for send-ping, che and Tab for check-progress. On a command as long as the namespace prefix, letting the shell finish names for you removes a whole class of typo.

Look in the empty drawer

Start by proving the client knows nothing. ip neigh (short for ip neighbour) prints the kernel's neighbor cache, which for IPv4 is the ARP cache. Run it inside the client namespace and it answers about the client.

Before you press Enter, decide what an empty table looks like on screen. A header with no rows? A message saying the table is empty? Something else?

sudo ip netns exec client ip neigh

prompt: student@arp-lab:~$ answer: sudo ip netns exec client ip neigh ||| sudo ip netns exec client ip neigh show ||| sudo ip netns exec client ip n ||| sudo ip netns exec client ip n show ||| sudo ip netns exec client ip neighbour ||| sudo ip netns exec client ip neighbour show output: hint: The neighbor table, read from inside the client namespace: sudo ip netns exec client ip neigh

Nothing at all. No header, no message, just your prompt back. An empty table prints no lines, and that silence is the answer: there is no line for 10.0.1.20, so the client does not know the server's MAC address.

Get used to reading this command as a list of what the machine has learned. One line per neighbor it can currently address. Zero lines means it has learned nothing yet, and reading that table is the first of the five things your lab grades.

Leave the sudo ip netns exec client prefix off and the command still works, but it answers about the wrong machine: you get the ARP table of the VM itself, which knows nothing about veth-c or 10.0.1.20. Every ip command in this lesson needs the prefix.

You can also use ip neigh show, ip n show, or simply ip n. They all do the same thing. On older systems, the arp -a command shows similar information in a different format.

Commit: what leaves the machine first

The table is empty. In a moment you will send a ping from the client to 10.0.1.20 and capture what crosses the wire.

[ ] The ICMP echo request. The ping is what you asked for, so it goes first.
[x] An ARP request, broadcast to every machine on the segment, asking who owns 10.0.1.20.
[ ] Nothing. The ping fails immediately, because the MAC address is unknown.

>>> An ARP request goes first. The kernel cannot build the Ethernet frame around your ping until it has a destination MAC to write on it, so it holds your packet and asks the network. If you picked the echo request, that is the natural reading and it is exactly one layer too high. Your ping is the cargo. Something has to address the envelope before the cargo can travel. If you picked failure, notice how much has to go right for that to be true: the kernel does not give up on an unknown MAC, it goes and finds one. A ping only fails this way when nobody answers the question at all, and then the entry lands in the table as FAILED, which you will meet shortly.

Watch it happen live

Now capture the exchange. -Y arp displays only ARP packets, and -V prints every field of each one, fully decoded.

Timing matters here more than anywhere else in this lesson. The whole ARP exchange is over in well under a millisecond, right at the start of the ping, and tshark needs a moment to open the capture socket. A ping launched at the same instant has already finished asking by the time tshark is listening, and you see nothing.

So on the real VM you hold the ping back and let the capture get there first. (sleep 2; send-ping) & runs both commands in the background: wait two seconds, then ping. Meanwhile cap is already capturing in front of you. The practice terminal handles that timing here, so you type only the capture.

cap -c 4 -Y arp -V

prompt: student@arp-lab:~$ answer: cap -c 4 -Y arp -V ||| cap -c 4 -V -Y arp ||| cap -Y arp -V -c 4 output: Running as user "root" and group "root". This could be dangerous. Capturing on 'veth-c' ... (Frame metadata: capture time, frame length, protocol chain) ... Ethernet II, Src: 02:00:0a:00:01:0a, Dst: ff:ff:ff:ff:ff:ff ... (destination and source address sub-lines) ... Type: ARP (0x0806)

Address Resolution Protocol (request) Hardware type: Ethernet (1) Protocol type: IPv4 (0x0800) Hardware size: 6 Protocol size: 4 Opcode: request (1) Sender MAC address: 02:00:0a:00:01:0a Sender IP address: 10.0.1.10 Target MAC address: 00:00:00:00:00:00 Target IP address: 10.0.1.20

... (the second packet, the reply, follows in the same shape) ... 4 packets captured hint: Four packets, display only ARP, decode every field: cap -c 4 -Y arp -V

That is the shout, printed field by field. Read every line of the request:

Ethernet destination: ff:ff:ff:ff:ff:ff is a broadcast. Every device on the network receives this frame.
Type: ARP (0x0806) is the EtherType you learned in the previous lesson.
Hardware type: Ethernet (1) says which link layer we are on. ARP works on others too.
Protocol type: IPv4 (0x0800) says which kind of address we want mapped.
Hardware size: 6 because MAC addresses are 6 bytes long.
Protocol size: 4 because IPv4 addresses are 4 bytes long.
Opcode: request (1) marks this as a question, not an answer.
Sender MAC: 02:00:0a:00:01:0a is the requester's MAC, the client.
Sender IP: 10.0.1.10 is the requester's IP.
Target MAC: 00:00:00:00:00:00 is unknown. This is the blank being filled in.
Target IP: 10.0.1.20 is the IP we want the MAC for.

The whole protocol is in those last four lines. Three of them are filled in and one is zeros, and the zeros are the entire point of the packet.

Capturing the exchange in full dissection is the second of the five things your lab grades.

Nothing printed? The cache filled in on an earlier attempt, so ARP had no reason to run. Flush it (a few steps from now) and try again. Silence from an ARP filter is almost always a full cache, not a broken filter.

Commit: who hears the answer

The question went to everybody. The server owns 10.0.1.20, so it is the machine that answers.

[ ] All of them. A broadcast question gets a broadcast answer, so everybody learns the mapping.
[x] Only the client. The reply is unicast, addressed to the MAC that asked.
[ ] Only the switch, which stores the answer and hands it out to whoever asks next.

>>> Only the client. The question had to be a broadcast because the sender had no address to aim it at. The answer does not: the request carried the sender MAC, so the server knows exactly where to send the reply and addresses it there. If you picked the broadcast answer, that is a reasonable symmetry and the network deliberately breaks it. Every broadcast interrupts every machine on the segment, so a protocol that broadcasts only when it has no other choice is a protocol that scales. If you picked the switch, no such service exists at this layer. A switch forwards frames and learns MAC addresses for its own forwarding decisions, but it never answers ARP on another machine's behalf. That is the next lesson.

Read the reply

The second packet in that same capture is the ARP Reply. Same eleven fields, different values. Trimmed here to the ones that changed:

Ethernet II, Src: 02:00:0a:00:01:14, Dst: 02:00:0a:00:01:0a
    Type: ARP (0x0806)

Address Resolution Protocol (reply)
    Opcode: reply (2)
    Sender MAC address: 02:00:0a:00:01:14
    Sender IP address: 10.0.1.20
    Target MAC address: 02:00:0a:00:01:0a
    Target IP address: 10.0.1.10

Four differences, and each one is worth naming:

The Ethernet destination is now unicast (02:00:0a:00:01:0a), not broadcast. The reply goes only to the requester.
Opcode: reply (2) instead of request (1).
The sender fields now hold the server's information, 10.0.1.20 and its MAC.
The target MAC is filled in, no longer zeros. It holds the client's MAC.

After this exchange the client knows that 10.0.1.20 has MAC 02:00:0a:00:01:14. The ICMP ping packets can now be wrapped in Ethernet frames and delivered.

Same command, one line appeared

You ran this command a few minutes ago and it printed nothing at all. Nothing about the command changes now. Everything about the answer does.

Before you press Enter, picture the line: which pieces of information does the kernel have that it did not have before?

sudo ip netns exec client ip neigh

prompt: student@arp-lab:~$ answer: sudo ip netns exec client ip neigh ||| sudo ip netns exec client ip neigh show ||| sudo ip netns exec client ip n ||| sudo ip netns exec client ip n show ||| sudo ip netns exec client ip neighbour ||| sudo ip netns exec client ip neighbour show output: 10.0.1.20 dev veth-c lladdr 02:00:0a:00:01:14 REACHABLE hint: Exactly the command you ran before, unchanged: sudo ip netns exec client ip neigh

One line where there were none, and it is the answer to the question you watched being asked.

Read it left to right: the IP, dev veth-c for the interface the answer arrived on, lladdr (link-layer address, meaning the MAC), and then the state. The mapping is cached, so the next packet to 10.0.1.20 goes out with no ARP at all.

REACHABLE means the kernel recently confirmed this mapping works, and it is the state you are most likely to catch right after a ping. Wait a minute, look again, and the same entry reads STALE. Entries move through states as the kernel's confidence ages:

State	What it means
`REACHABLE`	Recently confirmed. Trusted.
`STALE`	Not confirmed recently. Will re-verify on next use.
`DELAY`	Waiting to re-verify (a timer is running).
`PROBE`	Actively sending ARP to re-verify.
`INCOMPLETE`	ARP request sent, no reply received yet.
`FAILED`	ARP request got no reply. The target may be unreachable.

Two of those states are diagnoses in their own right. INCOMPLETE means the question went out and nothing came back yet. FAILED means nothing came back at all, and that is where you look first when a machine on your own segment is unreachable for no obvious reason.

Clear it and watch it go

Once ARP has run, the kernel caches the result, and a cached answer means no more broadcasts to watch. To see ARP happen again you have to clear the cache.

The flush-arp helper does it for the client namespace. Expect it to say so out loud.

flush-arp

prompt: student@arp-lab:~$ answer: flush-arp output: ARP cache flushed for client namespace. hint: The helper that empties the client neighbor table: flush-arp

The helper is a two-line script and it is worth knowing what is inside it: sudo ip netns exec client ip neigh flush dev veth-c, then the message you just read. Run that long form yourself and it does exactly the same job, except it prints nothing at all, which is normal for ip. It stays quiet when it succeeds.

Neither form tells you what it removed, so check the table again with sudo ip netns exec client ip neigh and you will find the line for 10.0.1.20 gone. The next packet aimed at that address will trigger a fresh ARP exchange.

Clearing the cache is the third of the five things your lab grades.

On a production server, flushing the ARP cache causes a brief interruption because the kernel has to re-learn the MAC for every active connection. In this lab it is safe because you only have one neighbor.

Milestone: you watched an address get resolved

Stop and count what you have seen. An empty table. A broadcast question with a hole in it where the answer belongs. A unicast reply that filled the hole. And then the same table, with a line in it that was not there before.

That loop is the whole protocol, and it is running on every network you will ever touch, thousands of times a minute, almost always invisibly. You have now seen it from the outside, in the capture, and from the inside, in the kernel's own table.

What is left is pulling the fields out on demand, watching ARP and the traffic it enables in one screen, and knowing why this protocol is dangerous.

Pull the four addresses out as columns

You extracted EtherType fields in the previous lesson with -T fields and one -e per field. ARP fields work the same way, and the four that matter are the sender and target hardware addresses and the sender and target protocol addresses.

On the real VM this needs the same two setup steps every time. Flush first, or there is no ARP to extract. Delay the ping with (sleep 2; send-ping) &, or the capture starts too late to see it. Here the practice terminal has done both, so you type only the capture.

cap -c 2 -Y arp -T fields -e arp.src.hw_mac -e arp.dst.hw_mac -e arp.src.proto_ipv4 -e arp.dst.proto_ipv4

prompt: student@arp-lab:~$ answer: cap -c 2 -Y arp -T fields -e arp.src.hw_mac -e arp.dst.hw_mac -e arp.src.proto_ipv4 -e arp.dst.proto_ipv4 output: Running as user "root" and group "root". This could be dangerous. Capturing on 'veth-c' 02:00:0a:00:01:0a 00:00:00:00:00:00 10.0.1.10 10.0.1.20 02:00:0a:00:01:14 02:00:0a:00:01:0a 10.0.1.20 10.0.1.10 2 packets captured hint: Fields mode with four -e flags, ARP only: cap -c 2 -Y arp -T fields -e arp.src.hw_mac -e arp.dst.hw_mac -e arp.src.proto_ipv4 -e arp.dst.proto_ipv4

Four columns: sender MAC, target MAC, sender IP, target IP. The first row is the request, and you can identify it without being told: the target MAC is all zeros because it is the unknown. The second row is the reply, with that same column filled in with the client's address.

The whole dissection you read earlier is available this way, one field at a time. These are the names worth remembering:

Field	What it shows
`arp.src.hw_mac`	Sender hardware (MAC) address
`arp.dst.hw_mac`	Target hardware (MAC) address
`arp.src.proto_ipv4`	Sender protocol (IP) address
`arp.dst.proto_ipv4`	Target protocol (IP) address
`arp.opcode`	Operation code (1 = request, 2 = reply)
`arp.hw.type`	Hardware type (1 = Ethernet)

Note the vocabulary in the field names. Hardware means MAC, protocol means IP. That is ARP's own language, and it is why the protocol can map addresses it has never heard of: nothing in the packet format is specific to Ethernet or to IPv4.

Extracting ARP fields is the fourth of the five things your lab grades.

Commit: an answer nobody asked for

One more property of this protocol, and it is the one that keeps security people awake. Consider a machine on the segment

Practice ARP in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.

ARP