Learn › Linux Foundations › Text Processing

join

join - a hands-on Linux lab on a real virtual machine.

Stitch two files into one with join, matching lines on a shared key field. Learn the default field-1 match, the -1 and -2 key selectors, the -t delimiter, and the -a outer join, plus the defining "is not sorted" error, all in the practice terminal on a tiny two-line fixture.

You have two small files sitting side by side. One is a list of ID numbers and names. The other is a list of the same ID numbers and cities. Nobody has stitched them together, so right now you cannot answer a simple question: which city does alice live in?

The names live in one file. The cities live in another. The only thing they share is the ID number in front of each line. That shared number is the thread you can pull to sew the two files into one. The command that pulls it is called join, and this lesson is all of it.

The black boxes in this lesson are a practice terminal: a safe sandbox that only checks the one command each step teaches, so you cannot break anything. At the end of the module one real Linux machine boots just for you, and you run the full mission there.

join takes two files and matches their lines on a shared value called the key. Think of the key as an ID number that appears in both files. When join finds the same key in both, it prints one combined line: the key, then the rest of the first file's line, then the rest of the second file's line. It is the same idea a database uses to link two tables by a shared column.

There is one rule you must respect from the very first command, and it is the rule that trips up almost everyone. Both files must already be sorted on the key field. join walks the two files top to bottom in lockstep, so it only works if the keys line up in order. If they do not, join stops and complains. You will see that exact complaint later in the lesson.

The key is just the field the two files have in common. By default that is the FIRST field of each line: the first word before a space. Later you will learn to point join at a different field, but field 1 is where it looks unless you tell it otherwise.

Here are the two fixture files, and they are already sorted on the ID in field 1.

ids.txt holds two lines:

1 alice
2 bob

cities.txt holds two lines:

1 NYC
2 LA

Both share the ID in field 1, so join has everything it needs. Hand it the two filenames, first file first, and let it match:

join ids.txt cities.txt

prompt: student@linuxcamp:~$ answer: join ids.txt cities.txt output: 1 alice NYC 2 bob LA hint: Type join, a space, the first file, a space, the second file: join ids.txt cities.txt

Two files became one. Read the first line: 1 alice NYC. join found ID 1 in both files, so it printed the key 1 once, then alice from ids.txt, then NYC from cities.txt. The second line did the same for ID 2. The shared key is never repeated; it appears once at the front, then the two files' remaining fields follow in order. That is the whole shape of a join.

That last command relied on a default: join matched on field 1 of BOTH files because you did not say otherwise. Two flags let you say it out loud, or point at a different field when the key is not the first one.

Since the ID is field 1 in both files, -1 1 -2 1 names exactly what the default already did. Run the explicit form and confirm you get the identical result:

join -1 1 -2 1 ids.txt cities.txt

prompt: student@linuxcamp:~$ answer: join -1 1 -2 1 ids.txt cities.txt output: 1 alice NYC 2 bob LA hint: -1 1 sets file one's key to field 1, -2 1 does the same for file two: join -1 1 -2 1 ids.txt cities.txt

Exactly the same two lines as the bare join. That is the point: -1 1 -2 1 is the long way of writing the default. The value of these flags shows up when the shared column is NOT field 1. If the ID sat in the third field of the second file, you would write -2 3 to point join at it. For this fixture the key is field 1 on both sides, so the explicit form and the default agree to the letter.

So far join split each line on spaces to find the fields. Real data often uses a different separator. The file /etc/passwd, for example, splits its fields with colons. The -t flag tells join which single character separates the fields, on both input and output.

Two colon-delimited fixture files are here. idc.txt holds 1:alice and 2:bob. citc.txt holds 1:NYC and 2:LA. To join on the colon-delimited key, write the flag as -t immediately followed by the character, so a colon separator is -t:. Run it:

join -t: idc.txt citc.txt

prompt: student@linuxcamp:~$ answer: join -t: idc.txt citc.txt output: 1:alice:NYC 2:bob:LA hint: Put the delimiter right after -t with no space, so a colon is -t: then the two files: join -t: idc.txt citc.txt

The output is glued with colons too: 1:alice:NYC. Notice -t did two jobs at once. It taught join that a colon splits the input fields, AND it used a colon to separate the output fields. Without -t, join would have treated each whole line as one field, found no matching key, and printed nothing. Match the delimiter to your data and join reads it correctly.

Here is the behaviour that surprises people, so meet it in the sandbox. By default join prints ONLY the keys that appear in both files. A key that lives in one file but not the other is silently dropped from the output. It is not an error and there is no warning; the unmatched line just does not show up.

The -a flag is the escape hatch. -a N tells join to ALSO print the unpaired lines from file N, which turns the plain match into what databases call an outer join. So -a 1 keeps every line of the first file, even the ones with no partner in the second. For this fixture both IDs match, so there is nothing left over to show, which is itself the lesson: with a full match, plain join and -a 1 look the same. Read the reveal for the shape you would see when a line has no partner.

Picture ids.txt gaining a third line, 3 carol, while cities.txt still stops at ID 2. Plain join ids.txt cities.txt would print only 1 alice NYC and 2 bob LA; carol has no city, so her line vanishes without a word. Run join -a 1 ids.txt cities.txt on that same pair and you also get a third line, 3 carol, printed with no city field because there was no match to fill it. That is the fix for the disappearing-line surprise: when you expect a line and it is gone, the key had no partner, and -a is how you keep it.

The one rule from the start of the lesson has teeth. If either file is not sorted on the key, join stops and tells you exactly which line broke the order. This is the defining join error, and knowing it on sight saves real time.

Imagine a file bad.txt whose two lines are in the wrong order: 2 bob on the first line, then 1 alice on the second. The key 1 comes AFTER 2, which is backwards. Point join at it and read what it says:

join bad.txt cities.txt

prompt: student@linuxcamp:~$ answer: join bad.txt cities.txt output: join: bad.txt:2: is not sorted: 1 alice join: input is not in sorted order hint: Just run the join on the out-of-order file to see the error: join bad.txt cities.txt

Read the error like a map. join: bad.txt:2: names the file and the exact line number, line 2, where the order broke. is not sorted: 1 alice shows you the offending line: key 1 arrived after key 2, out of order. The fix is always the same and it is the first thing to check: sort each file on the key BEFORE you join, usually with the sort command, which is its own lesson. Sort first, then join, and this error never appears.

Scaffolding off. No command is shown this time. You have every piece you need.

You are back with the two colon-delimited files, idc.txt (1:alice / 2:bob) and citc.txt (1:NYC / 2:LA). Both are already sorted on the key. Join them into one, matching on the shared ID in field 1, and remember that these files use a colon between fields, not a space. Pick the flag that sets the delimiter, and point join at the two files.

prompt: student@linuxcamp:~$ answer: join -t: idc.txt citc.txt output: 1:alice:NYC 2:bob:LA hint: The colon delimiter is -t: with no space, then the two filenames: join -t: idc.txt citc.txt

From memory, the two files merged on their colon-delimited keys: 1:alice:NYC and 2:bob:LA. You recalled two things at once, that join matches on field 1 by default, and that -t: teaches it the colon separator, and you typed them without a prompt. That is exactly the move the real machine will ask of you.

You earned this cheat sheet. Every row is a form of join you have already run or read:

Two rules carry the whole command. First, both files must be sorted on the key, or you get is not sorted. Second, by default only matching keys appear; use -a to keep the leftovers.

join almost never works alone. Because both inputs must be sorted, the real-world pattern is to sort each file on the key first, then join the sorted results. sort has its own lesson in this module; pair it with join and the not-sorted error disappears.

The practice terminal has shown you every form of join: the default match on field 1, the explicit -1 and -2 key selectors, the -t delimiter flag, and the -a outer join, plus the defining is not sorted error and the fact that unmatched keys silently drop.

The Text Processing module ends with one real machine, the text-processing capstone mission, and that is where you join files for real. A Linux VM boots just for you. Its objectives use exactly what you just ran, alongside the module's other text tools, sort, cut, paste, and the rest, each of which has its own lesson. One difference from here: the mission shows no commands. You read the objective, recall the flag, and type it. That recall is what makes it stick.

Finish the other Text Processing lessons, then go match the files for real.

Practice join in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.