Learn › Salt Configuration Management › Salt Fundamentals
Manage two minions at once. Accept both keys, target individual and groups, and compare grains.
Everything you have run so far went to a single machine. One master, one minion, one reply.
Today a second minion is listening on the same wire, and nothing about the command you type gets longer. You will type one line and two servers will answer. Run a second line and the two replies can land in the opposite order, with the second machine printing first.
Nothing is broken when that happens. That flip is the whole point of this lesson. You have stopped talking to a machine and started talking to a room.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output shown was captured from a real Salt master (salt-master 3008.2 (Argon), Debian 12) with two live minions named minion1 and minion2. Nothing is typed by hand.
On the lab machine you are the saltops user. Salt's key store and job cache belong to root, so the lab grants saltops permission to run Salt through sudo. That is why every command here starts with sudo.
A Salt command has three parts, always in the same order:
salt <target> <module.function>
The target is who. It answers the question which machines should hear this. The module.function is what: the job those machines should run. In salt '*' test.ping, the target is '*' and the function is test.ping.
Think of it as addressing an envelope. The target is the address line and the function is the letter inside. Change the address and the same letter goes to different servers.
Two details worth fixing in your head now. First, test.ping is not an ICMP ping, and Salt's own built-in help says exactly that. It asks the salt-minion process to answer True, which proves the Salt service is alive, not just the network.
Second, always wrap the target in quotes. Without them, bash expands a * against the files in your current directory before Salt ever sees it.
Salt was written by Thomas Hatch in 2011 because the tools of the day walked a server list one connection at a time. A thousand servers meant a long wait. He built Salt on a message bus instead.
Here is what that means when you press Enter. The master does not loop over your minions. It publishes one job onto a bus that every accepted minion is already subscribed to. All of them receive it.
Each minion then compares the target against its own ID, decides for itself whether it matches, and only the matching ones run the job and send a return. That design is why targeting is a first-class skill, and it is why replies arrive in whatever order the minions finish.
>>> minion2 decides. It receives the job like everyone else, compares the target minion1 against its own ID, finds no match, and simply does not run or return anything.
If you picked the master: the master has no per-server loop at publish time. It writes the job once and the bus fans it out. That is exactly why Salt stays fast at a thousand minions.
If you picked the bus: the bus is a dumb pipe here. It carries the job to every subscriber without inspecting it. Matching is minion-side work, which is also why a powered-off minion simply never answers instead of raising an error on the master.
Remember from the first Salt lab: a minion can shout at the master all day, but it cannot be commanded until its key is accepted. Two fresh minions booted into your lab.
salt-key -L lists every key the master knows about, sorted into buckets by trust state. Before you target anything, look at the door.
sudo salt-key -L
prompt: saltops@saltmaster:~$ answer: sudo salt-key -L ||| sudo salt-key --list-all output: Accepted Keys: Denied Keys: Unaccepted Keys: minion1 minion2 Rejected Keys: hint: The key manager is salt-key, and the flag that lists everything is -L: sudo salt-key -L
Four headings, and three of them have nothing under them. An empty heading is not an error, it is an empty bucket.
minion1 and minion2 both sit under Unaccepted Keys, which means the master has heard from them and is holding their keys unsigned. Until those two names move up to Accepted, every command you aim at them finds nobody home.
In the first lab you accepted one key by name with -a minion1. With two keys waiting there is a faster form.
-A accepts every pending key at once. -y answers the are-you-sure prompt for you, so the command does not stop and wait for a keystroke. Use them together.
sudo salt-key -A -y
prompt: saltops@saltmaster:~$ answer: sudo salt-key -A -y ||| sudo salt-key -y -A output: The following keys are going to be accepted: Unaccepted Keys: minion1 minion2 Key for minion minion1 accepted. Key for minion minion2 accepted. hint: Capital -A means accept all pending keys, and -y answers the prompt for you: sudo salt-key -A -y
Read it top to bottom. Salt first echoes what it is about to do, listing the keys under Unaccepted Keys so you can see the blast radius before it acts. Then it prints one confirmation line per key: Key for minion minion1 accepted. and the same for minion2.
Two lines, two minions, one command. Both machines are now under your control.
-A accepts whatever happens to be waiting, including a machine you did not rack. On a real network, anything that can reach the master can queue a key and hope you are in a hurry. In production, read the pending list first and accept by name with -a <minion-id>. Use -A when you already know exactly what is at the door, which is the case right now.
Type the exact command from two steps ago a second time. Nothing about the command changed. Something in the answer did. Before you press Enter, decide which of the four headings you expect to be empty this time.
sudo salt-key -L
prompt: saltops@saltmaster:~$ answer: sudo salt-key -L ||| sudo salt-key --list-all output: Accepted Keys: minion1 minion2 Denied Keys: Unaccepted Keys: Rejected Keys: hint: It is the same listing command you ran before you accepted anything: sudo salt-key -L
Put the two outputs side by side and exactly one thing is different. The names minion1 and minion2 used to sit under Unaccepted Keys. They now sit under Accepted Keys, and Unaccepted is the empty bucket.
Nothing about the minions themselves changed. No service restarted, no config was edited. The only thing that moved is which heading those two names live under, and that heading is the trust boundary. Above it a machine is a stranger. Below it a machine runs whatever you publish.
Now aim at both minions at once. The '*' target is a glob that matches every accepted minion, so salt '*' test.ping asks the whole room to prove it is alive.
Before you type it, commit to an expectation. Both minions are healthy and both keys are accepted.
>>> Both answer True, and the order is whatever the returns arrive in. You are about to see that for real.
If you picked sorted order: the master does not collect all the returns and sort them. It prints each one as it lands, so name order is a coincidence, never a promise.
If you picked a merged block: Salt keeps every minion's answer separate and labels it with the minion ID. That is what makes the output readable when one machine disagrees with the rest, which is most of the value of running a command fleet-wide.
Type it and watch which name lands first.
sudo salt '*' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt '*' test.ping ||| sudo salt "*" test.ping output: minion2: True minion1: True hint: The everything target is a quoted star, and the alive check is test.ping: sudo salt '*' test.ping
Look at the order. minion2 printed first. You did not ask for that, and you cannot ask for the opposite either.
That is the bus doing its job. Both minions got the job at the same instant, both ran it, and the master printed each return the moment it arrived. minion2 answered a hair sooner on this run. On the next run it might not.
The True under each name answers one specific question: is the salt-minion process alive and able to reply. It does not say the machine is healthy, only that Salt can reach it and it can run a job.
Never write a script that reads Salt output by position. The first block is not minion1, it is whoever was fastest. Read by name, always. This is the most common way beginner Salt automation breaks in production, and it breaks intermittently, which is worse than breaking every time.
Stop and take the win. Three commands ago you had two strangers knocking at the door. Now you have a fleet that answers a single line, and you can already read the two things that trip most people up. An empty key bucket is not a failure, and reply order is not information.
What is left is aim. '*' is a sledgehammer, and pointing a real command at every machine you own is how outages start. The rest of this lesson is about hitting exactly the machines you meant to hit.
The narrowest target is a minion ID typed out in full. 'minion1' has no wildcard in it, so it matches one machine and nothing else.
Pair it with a function that actually says something about the machine. Grains are the facts a minion reports about itself, gathered when the salt-minion service starts: its operating system, its kernel, its CPU count. You met them in the grains lab.
grains.item takes one or more grain names and returns just those. Ask minion1 for three of them:
sudo salt 'minion1' grains.item os osrelease kernel
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.item os osrelease kernel ||| sudo salt "minion1" grains.item os osrelease kernel ||| sudo salt minion1 grains.item os osrelease kernel output: minion1: ---------- kernel: Linux os: Debian osrelease: 12 hint: The target is the full minion ID in quotes, and the function takes the grain names after it: sudo salt 'minion1' grains.item os osrelease kernel
One block came back, labelled minion1. minion2 received the same published job, compared the target against its own ID, found no match and stayed silent. That is the prediction you made at the start, now on screen.
The row of dashes is how Salt opens a block of key and value pairs. Under it are the three grains: kernel is Linux, os is Debian and osrelease is 12.
Now notice something small. You asked for os, osrelease, kernel. Salt printed kernel, os, osrelease. The keys come back ordered by name, not in the order you asked for them. One more reason to read Salt output by label rather than by position.
A pattern only helps when the names share a shape. Sometimes you need three specific machines whose names have nothing in common.
The -L flag takes a literal comma-separated list of minion IDs. No wildcards, no guessing, just the names you typed. In a real fleet this is the flag for surgical work: -L 'web01,db03,cache07' touches exactly those three boxes.
sudo salt -L 'minion1,minion2' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -L 'minion1,minion2' test.ping ||| sudo salt -L "minion1,minion2" test.ping ||| sudo salt -L minion1,minion2 test.ping output: minion2: True minion1: True hint: The list flag is -L, then the IDs joined by commas with no spaces: sudo salt -L 'minion1,minion2' test.ping
Both answered, and minion2 came back first again. Note what did not happen: the order you typed the names in did not set the order they replied in. -L controls who runs the job, never who prints first.
The value of -L is that it cannot grow. A pattern might quietly pick up a machine somebody added this morning. A list matches the names you wrote and nothing else, which is what you want when the blast radius has to be provable.
Do not put a space after the commas. 'minion1,minion2' is correct. The -L matcher splits on the comma and does not strip surrounding spaces, so 'minion1, minion2' is not the same list you meant. The failure tends to read like a machine problem rather than a typo, so this one costs people real time.
Here is where two minions start earning their keep. Take the grains question you asked minion1 and point it at everything instead. The function does not change. Only the address on the envelope changes.
>>> Two blocks, each labelled. Salt never merges returns, even when they are identical. Every minion's answer stays attached to its own name.
If you picked one merged block: merging would hide the one machine that disagrees, which is the exact machine you are looking for. Salt keeps them apart on purpose.
If you picked an error: the target and the function are completely independent. Any function you can run against one minion runs against a thousand by changing only the target.
Run it and read the two answers against each other.
sudo salt '*' grains.item os osrelease
prompt: saltops@saltmaster:~$ answer: sudo salt '*' grains.item os osrelease ||| sudo salt "*" grains.item os osrelease output: minion1: ---------- os: Debian osrelease: 12 minion2: ---------- os: Debian osrelease: 12 hint: Keep the function and widen the target to the quoted star: sudo salt '*' grains.item os osrelease
Two labelled blocks with the same shape. Read straight down the os line and then the osrelease line: Debian and Debian, 12 and 12. The two machines agree, and it took one command to prove it.
That is fleet auditing in its smallest possible form. With two minions you are comparing two blocks by eye. With two hundred, the same command is how you find the one box still on the old release while everything else moved on.
And look at the order one more time. This run printed minion1 first. The '*' test.ping a few steps ago printed minion2 first. Same target, same pair of machines, nothing changed in between. If you needed a last piece of evidence that reply order carries no meaning, this is it.
Scaffolding off. No command is printed from here on. You have every piece you need.
Tomorrow somebody adds a database box called db01 to this master. You want a target that reaches your minion machines and could never accidentally include that new box, without typing every ID out by hand.
A * inside a target stands for any run of characters, so a target can carry a prefix in front of the star. Ping only the machines whose ID begins with minion.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion*' test.ping ||| sudo salt "minion*" test.ping output: minion1: True minion2: True hint: Keep the star, but put the prefix in front of it, still inside quotes. The function is the same alive check you used on the room.
sudo salt 'minion*' test.ping matched both machines, because both IDs begin with minion. Today that is the same set of machines '*' reaches. Tomorrow, with db01 accepted, it is not, and that is the point of writing the pattern.
This is why fleets are named on purpose. When your machines are web01, web02, db01 and cache07, the target 'web*' means all web servers and nothing else. Your naming convention quietly becomes your targeting language.
Order note, one last time: this run printed minion1 first, while the identical question sent to '*' printed minion2 first. Read by name.
Still no command shown.
You want to answer one question fast: what Debian release is every machine on. You do not want the labelled key and value block that grains.item prints. You want the bare value under each minion name, one grain only.
There is a second grains function for exactly this. It takes a single grain name and returns its value with no key printed. Fetch osrelease from every accepted minion.
prompt: saltops@saltmaster:~$ answer: sudo salt '*' grains.get osrelease ||| sudo salt "*" grains.get osrelease output: minion2: 12 minion1: 12 hint: The target is the everything glob. The function is the get form of grains, followed by the single grain name.
grains.get returns the value on its own. Compare that with the grains.item output higher up the lesson: same facts, but item prints the dashed block and the key names, while get gives you the bare value under each minion.
Reach for grains.item when you want to read several grains and see what each one is called. Reach for grains.get when you already know the grain and just want the answer across the fleet.
You earned this. Every row in the first table is a command you typed in this lesson.
| Command | What it does |
|---|---|
sudo salt-key -L | List every key the master knows, in four buckets |
sudo salt-key -A -y | Accept every pending key with no confirmation prompt |
sudo salt '*' test.ping | Ask every accepted minion to prove its Salt service answers |
sudo salt 'minion1' grains.item os osrelease kernel | Read named grains from one machine only |
sudo salt -L 'minion1,minion2' test.ping | Target an explicit list of minion IDs |
sudo salt '*' grains.item os osrelease | Read the same grains from every machine and compare |
sudo salt 'minion*' test.ping | Target every ID that starts with a prefix |
sudo salt '*' grains.get osrelease | Read one grain value from every machine |
And the ways to aim. The last two are named so you recognize them when you meet them, not because you used them here.
| Target kind | Flag | Example | What it matches | Used here |
|---|---|---|---|---|
| Everything | none | '*' | Every accepted minion | Yes |
| Glob | none | 'minion*' | Every ID matching the pattern | Yes |
| Exact ID | none | 'minion1' | Only that one |