Learn › Salt Configuration Management › Multi-Minion Orchestration
Master glob, regex, grain, pillar, and compound targeting patterns.
Every salt command you have run has had a target in it, and so far that target has been one of two things: the name of a machine, or a star meaning all of them. On two minions, those are all you need.
Real fleets are not two minions. Once there are five hundred, the useful question stops being which name do I type and becomes which machines am I talking about. All the web servers. Everything in staging. Everything except the databases. Not one of those is a name.
Salt answers that with a small selection language, and the flag in front of the target says what kind of question you are asking.
By the end of this lesson you will have selected the same single machine four different ways, asked for a set that cannot exist, and read a file already on this master where the same selection language decides which states each machine gets.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen was captured by booting this exact lab and doing exactly what you are about to do. The lab user is saltops, the master is saltmaster, and there are two minions, minion1 and minion2, both keys already accepted. Your progress in the lab is tracked automatically, so type commands naturally.
Before you select anything, confirm what there is to select from. salt-key is the master's key tool, and -L lists every key it knows about, grouped by state.
Before you press Enter, commit to a number. How many machines do you expect to find under Accepted Keys?
sudo salt-key -L
prompt: saltops@saltmaster:~$ answer: sudo salt-key -L ||| sudo salt-key --list-all ||| sudo salt-key -L all output: Accepted Keys: minion1 minion2 Denied Keys: Unaccepted Keys: Rejected Keys: hint: The key tool, with the flag that lists everything: sudo salt-key -L
Four headings, and only the first one has anything under it. minion1 and minion2 are accepted, which means the master will talk to them and they will answer.
The other three are empty, and that is worth noticing. Denied, Unaccepted and Rejected are the three ways a machine can be on the network and still be invisible to every command in this lesson. Two accepted machines is the whole population you are selecting from.
Grains are facts a minion works out about itself and reports to the master: its operating system, its memory, its CPU count. You met them in the grains lesson. You can also set your own, and this lab set two custom grains on each machine before you logged in: role and env.
grains.item prints named grains and nothing else, which is what you want when a machine has a couple of hundred. Ask both machines for those two.
sudo salt '*' grains.item role env
prompt: saltops@saltmaster:~$ answer: sudo salt '*' grains.item role env ||| sudo salt "*" grains.item role env ||| sudo salt '*' grains.item env role output: minion1: ---------- env: production role: web minion2: ---------- env: staging role: db hint: The grains module has a function that prints named grains: sudo salt '*' grains.item role env
Two blocks, one per machine, each opened by the machine's own name and a row of dashes.
minion1 reports env: production and role: web. minion2 reports env: staging and role: db. Those four values are the entire difference between these machines, and every grain matcher you write from here is a question about one of them.
The keys come back alphabetically, env before role, not in the order you asked for them. Salt sorts its output. Keep this screen in your head: from here on it is the table you predict from.
Pillar is the other half of that picture, and you met it in the pillar lessons. Grains come from the machine. Pillar comes from the master: data the master keeps about a machine, which the machine cannot change.
Pillar is assigned in a top file, exactly like states are. Read the assignment first.
cat /srv/pillar/top.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/top.sls output: base: 'minion1':
'minion2':
hint: Print the pillar top file: cat /srv/pillar/top.sls
Five lines, and the shape is the top file staircase. base: is the environment at column 0, each quoted target sits two spaces in, and each pillar file name sits four spaces in with a leading dash.
So minion1 is assigned the pillar file web, and minion2 is assigned db. The names carry no .sls, the same way state names do not, and both files live in /srv/pillar.
Now the two files that top file hands out. First the one assigned to minion1.
cat /srv/pillar/web.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/web.sls output: role: web http_port: 80 app_tier: frontend hint: Print the first pillar data file: cat /srv/pillar/web.sls
Three keys and three values, and the first line is the one to read twice. role: web here is pillar data, held on the master. role: web on the grains screen was a grain, reported by the machine. Same word, same value, two separate sources, and Salt gives you a different flag for each.
Keep app_tier in mind. You will target on it later.
Then the one assigned to minion2.
cat /srv/pillar/db.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/db.sls output: role: db db_port: 5432 app_tier: backend hint: Print the second pillar data file: cat /srv/pillar/db.sls
The same three keys with the other machine's answers: role: db, a database port, and app_tier: backend.
That is the fixture, complete. Two machines. Two grains each, set on the machine. Three pillar keys each, held on the master. Everything from here is a question asked against one of those, and the flag decides which.
The argument straight after salt is the target, and the code that decides whether a given machine belongs to it is the matcher. You choose the matcher with a flag in front of the target:
| Flag | Matcher | Matches against |
|---|---|---|
| none | glob | the minion id |
-E | regex | the minion id |
-L | explicit list | the minion id |
-G | grain | a fact the machine reports |
-I | pillar | data the master holds |
-C | compound | any mixture of the above |
Three of those you have used already. Glob is what you have typed since the first lesson without knowing it had a name, -G came with grains, and -I came with pillar. The new three are -E, -L and -C, and the last of those is what this lesson is really about.
One paragraph of history explains why there are six. Before configuration management, fleets were selected by hostname, so hostnames were made to carry meaning: web01, db-prod-03, cache-eu-west-2. That works until a machine needs to be two things, or a company is bought whose machines are named after cities. Salt's answer was to stop making the name carry the meaning. Grains and pillar hold the facts, so targeting became a question about facts rather than a pattern matched against a string.
Carry this sentence: the target is a question, and the flag says what kind of question it is. Almost every targeting problem is somebody asking the machine a question only the master can answer, or the other way round.
Glob is the default matcher. No flag, and the pattern is the same wildcard language your shell uses on filenames: * for any run of characters, ? for exactly one, [1-2] for a character in a range.
The star has to be quoted. Unquoted, your shell expands it against the files in your current directory before salt ever sees it, and what Salt receives is a list of filenames.
sudo salt '*' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt '*' test.ping ||| sudo salt "*" test.ping output: minion2: True minion1: True hint: Quote the star so the shell leaves it alone: sudo salt '*' test.ping
Two blocks, two True values, and both machines are up.
Look at the order. minion2 printed first. Nothing is wrong and nothing is sorted: salt prints returns in the order they arrive. Read the set, never the sequence.
test.ping is the right function for a lesson about selection: it costs the machine nothing and it names everything that matched.
Now the narrowest target there is, a pattern with no wildcards in it at all.
sudo salt 'minion1' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' test.ping ||| sudo salt "minion1" test.ping ||| sudo salt minion1 test.ping output: minion1: True hint: The same command with the id in place of the star: sudo salt 'minion1' test.ping
One block. There is no second matcher at work here: minion1 is still a glob, it simply contains no wildcard characters, so the only id it can match is itself.
The quoting is free here. 'minion1', "minion1" and a bare minion1 all arrive at Salt identically, because there is nothing in them for the shell to expand. Quote anyway. The day the target grows a * or a [ is the day an unquoted target starts meaning something else.
You are about to switch from globs to regular expressions. Both take a pattern, both are matched against the minion id, and both use the same punctuation characters for different jobs. Take a position before you type one.
* means zero or more of whatever came immediately before it.>>> Something different, and this is the most expensive confusion in the lesson. A regex * is a repeat operator: it applies to the item before it, so n* means zero or more letter n. The regex that means what a glob * means is .*. If you picked the first answer, you are in company: the two languages share *, ? and [ ] and give all three different meanings. If you picked the third, regular expressions express wildcards perfectly well. They spell it . for one character and .* for any number.
-E switches the matcher from glob to a full regular expression, the same pattern language grep -E and sed -E use. It buys three things a glob cannot express: alternation with (a|b), anchors with ^ and $, and quantifiers like + and {2,4}.
The pattern below uses a character class: [12] means one character, either a 1 or a 2, in that position. Before you press Enter, work out from the key list how many machines it can possibly match.
sudo salt -E 'minion[12]' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -E 'minion[12]' test.ping ||| sudo salt -E "minion[12]" test.ping ||| sudo salt --pcre 'minion[12]' test.ping output: minion1: True minion2: True hint: The regex flag, then a quoted pattern: sudo salt -E 'minion[12]' test.ping
Both machines, and both for the same reason: minion matched literally, then the character class matched the digit on the end.
Put this beside the glob screen. 'minion?' as a glob would have selected the same two machines, so on this fixture the two commands are indistinguishable. The difference shows up on a fleet of web-prod-01 and web-staging-01: a glob can say starts with web, and a regex can say has prod in the middle and ends in two digits.
-L is the matcher with no cleverness in it at all. You hand it a comma separated list of minion ids, and it selects exactly those. No wildcards, no patterns, no facts.
One collision to name before it bites you: this is not the -L you used at the top of the lesson. On salt-key, -L means list every key. On salt, -L means the target is an explicit list. Two different programs, the same letter, unrelated jobs.
That sounds like a step backwards until you meet the case it is for: a set with no pattern in it. Three machines a change goes to first, because somebody decided those three. There is no regex for that, and inventing a grain for a decision that is irrelevant tomorrow is worse than typing the names.
sudo salt -L 'minion1,minion2' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -L 'minion1,minion2' test.ping ||| sudo salt -L "minion1,minion2" test.ping ||| sudo salt -L minion1,minion2 test.ping output: minion1: True minion2: True hint: The list flag, then the ids separated by commas with no spaces: sudo salt -L 'minion1,minion2' test.ping
The same two machines as the regex screen, reached a different way. The regex described a shape and let Salt work out who fitted it. The list named the members and described nothing.
One piece of syntax to be careful with: no spaces after the commas. -L 'minion1, minion2' inside quotes gives Salt an id with a leading space in it, which matches nothing. Unquoted, the space ends the argument and minion2 becomes the module name, which fails loudly.
One line of reminder, because you have done this before: -G targets on a grain, written grain:value, and a grain is a fact the machine reports about itself.
You read both machines' grains earlier, so you can predict this screen exactly. Do that before you press Enter.
sudo salt -G 'role:web' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -G 'role:web' test.ping ||| sudo salt -G "role:web" test.ping ||| sudo salt --grain 'role:web' test.ping output: minion1: True hint: The grain flag, then the grain and its value joined by a colon: sudo salt -G 'role:web' test.ping
One machine, and it is the one whose grains said role: web. The colon in role:web separates the grain name from the value it has to hold, and is part of neither.
The match on the value side is exact and case sensitive, which is the source of most silent grain-targeting failures. role:Web selects nothing here, and it selects nothing quietly.
The role grain was set by the setup. Most grains are set by nobody: the minion works them out when it starts. Ask for one of those instead.
sudo salt -G 'os:Debian' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -G 'os:Debian' test.ping ||| sudo salt -G "os:Debian" test.ping ||| sudo salt --grain 'os:Debian' test.ping output: minion2: True minion1: True hint: The same grain flag with a grain nobody had to set: sudo salt -G 'os:Debian' test.ping
Both machines. Nobody set os on either of them: each minion worked out what it was running at startup and told the master.
On this fixture that selects the whole fleet, which makes it look pointless. On a mixed fleet it is one of the most used lines there is, because os is the fact that decides whether a state uses apt or dnf.
Grains are self-reported. The value the master matches against is whatever the machine said, and a tampered machine can say anything it likes. Never let a grain decide who receives a secret. Grains describe machines, they do not vouch for them.
One line of reminder again: -I has exactly the same key:value shape as -G, and it asks the master's pillar data instead of the machine.
That difference is why both flags exist. Pillar is written on the master and served to a machine that had no say in it, so a pillar match is a claim the master is making rather than one the machine is making about itself.
You read both pillar files a few steps ago. Predict this one too.
sudo salt -I 'role:db' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -I 'role:db' test.ping ||| sudo salt -I "role:db" test.ping ||| sudo salt --pillar 'role:db' test.ping output: minion2: True hint: The pillar flag, same key and value shape as the grain one: sudo salt -I 'role:db' test.ping
One machine, and this is the screen to hold the previous two against.
-G 'role:web' and -I 'role:db' are the same shape, the same punctuation and the same kind of answer. Only the side holding the truth changed. On this fixture grain and pillar agree, which is normal, and is why the distinction is easy to forget until the day they do not.
One warning that costs people an hour. Minions cache their pillar, so editing a pillar file and targeting on the new value immediately can select nothing at all. The fix is the saltutil.refresh_pillar call from the pillar lesson. The tell is a file that is obviously right and a matcher that insists otherwise.
Compound targeting is next. It takes an expression of several terms, and each term names its matcher with a one letter prefix and an @: G@ for grain, I@ for pillar, E@ for regex, L@ for a list. Then there is the term with no prefix at all.
>>> The minion id, as a glob. The prefixes exist to move a term away from the default, so a term with no prefix is the default. If you picked the second answer, prefixes never carry across terms: each one is read on its own, which is what makes an expression readable at a glance. If you picked the third, an unprefixed term is not just legal, it is the most common term in real expressions: a glob to pick a naming family, then a grain or pillar term to narrow it.
-C takes an expression rather than a pattern. Each term is a matcher, the terms are joined by and, or and not, and the whole thing is quoted as one argument.
This is the payoff of the lesson, and the reason is that -C adds no new way of matching. It composes the ones you already have.
The expression below has two terms. G@role:web is a grain, so it asks the machine. I@app_tier:frontend is pillar, so it asks the master. and means a machine has to satisfy both. Work out which machines pass both tests before you press Enter.
sudo salt -C 'G@role:web and I@app_tier:frontend' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -C 'G@role:web and I@app_tier:frontend' test.ping ||| sudo salt -C "G@role:web and I@app_tier:frontend" test.ping ||| sudo salt -C 'I@app_tier:frontend and G@role:web' test.ping output:
Practice Advanced Targeting in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.