Learn › Salt Configuration Management › Salt Fundamentals
Query and set grains on minions. Use grain-based targeting to run commands selectively.
You have never logged in to minion1. Nobody typed its operating system into a spreadsheet. No scanner has ever run against it.
And yet, sitting at the master, you can ask which operating system it runs, which release of that operating system, and which kernel it carries. The answer comes back in under a second.
So where is that answer stored, and who wrote it down? That is this lesson.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output shown was captured live from a real Salt topology, a Debian 12 master with one connected minion named minion1. That is the same shape as the machine your lab boots. Nothing here was written by hand.
A grain is a fact a minion knows about itself. The operating system name, the kernel, the release number, the CPU architecture, the disks, the network addresses.
Think of grains as a profile card each server fills out about itself. When the salt-minion service starts, the minion inspects its own hardware and software and stores what it finds as key-value pairs. A key-value pair is just a label and a value. The key os holds the value Debian. The key kernel holds the value Linux.
The master does not go and measure anything when you ask. It asks, and the minion reads back off a card it already filled in. That is why the reply is nearly instant, even across hundreds of machines at once.
Grains earn their keep in two ways:
1. Getting information about your servers without logging in to any of them. 2. Targeting commands at servers by what they are, not by what they are called.
Salt was written by Thomas S. Hatch in 2011, and the whole project leans on its own pun. The control server is the master. The managed servers are minions. The files that declare how a server should look are states. So the small facts each minion carries got called grains, as in grains of salt.
The naming is silly and it is also useful. Once you know the vocabulary is one extended joke, the parts stop feeling arbitrary and start being easy to remember.
One piece of vocabulary is worth separating now, because it gets mixed up constantly. Grains describe what a machine IS, and the minion collects them itself. Pillar is data the master hands to specific minions, written as plain YAML in files on the master and delivered over Salt's already-encrypted channel. Pillar is not an encrypted vault. Encrypting the values themselves at rest is a separate opt-in feature (the GPG renderer). Grains come up from the machine. Pillar comes down from the master.
Before you run anything, commit to an answer. Getting this wrong first is part of how it sticks.
/etc/os-release each time you ask>>> The minion inspected itself. When the salt-minion service starts, it collects facts about its own hardware and software and keeps them as grains. The master asks for a value off that card, which is why the reply is nearly instant.
If you picked the shell option: that is what a scanning tool does, and it is exactly what Salt avoids. Opening a shell on every machine and parsing files would be slow, would need a login on each box, and would stall the moment one machine was busy.
If you picked the hand-recorded file: that is a static inventory list, and it starts going stale the day after you write it. The point of grains is that the machine describes itself, so the record cannot drift from reality without the machine itself changing.
Before you can read a grain, you need to know which keys exist. grains.ls is the function that lists them.
The shape of every Salt command is the same: salt '<target>' <function>. The target comes first in quotes, then the function you want run. Here the target is minion1 and the function is grains.ls.
There are dozens of keys, so this lesson trims the list with head -15 to keep the screen readable. On the real machine you would leave the trim off and read the whole thing.
Before you run it, decide what you expect to see: names, or names with their values? Ask minion1 for its list of keys:
sudo salt 'minion1' grains.ls | head -15
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.ls | head -15 ||| sudo salt "minion1" grains.ls | head -15 ||| sudo salt 'minion1' grains.ls | head -n 15 output: minion1:
hint: Target first in quotes, then the function: sudo salt 'minion1' grains.ls | head -15
Names only. Every line is a key, and not one of them shows a value.
Read the shape before the words. The first line, minion1:, names the machine that answered. Everything below it is indented, because Salt groups each reply under the minion that produced it. One command can reach many machines, so the indent is how you tell whose answer you are reading.
The keys arrive alphabetically, which is why the list opens in the b's and c's: biosreleasedate, biosvendor, boardname, cpu_model, cpuarch. Further down sit disks, dns, domain, efi. This is a table of contents for the machine. Now you need a way to open one of the entries.
grains.get takes one key name and returns its value. The shape is salt '<target>' grains.get <key>.
os is the operating system name. It is the grain engineers reach for more than any other, because almost every real decision starts with which family of Linux you are talking to.
Before you run it, hold the contrast in your head: grains.ls gave you a list of names. This one asks for a single key. Read the os grain off minion1:
sudo salt 'minion1' grains.get os
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.get os ||| sudo salt "minion1" grains.get os output: minion1: Debian hint: The function is grains.get, then the key you want: sudo salt 'minion1' grains.get os
Two lines. minion1: is who answered, and Debian is the answer.
That is the delta worth staring at. grains.ls printed a wall of key names with no values. grains.get os printed one value with no key name, because you already said which key you wanted. Names come from grains.ls. Values come from grains.get.
You now know this machine is a Debian box, and you learned it without a login, a password, or an SSH session.
Asking one key at a time gets tedious. grains.item takes several key names and returns all of them in one reply.
Three keys are worth knowing immediately:
os is the operating system nameosrelease is the version number of that operating systemkernel is the kernel name, which reads the same on every Linux boxBefore you run it, predict the count: three keys in, so how many values back, and from how many machines? Ask minion1 to describe itself in one call:
sudo salt 'minion1' grains.item os osrelease kernel
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.item os osrelease kernel ||| sudo salt "minion1" grains.item os osrelease kernel output: minion1: ---------- kernel: Linux os: Debian osrelease: 12 hint: The function is grains.item and the keys follow it, separated by spaces.
Three keys in, three values back, all from one machine: kernel Linux, os Debian, osrelease 12. This is a Debian 12 machine running a Linux kernel.
Two details in the shape are worth naming. The row of dashes is Salt's separator, printed whenever the reply is a set of key-value pairs rather than a single bare value. And the keys came back alphabetically, not in the order you typed them. Salt sorts the keys inside a minion's reply, so never read meaning into the sequence.
Notice the deeper indent too. The minion name sits at the left, the key names are indented under it, and each value is indented under its key. Three levels, so you can always tell machine from key from value.
Salt collects os and kernel for you. It cannot possibly know that a box is your London staging database, so you are allowed to write your own grains and tag machines however you like.
>>> It survives. When you set a custom grain, Salt writes it into /etc/salt/grains on the minion itself. The next time salt-minion starts, it collects the built-in facts again and then reads that file back in, so your tag is still there.
If you picked lost: that is true of the built-in grains and it is exactly why the custom ones are written to disk. A tag you had to reapply after every reboot would be useless for targeting.
If you picked only with a master restart: the file lives on the minion, not on the master, so the master is not involved. That is deliberate. It means a minion carries its own identity even if it is rebuilt or moved to a different master.
grains.setval writes a custom grain. The shape is salt '<target>' grains.setval <key> <value>.
Pick a key that describes a role rather than a fact, because the built-in grains already cover the facts. role is the conventional choice, and webserver is a conventional value for it.
Before you run it, decide what a write ought to print back. Tag minion1 with a role:
sudo salt 'minion1' grains.setval role webserver
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.setval role webserver ||| sudo salt "minion1" grains.setval role webserver output: minion1: ---------- role: webserver hint: The classic setter is grains.setval, then the key, then the value.
Salt echoed the grain back to you: role is now webserver. It is the same three-level shape you read from grains.item, dashes and all, because a write returns the same kind of key-value reply that a read does.
Salt ships two setters and both work here. grains.setval is the classic one and it is the form captured above. grains.set does the same job for a simple key and value, and also handles nested values. The lab accepts either, so use whichever you remember.
Grain names and values are both case-sensitive. Role and role are two different grains, and a value of Webserver will not match a filter looking for webserver. Stick to lowercase for everything you invent, and you will never have to think about it again.
You wrote a grain that Salt did not collect. The question worth answering is whether it now behaves like an ordinary one.
You already know the reader: grains.get plus a key name. The only thing changing is which key you ask for. Read the role grain back off minion1:
sudo salt 'minion1' grains.get role
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.get role ||| sudo salt "minion1" grains.get role output: minion1: webserver hint: Same reader as before, different key: sudo salt 'minion1' grains.get role
Put this reply next to the os one you read earlier. minion1: then Debian. minion1: then webserver. Identical shape, identical command form.
That is the point. Salt draws no line between a grain it collected and a grain you invented. Once role exists, every tool that can read a grain can read yours, including the one that picks which machines a command reaches.
So far you have targeted by name. 'minion1' works fine until it does not. With thirty web servers you would type thirty names, and the day someone builds the thirty-first, your command quietly misses it.
The -G flag targets by grain instead. The shape is salt -G 'key:value' <function>. Read it as a sentence: every minion whose key grain equals value, run this.
>>> It filters. -G tells Salt to read the text as a grain expression instead of a name, so it checks the role grain on every minion it knows and keeps the ones that say webserver. The set of machines is never typed by you. It is worked out from what the machines said about themselves.
If you picked the literal ID: that is what happens WITHOUT the -G flag. Salt would hunt for a minion actually called role:webserver, find none, and match nothing. The flag exists precisely to change that reading.
If you picked a list on the master: no such list is kept. That is the difference between Salt and an inventory file. The master asks the fleet what it is rather than trusting a record someone maintained by hand.
Now run it. test.ping is the safest function Salt has. It asks each matched minion to confirm it is alive and changes nothing on the way. That makes it the standard way to check that a target expression selects what you meant, before you point anything real at it.
The separator inside the quotes is a colon, not an equals sign. -G 'role:webserver' works. -G 'role=webserver' matches nothing.
Ping every machine tagged as a webserver:
sudo salt -G 'role:webserver' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -G 'role:webserver' test.ping ||| sudo salt -G "role:webserver" test.ping output: minion1: True hint: The grain-targeting flag is -G, and the filter goes in quotes with a colon between key and value.
minion1 answered True, and you never typed its name.
Trace how it got there. You wrote role: webserver onto the machine. Salt stored it on the minion. Then -G 'role:webserver' asked the fleet who matched, and this machine put its hand up. The command text contains no hostnames at all.
That is why the same command text would work unchanged against two hundred machines. You describe a property, and the set of machines that fit is whatever it is that day.
Stop for a second, because you crossed the line that separates running commands from managing infrastructure.
Before this lesson, reaching a machine meant knowing its name. Now you describe a property and Salt works out the list. Add a machine tomorrow, tag it role: webserver, and every command you already wrote picks it up with no edit.
You stopped addressing servers and started addressing categories.
Scaffolding off. No command is shown from here on.
A colleague asks you to confirm that every Debian machine on this master is alive. Two constraints. You may not type any machine names, because the check has to keep working when machines are added later. And you may not lean on the role grain you invented, because the machines added later will not carry it.
Use a grain Salt collected by itself, and the safest function Salt has.
prompt: saltops@saltmaster:~$ answer: sudo salt -G 'os:Debian' test.ping ||| sudo salt -G "os:Debian" test.ping output: minion1: True hint: Same targeting flag as before, but filter on the built-in grain you read in the second walkthrough. Watch the capital D.
-G 'os:Debian' filters on a grain the minion collected about itself, so it keeps working on machines nobody has tagged yet.
The capital D matters. You read that value earlier and it came back as Debian, so -G 'os:debian' would not match it. When a grain filter comes back with no minions, suspect the case and the colon before you suspect the fleet.
Second challenge, still no command shown.
You are handing this machine over to someone else, and they want a record of what grains it carries. Save the full list of grain keys from minion1 into a file called grains-report.txt in your home directory.
You know the function that lists key names. This time send its output into a file instead of onto the screen, and do not trim it.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.ls > ~/grains-report.txt ||| sudo salt "minion1" grains.ls > ~/grains-report.txt ||| sudo salt 'minion1' grains.ls > /home/saltops/grains-report.txt output: hint: The lister is grains.ls, and the redirect that sends output to a file is the greater-than sign, then the path ~/grains-report.txt
Nothing printed, and that is the success signal. The > redirect sent every byte into the file instead of onto your screen, so a silent return means the write happened.
Run cat ~/grains-report.txt on the real machine and you will see the same key list you met in the first walkthrough, this time complete instead of trimmed to fifteen lines. That file is the deliverable the lab is looking for, and it has to be in your home directory and not empty.
You earned this table. Every function in it you ran yourself in this lesson, against real output.
| Command | What it does |
|---|---|
sudo salt 'minion1' grains.ls | List every grain key the minion has, names only. It is long, so trim it with head when you just want a look |
sudo salt 'minion1' grains.get os | Read one grain value |
sudo salt 'minion1' grains.item os osrelease kernel | Read several grains in one call |
sudo salt 'minion1' grains.setval role webserver | Write a custom grain, stored in /etc/salt/grains on the minion |
sudo salt 'minion1' grains.get role | Read a custom grain back, exactly like a built-in one |
sudo salt -G 'role:webserver' test.ping | Target by a grain you set yourself |
sudo salt -G 'os:Debian' test.ping | Target by a grain Salt collected itself |
sudo salt 'minion1' grains.ls > ~/grains-report.txt | Save the key list to a file |
Three things to burn in:
-G 'key:value' uses a colon, and both the key and the value are case-sensitive.Custom grains are where targeting becomes genuinely powerful. Salt can tell you a box
Practice Grains in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.