Learn › Salt Configuration Management › Pillar and Data Management
Assign different pillar data to different minions and use -I targeting.
Behind this terminal there are two machines again. minion1 is meant to be a web server. minion2 is meant to be a database server. Both are connected, both are trusted, and both will answer anything this master asks.
On the master there is a folder called /srv/pillar. Two files are already sitting in it. One of them describes a web server and one of them describes a database server, and right now nothing anywhere says which machine gets which.
By the end of this lesson those two machines will hold different configuration, produced by the same state file, without you naming either of them on the command line. One of the screens along the way will show you nothing at all where you expected a value. That screen is not a fault, and working out what it means is half of what this lesson is for.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do, so what you type here is what you will type there. Your own timestamps and byte counts will be your own. The lab user is saltops, the master is named saltmaster, and the two minions are minion1 and minion2. Your progress in the lab is tracked automatically, so type commands naturally.
Salt reads pillar files from one directory on the master, called the pillar root. On this box it is /srv/pillar, and it was set up before you logged in. A pillar file is a plain text file ending in .sls that holds data the master hands down to a machine.
Start by looking at the directory rather than guessing what is inside it. ls -l lists a directory one entry per line, with permissions, owner, size and time on each line.
Before you run it, commit to a number: how many entries should come back?
ls -l /srv/pillar
prompt: saltops@saltmaster:~$ answer: ls -l /srv/pillar ||| ls -l /srv/pillar/ output: total 8 -rw-r--r-- 1 saltops saltops 74 Jul 31 18:18 db.sls -rw-r--r-- 1 saltops saltops 63 Jul 31 18:18 web.sls hint: The long listing of a directory is ls -l, then the path: ls -l /srv/pillar
Two files, nothing else. db.sls and web.sls. Both lines start with -, which means an ordinary file rather than a directory, and both are tiny: 74 and 63 bytes.
Read the owner column next, because it decides how you work for the rest of this lesson. Both files say saltops saltops, which is your own account. The setup handed this whole tree to you, so every file you write here is a plain redirect with no sudo in front of it.
Now notice what is not here. There is no top.sls. In the pillar tree, exactly as in the state tree, top.sls is the file that says who gets what. Without it, these two files are data nobody is assigned, and every minion on this master currently receives nothing at all.
Two per-minion pillar files, one per machine, is the first thing your lab grades.
Pillar is data the master hands down to a minion. You write it on the master, and each machine receives only the pillar assigned to it.
Pillar targeting is the two halves of that idea used together:
The first half is what lets one state file configure a whole fleet. The file asks for app_port and gets 80 on a web server and 5432 on a database server. The second half is what lets you say "every web server" out loud in a command, without remembering which machine IDs happen to be web servers this month.
That second half has a flag of its own, -I, and it is the flag this lesson is named after.
Salt started as a remote execution bus, the salt command you have been typing since the first lesson. Grains came early, and grains answer the question the machine can answer for itself: what operating system am I, how much memory do I have, what is my CPU.
That left the opposite question with nowhere to live. What a machine is for is not something the machine can detect. No amount of inspecting hardware tells a server that it is the production database. Somebody has to decide it, and the natural place for that decision is the master, where the whole fleet is visible at once.
So pillar was added as the master's side of the conversation, and the name is the point. It is the data the rest of a machine's configuration stands on.
The split is worth carrying around: grains are discovered, pillar is assigned. A grain is a fact the minion found out about itself. A pillar value is a decision you made and wrote down. When you cannot remember which one holds something, ask yourself whether a brand new machine would know it without being told.
The two files exist, but neither one carries a port yet, and a port is the value that is going to make the difference visible. You are going to write both files now.
A heredoc is the shell's way of typing several lines into a file in one go: everything between the << 'EOF' marker and the closing EOF becomes the content of the file. /srv/pillar belongs to your account, so there is no sudo here.
Write the web server data first:
cat > /srv/pillar/web.sls << 'EOF'
role: web
environment: production
app_port: 80
packages:
- nginx
- curl
EOFThen the database data:
cat > /srv/pillar/db.sls << 'EOF'
role: db
environment: production
app_port: 5432
packages:
- sqlite3
- libsqlite3-dev
EOFRead the two files against each other, because the shape is the whole technique. Both use the same four key names: role, environment, app_port and packages. Two of those keys hold different values in each file. One of them, environment, holds the same value in both, which is perfectly normal: not every key has to differ.
That is what makes a state file portable. A state that asks for app_port never has to know which machine it landed on. It asks the same question everywhere and gets the answer that machine was given.
packages: is a YAML list. Each item goes on its own line, indented two spaces under the key, and starts with a dash followed by a space. - nginx is a list item. -nginx is not an error, which is worse: YAML reads it as ordinary text, so packages quietly becomes a single string instead of a list and nothing complains until something tries to use it. Keep every item at the same indentation, and use spaces, never tabs.
You are about to write a file that sends web.sls to one machine and db.sls to the other. Before you make anything happen, take a position on what a single minion ends up holding.
>>> Only the db data. The master compiles pillar per machine: it works out which entries in the top file match that minion, renders just those files, and sends the result to that one minion. minion2 never receives minion1's values, so it cannot read them, print them, or leak them. If you picked the first answer, that is the assumption worth unlearning today, because pillar isolation is exactly why pillar is where credentials belong. If you picked the third, you are describing a real Salt behaviour aimed at the wrong thing. When one minion is assigned two files that both set the same key, the later assignment does win. That merge happens inside one machine's pillar. It never reaches across machines.
The pillar tree gets its own top file at /srv/pillar/top.sls, separate from the state tree's top.sls in /srv/salt. Same file name, same grammar, different job: this one assigns data to machines, not states.
cat > /srv/pillar/top.sls << 'EOF'
base:
'minion1':
- web
'minion2':
- db
EOFFive lines, three jobs:
base: is the environment name, at column 0 with no indentation.'minion1': is a target pattern, indented 2 spaces. Here it is a plain minion ID. The quotes keep YAML from reading the value as something other than text.- web is the pillar file to hand over, indented 4 spaces. It is web.sls without the extension, and the leading - makes it a list item, which is how one machine can be given several pillar files.A heredoc prints nothing when it works, so read the file back and check your own typing before you go anywhere near the minions.
cat /srv/pillar/top.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/top.sls output: base: 'minion1':
'minion2':
hint: Print your new file with cat and its full path: cat /srv/pillar/top.sls
There it is on disk. One environment, two targets, one pillar file each, and the two targets point at different files. That difference is the second thing your lab grades.
Read it as a staircase. base: sits flush left. Both machine names sit one step in at 2 spaces. Both assignments sit two steps in at 4 spaces. The depth is the meaning: - web belongs to 'minion1': because it is indented under it, and for no other reason.
The targets on the left accept the same patterns you already use on the command line. 'minion1' is one exact machine. 'web*' would be every machine whose ID starts with web. '*' would be every machine, which is how you hand a value to the whole fleet.
Misaligned YAML in this file does not produce an error. Salt reads the structure you actually typed, which is a different structure from the one you meant, and the affected machines quietly receive nothing. When a minion has no pillar and you are sure the data exists, read the indentation of this file before you read anything else.
Changing a file on the master does not reach out and update anybody. Each minion holds its own copy of its pillar, and it keeps using that copy until it is told to fetch a fresh one. saltutil.refresh_pillar is that instruction.
This is the step people skip. They edit a pillar file, immediately query it, see the old value, and start debugging a file that was already correct. Refresh first, every time.
Send it to every machine at once. Watch what the answer is: each minion is confirming it accepted the instruction, not reporting what it received.
sudo salt '*' saltutil.refresh_pillar
prompt: saltops@saltmaster:~$ answer: sudo salt '*' saltutil.refresh_pillar ||| sudo salt "*" saltutil.refresh_pillar output: minion2: True minion1: True hint: Quote the target so the shell leaves the star alone, then the module and function: sudo salt '*' saltutil.refresh_pillar
Two machines, two True values. Both minions took the instruction and refreshed.
True here means "I did that", nothing more. It is not telling you what data arrived, whether the data was what you intended, or even that any data arrived at all. A machine with an empty pillar returns exactly the same True. So refresh proves delivery of the instruction, and you still have to look at the data yourself.
One detail to file away: minion2 printed first. Salt prints each machine's block as that answer arrives, so output order is arrival order and nothing else. Never read meaning into which machine appears at the top.
pillar.get reads one key out of a minion's pillar and prints it. Point it at every machine at once and you get a column of answers, one per minion, which is the fastest way to compare a value across a fleet.
Start with the key that names each machine's job. Before you press Enter, decide what you expect: two lines with two different words under them, one saying web and one saying db.
sudo salt '*' pillar.get role
prompt: saltops@saltmaster:~$ answer: sudo salt '*' pillar.get role ||| sudo salt "*" pillar.get role output: minion2: ******** minion1: ******** hint: Same three-part shape as always, and the function that reads one pillar key: sudo salt '*' pillar.get role
Not what you were promised anywhere. Both machines answered, both answers are a run of asterisks, and the two are identical. There is no web and no db on this screen.
Nothing is broken. Both minions have their data, and you are about to prove it. What you are looking at is Salt's output layer refusing to print this value back to your terminal. Pillar is where secrets live, and a tool that echoes pillar values onto the screen is a tool that writes secrets into your scrollback, your terminal recording and your support ticket.
Look closely at the two runs of asterisks: they are the same length. web is three characters and db is two. The mask is a fixed width, so it does not even leak how long the value was.
Take the working rule away from this screen: a masked value cannot be compared, so compare a different key. That is exactly what the next command does.
Asterisks on the screen are not encryption. Your pillar files sit in /srv/pillar on the master as plain text YAML, which is why the permissions on that directory matter and why nothing in that folder belongs in a public repository. What pillar does give you is real, and it is worth naming precisely. The master renders each machine's data separately. It sends that data only to that machine, over a channel that is already encrypted end to end. If you also need the contents encrypted where they sit on disk, that is a separate opt-in feature called the GPG renderer. It stores ciphertext in the .sls file and decrypts it on the master as the pillar is rendered.
Same command, same target, one word changed. app_port is a port number rather than a role name, and this is the screen the whole lesson has been walking toward.
Before you run it, commit to the two numbers you wrote into those files a few steps ago, and to which machine each one belongs.
sudo salt '*' pillar.get app_port
prompt: saltops@saltmaster:~$ answer: sudo salt '*' pillar.get app_port ||| sudo salt "*" pillar.get app_port output: minion1: 80 minion2: 5432 hint: The command you just ran, with the other key name on the end: sudo salt '*' pillar.get app_port
There is the difference, in one screen. minion1 answers 80. minion2 answers 5432. One command, one key name, two machines, two different values.
Read what that proves. minion1 does not know 5432 exists. minion2 does not know 80 exists. Each machine was handed its own rendered pillar and nothing else, which is the isolation you took a position on earlier, now visible on screen. Pillar values differing per minion is the third thing your lab grades.
Now put this screen beside the previous one, because together they teach you how to verify pillar for the rest of your career. Identical command shape, identical target, identical machines. The only thing that changed is which key you asked for, and one key printed while the other masked. When you need evidence that per-minion data is landing correctly, choose a key whose value is not a secret, and read that.
One more detail: minion1 printed first this time, and minion2 printed first on the last two screens. Same two machines, same master, order flipped. Arrival order, every time.
You now know something uncomfortable. The key that best describes what each machine is for is the one key that will not show you its value. The obvious next question is whether that value is still usable.
>>> Yes. Masking is a property of what gets printed, not of what gets stored or compared. The real value is on the master in plain text, and it is the real value that Salt uses when it works out which machines a target matches. If you picked the first answer, notice how limiting it would be: the keys most worth targeting on are exactly the descriptive ones, and half of them would be off limits. If you picked the third, that is worth naming so you never try it: the asterisks are not the value and not a placeholder for it. They are ten characters of nothing, printed after the matching has already happened.
Every Salt command you have typed so far picks machines by name: an exact ID, or a glob like 'minion*'. -I picks them by pillar data instead.
The flag is a capital i. There is no helpful mnemonic, so learn it as a pair with the one you already know: -G targets grains, -I targets pillar.
The value after it is written key:value, with a colon and no space. That trips people, because in the YAML file you write role: web with a space and here you write role:web without one. The file is YAML. The target is not.
So the command asks: every machine whose pillar says its role is web, answer me.
sudo salt -I 'role:web' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt -I 'role:web' test.ping ||| sudo salt -I "role:web" test.ping ||| sudo salt -I role:web test.ping ||| sudo salt -I 'app_port:80' test.ping output: minion1: True hint: The pillar targeting flag is a capital I, and the value is key:value with no space: sudo salt -I 'role:web' test.ping
One machine answered. minion1 is the only machine on this master whose pillar says role: web, so it is the only machine the command was sent to. minion2 was not slow and did not fail. It was never asked.
That is the shift worth sitting with. You did not name a machine. You described one, and Salt worked out the roster for you. Add a second web server tomorrow, give it the web pillar, and this exact command reaches both without being edited. Take one out of service and the command stops reaching it. The command becomes a description of intent instead of a list of hostnames, and the list maintains itself.
Using -I at all is the fourth thing your lab grades.
-G and -I look alike and answer different questions. -G 'os:Debian' selects on a fact the machine discovered about itself, so it is the flag for anything hardware or platform shaped. -I 'role:web' selects on a decision you made and wrote down, so it is the flag for anything role or configuration shaped. "All my Debian boxes" is a grain question. "All my web servers" is a pillar question.
Targeting is half the payoff. The other half is that a single state file can now configure both machines differently, and the master already has one waiting for you. This is what is sitting in /srv/salt/app.sls on this box:
app-setup:
file.managed:
- name: /etc/app/role.conf
- contents: |
rPractice Pillar Targeting in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.