Learn › Salt Configuration Management › Pillar and Data Management
Store sensitive data in pillar and verify it stays hidden from grains and public interfaces.
Under /srv/pillar on this machine there is a file with a database password in it, written in plain text. That part is correct. Pillar is exactly where a password belongs.
What is not correct is who can read it. Salt did not make that mistake. Whoever set this master up did, and the whole mistake fits in one digit of one number.
Start at the door, not at the file. A directory is a door: its permissions decide who is even allowed to walk in and look at what is inside.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output printed below was captured from this lab's own master with one real minion attached, which is exactly the machine your lab boots. Some commands here print nothing at all when they work, and those boxes are blank on purpose. The lab user is saltops, the master is named saltmaster, and there is exactly one minion, named minion1.
ls -l lists what is inside a directory. That is not what you want here. You want the permissions of the directory itself, so add -d, which means show me this thing, not its contents.
Before you run it, commit to one thing: how many of the three permission groups do you expect to be open?
ls -ld /srv/pillar
prompt: saltops@saltmaster:~$ answer: ls -ld /srv/pillar ||| ls -dl /srv/pillar output: drwxr-xr-x 2 saltops salt 4096 Jul 31 18:38 /srv/pillar hint: Long listing, plus the flag that means the directory itself instead of its contents: ls -ld /srv/pillar
Read drwxr-xr-x left to right. The leading d says directory. The nine characters after it are three groups of three: owner, group, everyone else.
rwx for the owner, saltops, which is you. Read, write, enter.r-x for the group, salt. Read and enter, no writing.r-x for everyone else. Read and enter.That last group is the hole. Every account on this master, including one you never created, can walk into /srv/pillar and read what is in there.
Now hold on to the group name. It is salt, not saltops, and that single word decides which fix works and which fix quietly breaks your fleet.
Pillar is data the master keeps for minions: values that belong to a specific machine and have no business sitting in a state file.
The split is worth saying plainly. A state file is instructions. It says install this package, write this config, start this service. Instructions get shared, reviewed and committed to version control. Pillar is the private data those instructions look up while they run: passwords, tokens, licence keys, the address of the database this one machine is allowed to talk to.
The master builds pillar separately for each minion and hands each minion only its own. Nothing about that requires the minion to be trusted with the rest.
Salt is full of one metaphor and the names are not decoration. Grains are the many small facts a minion works out about itself: its operating system, its memory, its network cards. Ground up, plentiful, coming from below.
A pillar is the opposite shape. One structure, deliberately placed, holding weight from above. Grains flow up from the minion to the master. Pillar flows down from the master to one chosen minion.
That direction is the whole security argument. A minion decides its own grains, so a minion can lie about them. The master decides pillar, so a minion gets exactly what the master says it gets and no more. When you are deciding where a secret goes, ask which end of the wire owns the value. The answer is always the master.
drwxr-xr-x is readable but hard to compare. The same permissions written as numbers are easier to check and easier to grade, and the tool that prints them is stat.
stat -c takes a format string and prints only the fields you asked for:
%a is the mode in octal, the three digit form.%U is the owner's name.%G is the group's name.Quote the format string so your shell leaves the percent signs alone. Ask for all three:
stat -c '%a %U:%G' /srv/pillar
prompt: saltops@saltmaster:~$ answer: stat -c '%a %U:%G' /srv/pillar ||| stat -c "%a %U:%G" /srv/pillar ||| sudo stat -c '%a %U:%G' /srv/pillar output: 755 saltops:salt hint: The tool is stat, the flag that takes a format string is -c, and the three fields are %a %U:%G
755 saltops:salt. Same information as the letters, three characters instead of nine.
Each digit is one of the groups you just read. 7 is the owner: read, write, enter. 5 is the group salt: read and enter. The final 5 is everyone else, and everyone else should not be here at all.
Read that last digit first for the rest of your career. On a directory holding secrets it should be 0 and nothing else. Your lab grades this exact number, and it looks at the last digit.
Never lock something down until you know what it looks like working. Otherwise you cannot tell a fix from a break.
pillar.items asks a minion to report the pillar the master built for it. It is a question aimed at the minion, not at the file. That tests the whole path: the file on disk, the master reading it, the master rendering it for one machine, and the minion receiving it.
Before you run it, commit to a number. How many keys come back?
sudo salt 'minion1' pillar.items
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' pillar.items ||| sudo salt "minion1" pillar.items ||| sudo salt minion1 pillar.items ||| sudo salt '*' pillar.items output: minion1: ---------- app_name: ******** app_user: ****** db_password: ******** hint: Target the minion in quotes, then the pillar module and the function that dumps everything: sudo salt 'minion1' pillar.items
Three keys: app_name, app_user, db_password. The row of dashes under the minion ID is how Salt opens a mapping on screen, and everything indented under it belongs to that mapping.
Now the part people stare at. Every value came back as a row of asterisks. Salt is not printing these values back at you on the command line, and every value in this pillar is text, so every one of them is masked.
So read the key names, not the values. Key names arriving is the signal that pillar reached this minion. An empty mapping would mean it did not. Running pillar.items is one of the five things your lab grades, so this is a form worth knowing by heart.
Asterisks on screen feel like protection. Take a position on whether they are.
>>> Plain text. This is the most misunderstood thing about pillar, so be blunt about it: pillar is not encrypted at rest. /srv/pillar/app.sls is an ordinary YAML file. The password in it is legible to anything that can open it. If you picked the asterisks, notice what you actually saw: a choice the command line made about what to print. Masking output is not storage. If you picked the encrypted blob, that does exist in Salt as an opt-in GPG renderer. It is something you deliberately turn on and set up keys for, and nobody has done that here. Until then, the only thing between that file and a curious account on this box is the permissions you are about to fix.
Here is the whole of /srv/pillar/app.sls on your lab box. Three keys, one value each, no ceremony:
app_name: mywebapp
app_user: apprunner
db_password: s3cretP@ss2026That is the file the master read to answer the command you just ran. Those are the three keys that came back. The value that came back as asterisks is right there in the third line.
So what does protect pillar? Four things, and only one of them is yours today:
1. The file never leaves the master. Minions receive rendered values, never the file. 2. The master renders pillar per minion, so a machine only ever gets its own. 3. The master and minion already talk over an encrypted channel, so the value is not readable on the wire. 4. File permissions on the master decide who can read the plain text.
Numbers 1 to 3 are Salt's job and Salt is already doing them. Number 4 is your job, and on this box it is currently wrong.
The obvious fix is to slam the door shut: 0700, owner only, nobody else at all. Before you type that, one fact about this machine.
This master does not run as root. Its config sets user: salt, so the process that reads /srv/pillar authenticates as the user salt. Look back at what you read a minute ago: the directory is owned by saltops and its group is salt. The master is not the owner. The master is the group.
>>> An empty pillar, in silence. 0700 means owner only, and the owner is saltops. The group salt loses read and enter, so the master can no longer open the directory it is configured to read from. If you picked the error, that is the reasonable expectation and it is what makes this so expensive in real life: there is no error to grep for. If you picked the answer about root, that is true on plenty of masters and false on this one, which is the actual lesson. Read user: in the master config before you choose a mode, because the right number depends entirely on who the master logs in as.
This is pillar.items on this fixture with the directory left at 0700, captured so you never have to discover it during an outage:
minion1:
----------The minion answered. The mapping opened. There is nothing in it.
No error line. No warning. Nothing failed loudly enough to page anyone. Meanwhile every state that reads db_password now renders with a missing value or dies on an undefined key, and the reason is three directories away from wherever you will be looking.
Both 0700 and 0750 end in a zero, so a permissions check that only reads the last digit is satisfied by either. The check is happy and your fleet is broken. This is the difference between passing a check and understanding a system, and it is why the number to reach for is 0750.
0750 reads digit by digit:
7 for the owner saltops: read, write, enter. You still edit these files.5 for the group salt: read and enter. The master still reads pillar.0 for everyone else: nothing. No entering, no listing, no reading.You own this directory, so no sudo is needed. chmod prints nothing at all when it succeeds, so expect a bare prompt back.
chmod 0750 /srv/pillar
prompt: saltops@saltmaster:~$ answer: chmod 0750 /srv/pillar ||| sudo chmod 0750 /srv/pillar ||| chmod 750 /srv/pillar ||| sudo chmod 750 /srv/pillar output: hint: The mode you want is owner 7, group 5, everyone else 0: chmod 0750 /srv/pillar
Silence, and a prompt back. That is chmod working. It only speaks up when it fails, usually with Operation not permitted if you do not own the thing you are trying to change.
Silence is also why the next step exists. A command that says nothing has told you nothing. Go and read the result yourself.
You ran this exact command a few minutes ago. Nothing about it changes. Everything about the answer does.
Before you run it, decide which of the three digits should be different now.
stat -c '%a %U:%G' /srv/pillar
prompt: saltops@saltmaster:~$ answer: stat -c '%a %U:%G' /srv/pillar ||| stat -c "%a %U:%G" /srv/pillar ||| sudo stat -c '%a %U:%G' /srv/pillar output: 750 saltops:salt hint: Exactly the command you ran before the chmod, unchanged: stat -c '%a %U:%G' /srv/pillar
750 saltops:salt. One digit moved and it was the right one.
The 7 did not move, so you can still edit pillar files. The 5 did not move, so the group salt still reads and enters, and the master is in that group. The final digit went from 5 to 0, so an account on this box that is neither saltops nor in the group salt now gets refused at the door.
Owner and group are unchanged, which matters as much as the number. chmod never touches ownership. That is chown, a different tool for a different job.
The last digit being 0 is the first of the five things your lab grades.
The mode reads correctly. That is not the same claim as the master still working, and you have just seen how quiet this particular failure is.
So ask the machine that would notice. Same command as before, same target, same everything.
Commit before you run it: three keys again, or an empty mapping?
sudo salt 'minion1' pillar.items
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' pillar.items ||| sudo salt "minion1" pillar.items ||| sudo salt minion1 pillar.items ||| sudo salt '*' pillar.items output: minion1: ---------- app_name: ******** app_user: ****** db_password: ******** hint: The same question you asked before the lockdown, unchanged: sudo salt 'minion1' pillar.items
The same three keys, in the same order, laid out the same way. Nothing about this output changed, and that is the whole point of running it.
Hold this screen next to the empty mapping you were shown for 0700. Identical command. One digit of difference in a directory mode. On one side a working fleet, on the other side a silent outage with no error to search for.
That is the habit to keep: change a permission, then ask the consumer of that permission whether it still works. The mode is your intention. The pillar.items answer is the outcome.
Count what changed. You arrived on a master that was handing a database password to every account on the box, and you found it by reading a directory rather than guessing. Then you picked a mode that fits who the master actually is, and proved the fix did not break the thing it was protecting.
That is the hard half of secret handling in Salt, and it is done. What is left is making sure the same secret does not stroll out through a different door.
A locked directory is worth nothing if the password ends up copied into a state file, because state files are the part of Salt that gets shared, reviewed and committed.
Your box already ships the correct pattern at /srv/salt/dbconfig.sls. Read it rather than typing it, because the shape is what matters:
db-config-dir:
file.directory:
- name: /etc/myapp
- makedirs: True
db-config-file:
file.managed:
- name: /etc/myapp/database.conf
- contents: |
[database]
user={{ pillar['app_user'] }}
password={{ pillar['db_password'] }}
database=appdb
- template: jinja
- mode: 0640
- require:
- file: db-config-dirTwo states, not one. db-config-dir makes sure /etc/myapp exists. db-config-file writes the config into it, and the - require: stanza at the bottom is what orders them: the file is not written until the directory state has succeeded.
Four lines inside the second state carry the whole idea:
{{ pillar['db_password'] }} is a placeholder, not a value. It is filled in when the state runs, from the pillar the master built for that one minion.template: jinja is what makes the placeholder mean anything. Without it those braces are written into the file literally, password and all missing.mode: 0640 locks the file this state creates on the minion. A config file holding a password has no business being world readable at the other end either.This state and the pillar file both ship with your lab already written, and the grader counts both. That is deliberate. Your job in this lesson is to be able to read them, spot the pattern, and rebuild it from memory on a machine that has neither.
There is always someone who suggests putting the value in grains so the minion just has it locally. Take a position before you read on.
>>> No, and the reason is direction. Grains belong to the minion and travel upward to the master. The machine you are trying to protect is the machine that owns the value. Anything on that box that can run Salt can print them back, and any state or template on that box can read them. Worse, a minion can set its own grains, so a compromised machine can claim to be anything it likes. If you picked the answer about the network, grains do cross it, in the direction you did not want. If you picked the answer about the minion config, where a grain is defined changes nothing about who can read it. Pillar exists precisely because grains are the wrong end of the wire for anything private.
Scaffolding off. No command is printed from here on.
You have argued that the password is not in grains. Now go and get evidence, because on a real incident call an argument is worth nothing and one line of output is worth everything.
Ask minion1 for every fact it knows about itself, then filter that answer down to anything that mentions a password, ignoring upper and lower case. One command, one pipe.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.items | grep -i password ||| sudo salt "minion1" grains.items | grep -i password ||| sudo salt minion1 grains.items | grep -i password ||| sudo salt '*' grains.items | grep -i password output: hint: The facts module is grains and the function that dumps all of them is items. Pipe that into grep -i password.
Nothing came back, and nothing is the answer you wanted. grep read every line of that minion's facts and found no mention of a password in any of them.
Get comfortable with an empty result as a real finding. It is not the command failing. It is the command reporting that the thing you were afraid of is not there. Compare it with what pillar.items returned for the same secret: one channel has it, the other does not, and that is the separation you are being paid to keep.
Running grains.items is the fourth of the five th
Practice Secure Pillar Data in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.