Learn › Salt Configuration Management › Pillar and Data Management
Create /srv/pillar, write pillar data, create a pillar top.sls, and query values.
Every fleet runs on values. A port number. A site name. A database password. Those values have to live somewhere, and the obvious place is wrong.
If you type them into your state files, then a server that needs a different port needs its own copy of the state. Two servers, two files. Fifty servers, fifty files. Worse, state files get kept in a repository so a team can work on them, and every password you typed in is now readable by everyone who can read that repository.
Salt's answer sits in a second directory on this master, beside the state tree, holding nothing but data. It already exists on this box and it is empty. By the end of this lesson a machine on the other side of the network will be reading values out of it that you typed here and never sent by hand.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do, so what you type here is what you will type there. The lab user is saltops, the master is named saltmaster, and there is exactly one minion, named minion1. Your progress in the lab is tracked automatically, so type commands naturally and run check-progress whenever you want your score.
Start by looking at it. ls -l lists a directory one entry per line. Before you press Enter, commit to what an empty directory prints, because it is not nothing.
ls -l /srv/pillar
prompt: saltops@saltmaster:~$ answer: ls -l /srv/pillar ||| ls -l /srv/pillar/ output: total 0 hint: The long listing of a directory is ls -l, then the path: ls -l /srv/pillar
One line, and not a single entry under it. total 0 is the header ls -l always prints for a directory: it is the number of disk blocks used by the files inside. Zero blocks, because there are no files.
That directory is the pillar root, and its path is /srv/pillar, exactly parallel to /srv/salt being the file root for states. It was created for you here. On a master you build yourself it is one mkdir away, and Salt will not invent it.
That this directory exists is the first thing your lab grades. It is also the only one of the five you get for free.
Pillar is Salt's data store. It is a set of YAML files on the master that hold key and value pairs, compiled separately for each minion and handed only to the minion they belong to.
Think of it as a filing cabinet in the manager's office. Every worker has a folder in it with their name on the tab. A worker gets handed their own folder and never sees anyone else's. The state files say what to do with the contents. The pillar holds the contents.
That split is the whole idea. A state file says: deploy a config file, and take the port from pillar. The pillar says: on this machine the port is 8080. One state file, many machines, different answers, no copies.
Pillar has three pieces and you will build all three in the next few minutes:
1. Pillar data files, YAML files in /srv/pillar holding your keys and values. 2. The pillar top file, /srv/pillar/top.sls, which says which machines get which data files. 3. The pillar functions, pillar.get and pillar.items, which read the data back.
Salt learns where to look from the master config. On this box /etc/salt/master.d/lab.conf lists /srv/pillar under pillar_roots and /srv/salt under file_roots. Two roots, two jobs: one tree of instructions, one tree of data, and neither one knows anything about the other until the top files connect them.
Salt did not start with pillar. It started as a message bus for running commands, and states came afterwards. The moment states existed, people began writing real production values into them, because there was nowhere else to put them. Passwords, licence keys, per-customer settings, all sitting in a file meant to be shared and versioned and reviewed.
Pillar is the correction. It gives the data its own tree, its own permissions, and a compile step that runs per minion on the master, so a machine is only ever handed the values assigned to it.
Read that carefully, because pillar gets oversold constantly. Pillar is not encrypted at rest. It is plaintext YAML on the master's disk. Three other things protect it. File permissions on /srv/pillar decide who on the master can read it. The per-minion compile decides which slice a machine is handed. The transport that carries it to the minion is already encrypted. If you need the values encrypted inside the file itself, that is a separate opt-in feature called the GPG renderer, and it is not on by default here or anywhere.
A pillar file is the simplest YAML you will ever write. No state IDs, no modules, no requisites. Keys on the left, values on the right, one per line. You choose the names.
/srv/pillar belongs to your account on this box, so this is a plain redirect with no sudo in front of it. A heredoc is the shell's way of typing several lines into a file in one go: everything between the << 'EOF' marker and the closing EOF becomes the content of the file.
cat > /srv/pillar/app.sls << 'EOF'
app_name: linuxcamp
app_port: 8080
EOFTwo keys. app_name is the name of the application and app_port is the port it should listen on. The file ends in .sls, the same extension state files use, because Salt renders both through the same YAML pipeline.
The heredoc prints nothing when it works. Silence is success, which is unnerving the first time, so read the file back and check your own typing before any minion goes near it.
cat /srv/pillar/app.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/app.sls output: app_name: linuxcamp app_port: 8080 hint: Print a file to the screen with cat and its full path: cat /srv/pillar/app.sls
Two lines on disk, exactly as typed. That is a complete pillar file.
Look at the two values rather than the two keys, because they are not the same kind of thing. linuxcamp is text. 8080 has no quotes around it, so YAML reads it as a number. You did not choose that and nothing on this screen shows it yet. Hold on to it. It comes back later in this lesson as the most surprising screen in it.
Writing a pillar file is the second thing your lab grades. Note what the check is looking for: a file in /srv/pillar, ending in .sls, that is not empty. The key names are yours.
YAML indentation matters even in a file this small. These two lines start at column 0 with no leading spaces, and the space after each colon is required. Never use a tab anywhere in a YAML file. When Salt reports a rendering error on a pillar file, indentation or a missing space after a colon is almost always the cause.
The file exists on the master. The master is running, the minion is connected and accepted, and the values are sitting there in plain sight. Take a position before you write anything else.
>>> None of them. Dropping a file into /srv/pillar publishes it to nobody. Salt compiles a minion's pillar by reading the pillar top file, finding the entries whose target matches that minion, and merging only the files listed there. No entry, no data, and no complaint either: a pillar file nothing assigns just sits on disk being ignored. If you picked the first answer, that is the assumption worth killing early. It would mean every machine in a fleet could read every secret in the tree, which is the exact problem pillar was built to solve. If you picked the third, note that minions never request files by name. A minion asks for its pillar, once, as a whole, and the master decides what that contains.
The file that does the assigning is /srv/pillar/top.sls. It is the same shape as the state top file you wrote in the last lesson: an environment, a target, and a list.
cat > /srv/pillar/top.sls << 'EOF'
base:
'*':
- app
EOFThree lines, three jobs:
base: is the environment name, at column 0. The default, and the only one this master has.'*':** is the target pattern, indented 2 spaces. The star means every accepted minion. Quoting it keeps YAML from reading the star as something other than text.- app is the pillar file to hand over, indented 4 spaces. It is app.sls without the extension, and the leading - makes it a list item, so a target can be given several files by adding more lines.Read it back before you go near the minion. One second of cat beats five minutes of wondering why a value is empty.
cat /srv/pillar/top.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/top.sls output: base: '*':
hint: Same tool, the other file you just wrote: cat /srv/pillar/top.sls
Read it as a staircase. base: sits flush left. The target sits one step in at 2 spaces. The file name sits two steps in at 4 spaces. The depth is the meaning: - app belongs to '*': because it is indented under it, and for no other reason.
There are now two top files on this master and they do the same job for different trees. /srv/salt/top.sls assigns states to machines. /srv/pillar/top.sls assigns data to machines. Same grammar, same environment name, completely separate contents, and neither one is a substitute for the other.
A pillar top file containing base: is the third thing your lab grades.
Both files are written. The master can compile a pillar for minion1 the moment anything asks it to. The question is whether anything has.
Remember what a minion is: a process that started up some time ago, worked out its pillar then, and has been sitting there ever since.
>>> The old copy. A minion holds its compiled pillar in memory and keeps using it. Nothing about writing a file on the master reaches out and updates that, which is why Salt gives you a function whose entire job is to say: go and fetch yours again. If you picked the first answer, that instinct will cost you an afternoon one day. It is the single most common pillar bug, and it looks exactly like your data being wrong when in fact your data is fine and stale. If you picked the third, note that a refresh is a top-up rather than a wipe. A minion keeps serving what it has until the new copy arrives, which is what you want on a machine doing real work.
The function is saltutil.refresh_pillar. The saltutil module is Salt's housekeeping module: the functions in it manage the minion itself rather than the machine underneath it.
The command is the ordinary three part Salt shape you already use: sudo salt, a quoted target, then module.function. Keep the target quoted so your shell hands the star to Salt instead of expanding it into filenames.
This one returns a single word. Before you press Enter, decide whether that word will be telling you the refresh finished, or only that it was accepted.
sudo salt '*' saltutil.refresh_pillar
prompt: saltops@saltmaster:~$ answer: sudo salt '*' saltutil.refresh_pillar ||| sudo salt "*" saltutil.refresh_pillar ||| sudo salt 'minion1' saltutil.refresh_pillar output: minion1: True hint: Housekeeping lives in the saltutil module, and the function says what it does: sudo salt '*' saltutil.refresh_pillar
True, in the standard Salt return shape: the minion ID flush left with a colon, then its result indented four spaces underneath.
Read that True precisely. It means the minion accepted the instruction to refresh, not that a specific value has landed. The refresh itself takes a moment on the minion. On a busy machine, or a fleet of hundreds, a query fired instantly after the refresh can still answer from the old copy. The fix is to ask again a second later, not to refresh twice.
Build the habit now: every time you change a pillar file or the pillar top file, refresh. Forgetting it is the number one cause of the sentence beginners type into search engines: my pillar value is empty. Running this refresh is the fourth thing your lab grades.
Now ask the minion what it holds. pillar.get takes one key name and returns that key's value, straight from the minion's own copy.
This is the moment the whole lesson is aimed at, so be clear about what is being tested. The number you are about to see was typed on this master, in a file you wrote a few minutes ago. Nothing copied it across. The minion asked, the master compiled, and the value travelled.
sudo salt '*' pillar.get app_port
prompt: saltops@saltmaster:~$ answer: sudo salt '*' pillar.get app_port ||| sudo salt "*" pillar.get app_port ||| sudo salt 'minion1' pillar.get app_port output: minion1: 8080 hint: The pillar module reads pillar, get takes one key, and the key is the one you wrote: sudo salt '*' pillar.get app_port
8080. Your value, on a machine you never logged into, read out of a file you wrote on a different computer.
Trace the whole path once, because you built every link in it. You wrote app.sls. You wrote a top file assigning it to every minion. You told the minion to refresh, so it asked the master for its pillar. The master read the top file, saw that this minion matched '*', compiled app.sls into that minion's pillar and sent it. Now pillar.get reads the key out of the copy already sitting on the minion, which is why the answer comes back immediately.
Getting a value back from pillar.get is the fifth and last thing your lab grades.
Count what changed. Twenty minutes ago this master had an empty directory and a minion that knew nothing about your application. It now has a data tree, an assignment sheet, and a machine that can be asked for a value and answers with the one you chose.
That is the split that makes configuration management scale. Your states describe shape. Your pillar carries the numbers. Adding a second machine with a different port is one target line and one file, and no state file changes at all. Adding a third environment with completely different values is a second pillar file, and no state file changes at all.
The next lesson spends its whole length on the other half of this: pulling these values into state files and config templates so a machine configures itself from the data you just wrote. The rest of this lesson is about reading pillar well, including one screen that catches almost everybody.
One screen left that you have not seen. The next command prints every pillar key this minion holds, so app_name: linuxcamp and app_port: 8080 come back side by side in a single return. Take a position on those two lines before you run it.
>>> One of the two comes back changed, and the rule behind it has nothing to do with which key you consider sensitive. Nothing you did causes it and nothing is broken. Run the next command, read the two lines against each other, and work out which one changed before the step after it names the rule. If you picked the last answer, you are about to see exactly where that belief comes from.
Scaffolding off. No command is printed from here on.
pillar.get answers one key at a time, which is fine when you know what you are looking for and useless when you do not. You want the other view: one screen listing every pillar key this minion actually holds, so you can see the whole slice the master compiled for it. Same module, and the function name is the plural, list-everything companion to get.
prompt: saltops@saltmaster:~$ answer: sudo salt '*' pillar.items ||| sudo salt "*" pillar.items ||| sudo salt 'minion1' pillar.items output: minion1: ---------- app_name: ********** app_port: 8080 hint: The pillar module has a plural companion to get that takes no key at all: sudo salt '*' pillar.items
Two keys, which is exactly what you assigned. app_name and app_port, and nothing else, because app.sls is the only file the top file hands to this minion.
Read the layout first. The minion ID is flush left. A row of dashes opens the block, and that row is Salt's way of saying a nested structure starts here. Each key sits indented under it, and each key's value sits indented under its key. It is the same ID, colon, indent shape as every other Salt return, just one level deeper because there is more than one answer in it.
Then read the two values against each other, and notice that only one of them is a value.
app_port printed as 8080. app_name printed as a row of ten asterisks. Same file, same minion, same command, same second.
Nothing is broken and nothing is hidden from you. The Salt command line masks pillar values of one kind when it prints them: text is replaced with a fixed row of asterisks, numbers are printed as they are. It is a shoulder-surfing guard on your screen output, not a property of the data.
That is why the two lines differ. You wrote app_name: linuxcamp and YAML read linuxcamp as text, so it is masked. You wrote app_port: 8080 with no quotes and YAML read 8080 as a number, so it prints. Quoting a value changes its YAML type, and its type is what decides whether the command line prints it back to you.
Do not read those asterisks as security. You already have the proof: a few steps ago cat /srv/pillar/app.sls printed linuxcamp in the clear, and nothing stopped it. The stars are a display filter on the output. The file is plaintext, and anyone who can read the file can read the value. Count the two if you want the point driven home: linuxcamp is nine characters and the mask is ten asterisks, so the mask is not even telling you the length.
Reach for pillar.items first whenever pillar behaves strangely, before you reach for pillar.get. get tells you about one key you already believe in. items tells you the truth about the whole slice. It shows whether the file arrived at all, whether the top file matched this machine, and whether the key you keep typing is spelled the way you think.
Practice Pillar Basics in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.