Learn › Salt Configuration Management › Jinja and Advanced States
Configure Salt environments (base, dev, prod) and assign minions to different environments.
There are two state folders on this master right now. /srv/salt is the one every state file you have written so far has lived in. /srv/salt-dev was created before you logged in, and it is sitting there empty.
Both are real directories. Both belong to your account. Both are readable by everything on the box. Write the same state file into either one, ask a minion to run it, and the two folders do not behave the same way.
Nothing about the file explains that. It is spelled correctly, it has the right permissions, and it is right there on the disk either way. The difference is something the master was told, and working out what is the whole of this lesson. By the end you will have two named environments on one master, a state that exists in only one of them, and the exact error Salt gives when you ask for it from the wrong one.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do. Your own timestamps and durations will be your own. The lab user is saltops, the master is named saltmaster, and there are two minions, minion1 and minion2. Both keys are already accepted for you this time. Your progress in the lab is tracked automatically, so type commands naturally.
The master keeps its settings in /etc/salt/master.d/, a drop-in directory. Any file ending in .conf in there is read as part of the master's configuration, which is tidier than editing one giant file. This box has a single drop-in called lab.conf.
Before you print it, commit to a number. Two directories exist on this box. How many of them do you expect this file to name?
cat /etc/salt/master.d/lab.conf
prompt: saltops@saltmaster:~$ answer: cat /etc/salt/master.d/lab.conf output: interface: 0.0.0.0 auto_accept: True file_roots: base:
pillar_roots: base:
hint: Print the file with cat and its full path: cat /etc/salt/master.d/lab.conf
Eight lines, four settings. interface: 0.0.0.0 is the address the master listens on, and auto_accept: True is why both minion keys were already accepted before you arrived. Those two are setup, not today's subject.
file_roots: is. It is a mapping from a name to a list of directories, and there is exactly one name under it: base, pointing at /srv/salt. Underneath, pillar_roots: has the same shape for pillar data, with its own base pointing at /srv/pillar.
Now read what is missing. /srv/salt-dev appears nowhere in this file. The directory exists on the disk and the master has never been told about it, which is the answer to the mystery you opened with. Salt is not reading your disk. It is reading this file.
A Salt environment is a name in file_roots with one or more directories behind it. That is the entire definition, and everything confusing about environments comes from expecting more.
Three consequences follow from it, and they are worth taking one at a time.
One name is always there: base. It is the default, and it is the environment every state you have applied so far has come from without you ever naming it.
The names you add are yours to choose. dev, staging and prod are the common three, because most teams want the same shape: try a change somewhere harmless, then move it somewhere that matters.
Salt did not invent this pattern. It borrowed the shape of it from the way software has been released for decades. You write a change somewhere it can do no damage, and you prove it there. Then you promote the same change to the machines that matter. Version control taught everyone the same habit with branches.
What Salt added is that configuration code deserves the same treatment as application code. Your state files decide which packages exist on a production database server. Editing them in place, on the one master that talks to production, is the sort of arrangement that stays fine right up until it is not.
So file_roots was built as a mapping rather than a single path, and the value of each name is a list rather than one directory. That is why an environment can be assembled from several folders, and why adding one is a two-line edit rather than a second master.
Carry this sentence around: an environment is a label the master hands out, not a place on disk. Almost every environment problem is somebody reasoning about the folder while Salt reasons about the label. When a state file that plainly exists cannot be found, the question is never where is the file. It is which labels can see it.
Before you change any configuration, confirm what is actually on the disk. ls -d lists a directory as one entry instead of listing what is inside it, and -l gives you the long form with permissions, owner and time. Together, ls -ld is the way to ask about the folders rather than their contents.
Give it both paths at once so the two lines sit next to each other. Watch the owner column, because it decides how you do every write for the rest of this lesson.
ls -ld /srv/salt /srv/salt-dev
prompt: saltops@saltmaster:~$ answer: ls -ld /srv/salt /srv/salt-dev ||| ls -dl /srv/salt /srv/salt-dev output: drwxr-xr-x 2 saltops saltops 4096 Jul 31 19:18 /srv/salt drwxr-xr-x 2 saltops saltops 4096 Jul 31 19:18 /srv/salt-dev hint: The long listing of the directories themselves, not their contents, is ls -ld: ls -ld /srv/salt /srv/salt-dev
Two lines, and they are near enough identical. Both start with d, so both are directories. Both carry the same permissions. Both show the same modification time, down to the minute the listing prints.
Read the owner column: saltops saltops, your own account, on both. The setup handed you these trees, so every file you write into either one is a plain redirect with no sudo in front of it.
The number after the permissions is the link count, and 2 on a directory means it holds no subdirectories. Both folders are bare. Nothing has been written into either yet, which is worth knowing before you go looking for a state file that is not there.
So the disk shows two equal folders and the config file shows one name. The difference between them is entirely in the master's head.
You have both facts in front of you now. The folder is there, it is yours, and the master has never heard of it. Take a position before you change anything.
>>> No. Salt never goes hunting through the filesystem for state files. It builds a list of directories from file_roots, and it searches that list and nothing else. /srv/salt-dev is not on the list, so nothing in it can be found, whatever its permissions say. If you picked the first answer, that is the assumption this whole lesson exists to correct, and it is the reason people lose an afternoon to a state file that is plainly present. If you picked the third, note that state.apply does not take paths at all. It takes a state name, webserver for webserver.sls, and resolves it against the environment's directory list. There is no syntax for reaching outside that list, by design: if there were, a typo in a path would be a way to run whatever happened to be on the master's disk.
Adding an environment means adding a name under file_roots and a directory under that name. Two lines, in the file you just read.
You are going to rewrite the whole file rather than edit it in place, because retyping eight lines is faster than fighting an editor over YAML indentation. A heredoc is the shell's way of typing several lines into a file at once: everything between the << 'EOF' marker and the closing EOF becomes the content of the file.
The three original settings are kept exactly as they were. Only the dev block is new:
cat > /etc/salt/master.d/lab.conf << 'EOF'
interface: 0.0.0.0
auto_accept: True
file_roots:
base:
- /srv/salt
dev:
- /srv/salt-dev
pillar_roots:
base:
- /srv/pillar
EOFNo sudo on that write. The setup handed lab.conf to your account along with the two state trees, so it is yours to change. If a write into /etc is ever refused, ownership is the first thing to look at, not the command.
A heredoc prints nothing when it works, so read the file back before you trust it. You only care about one block, so ask grep for the file_roots line plus the four lines after it. That is what -A4 means: after.
grep -A4 file_roots /etc/salt/master.d/lab.conf
prompt: saltops@saltmaster:~$ answer: grep -A4 file_roots /etc/salt/master.d/lab.conf ||| grep -A 4 file_roots /etc/salt/master.d/lab.conf ||| grep -A4 'file_roots' /etc/salt/master.d/lab.conf output: file_roots: base:
dev:
hint: Print the matching line plus the four after it: grep -A4 file_roots /etc/salt/master.d/lab.conf
Five lines, two names. base still points at /srv/salt and dev now points at /srv/salt-dev, and that second pair is the thing that did not exist a minute ago. A file_roots block naming a dev environment is the first of the five things your lab grades.
Read it as a staircase, because the depth is the meaning. file_roots: sits flush left at column 0. Both environment names sit one step in at 2 spaces. Both directory paths sit two steps in at 4 spaces, each with a leading - that makes it a list item.
That leading dash matters more than it looks. Each name holds a list of directories, not one directory, so an environment can be spread across several folders and Salt will search them in order. Today each list has one entry, and it still has to be written as a list.
The indentation in this file is not decoration. file_roots: at column 0, environment names at 2 spaces, paths at 4 spaces with - in front, spaces and never tabs. Get it wrong and you rarely get an error. YAML reads the structure you actually typed, which is a different structure from the one you meant, and the master quietly ends up with fewer environments than you think it has.
The file on disk now names two environments. You have not touched the salt-master service. Take a position on what the running master currently believes.
>>> No. The master reads its configuration once, at startup, and holds the result in memory. Editing the file changes the file. It does not reach into a running process and change its mind. If you picked the first answer, plenty of daemons do watch their config, which is exactly why this one catches people out. Save the habit instead of the exception: after touching anything in /etc/salt/master.d/, restart the master. If you picked the third, notice which machine is holding the stale information. file_roots is master-side only, and the minions never see it. Refreshing a minion cannot fix a master that has not reread its own file.
Restarting is one command, and like most systemctl verbs it prints nothing at all when it works:
sudo systemctl restart salt-masterSilence is success, which is unnerving the first time. It is also not the reassurance you want here, because the thing you actually care about is whether both machines are talking to the master again on the other side of the restart.
So ask them. test.ping publishes a tiny job over the Salt bus and a healthy minion answers True, which proves the connection, the accepted key and the round trip in one line. Give the master a couple of seconds to come up first.
Before you press Enter, commit to a number: how many blocks should come back?
sudo salt '*' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt '*' test.ping ||| sudo salt "*" test.ping output: minion1: True minion2: True hint: Quote the target so the shell leaves the star alone, then the module and function: sudo salt '*' test.ping
Two blocks, two True values. Both machines reconnected on their own and the master is serving again with a second environment in its head.
Notice who did the work. You restarted the master, not the minions. A minion whose master goes away does not fail and does not need touching. It keeps trying, reconnects when the master answers again, and carries on. That is why restarting a master is a routine operation rather than a maintenance window.
One habit worth forming from this screen: after any master config change, restart and then ping. The restart is the change, and the ping is the evidence. A master that comes back with a broken config file will not answer this at all, and finding that out now is much cheaper than finding it out during a highstate.
Now put something in the folder that only the dev name can see. The point of this file is not what it does. It is where it lives.
cat > /srv/salt-dev/devonly.sls << 'EOF'
/tmp/which-env.txt:
file.managed:
- contents: this came from the dev environment
EOFThree lines, and the shape is one you have seen since the first state lesson. The first line is the state ID, the label for this piece of work. file.managed is the state function, meaning make this file exist with this content. - contents: is the text to put in it.
One detail is doing extra duty here. There is no - name: line, so Salt falls back to using the state ID as the file path. That is why the ID is written as /tmp/which-env.txt rather than a friendly word: the label and the target are the same string.
Read the file back and check your own typing before any minion sees it.
cat /srv/salt-dev/devonly.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/salt-dev/devonly.sls output: /tmp/which-env.txt: file.managed:
hint: Print your new file with cat and its full path: cat /srv/salt-dev/devonly.sls
There it is on disk, in the folder the dev name points at and nowhere else. A state file in /srv/salt-dev is the second of the five things your lab grades.
The file name is the important part. devonly.sls on disk becomes the state name devonly on the command line, because state.apply takes the name without the extension. There is no devonly.sls in /srv/salt, so that name exists in exactly one of this master's two environments.
That is the experiment set up. One name, present in one environment, absent from the other, on a master that now knows both. Everything after this is reading what Salt does with it.
Every state command you have run so far has used the base environment without saying so. To reach a different one you add saltenv= to the end of the command, with the environment name after it.
It goes on the end because it is a keyword argument to the function, not a flag on the salt command. Flags like -G and -I sit before the target. Keyword arguments sit after the function and its arguments, written as key=value with no spaces around the equals sign.
So this says: on minion1, apply the state named devonly, looking for it in the dev environment.
Before you press Enter, commit to two things. How many state blocks should come back from a file holding one state, and what should Changes show for a file that does not exist on that machine yet?
sudo salt 'minion1' state.apply devonly saltenv=dev
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' state.apply devonly saltenv=dev ||| sudo salt "minion1" state.apply devonly saltenv=dev ||| sudo salt minion1 state.apply devonly saltenv=dev output: minion1: ---------- ID: /tmp/which-env.txt Function: file.managed Result: True Comment: File /tmp/which-env.txt updated Started: 19:23:50.286472 Duration: 4.32 ms Changes: ---------- diff: New file
Summary for minion1 ------------ Succeeded: 1 (changed=1) Failed: 0 ------------ Total states run: 1 Total run time: 4.320 ms hint: The usual apply, with the environment named as a keyword argument on the end: sudo salt 'minion1' state.apply devonly saltenv=dev
It ran. Read the block from the top. minion1: is the machine reporting. The row of dashes opens one state block, and there is one because your file held one state.
ID: /tmp/which-env.txt is the label you chose, which is also the path, exactly as promised. Function: file.managed is what ran. Result: True is the only field that says whether it worked. Comment: File /tmp/which-env.txt updated is Salt's sentence for this case.
Under Changes: there is a diff: reading New file. Salt is telling you it did not edit anything, it created something that was not there. On a second run of this same command that entry would be empty, because the machine would already match the description.
Then the summary. Succeeded: 1 (changed=1) counts state blocks, not files: one block ran, one block had to change something. Failed: 0. Total run time: 4.320 ms, which is the whole cost of pushing a file to a machine you never logged into.
Now the part that is actually the lesson. Salt found devonly because you told it which environment to look in. Take saltenv=dev off that command and every other character stays the same. Running state.apply at all is the fourth of the five things your lab grades.
Count what changed. You arrived on a master that knew one folder. It now knows two, by name, and you have proved the second one works by pushing a file out of it onto a real machine.
That is the mechanism behind every dev, staging and prod setup you will ever be handed. There is no second master, no second copy of Salt, and no clever tooling. There is a mapping from names to folders, a restart, and an argument on the end of a command.
The rest of this lesson is the two halves that make it useful in real work: giving each machine its own environment permanently, and learning what a miss looks like.
Typing saltenv=dev on every command is fine for on
Practice State Environments in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.