Learn › Salt Configuration Management › State Files
Combine pkg, file, and service states with the watch requisite for automatic service restarts.
A state run finishes. Every block on the screen reads Result: True. Nothing failed, nothing errored, the summary counts zero failures.
Ten seconds later you ask that same machine a direct question about the same service, and the answer is one word that does not agree with the report you just read.
Nobody lied to you. Salt told you the truth twice. You were reading the wrong field.
This lesson fixes that habit for good. You will write the state that puts a service into the running state. You will build the three-state pattern that almost every application on a real fleet is described with. And you will add the requisite that restarts a service when its config file moves under it.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output shown was captured from a real Salt master with one real minion attached, which is exactly the machine your lab boots. The lab user is saltops, the master is named saltmaster, and there is exactly one minion, named minion1.
Installing software copies files onto a disk. That is all it does. A service is a program that is actually running in the background right now, holding a port open and answering requests.
Those are two separate facts about a machine, and Salt keeps them in two separate states.
service.running is the state that covers the second one. It says: this service must be up. If it is down, start it. If it is already up, leave it alone.
Notice the wording. You are not telling Salt to start anything. A state is a description of how the machine should end up, not a command to run. Salt reads your description, compares it with the machine, and decides for itself whether there is any work to do.
Thomas S. Hatch shipped Salt in 2011 to get answers out of a whole fleet in under a second. The states layer followed quickly, and it changed the question being asked. Not what should I run on these machines, but what should be true about them.
That change creates a new problem immediately. Once you are describing five facts about a machine instead of running five commands in order, something has to say which fact depends on which. Salt calls a declared relationship between two states a requisite.
There are several requisites in Salt and you will meet two of them today. require says run that other state first, and only continue if it worked. watch says the same thing, and adds: if that other state actually changed something, react. Every other requisite in Salt is a variation on those two ideas.
Start with two states in one file. Write it on the master, in the directory Salt serves files from:
cat > /srv/salt/websvc.sls << 'EOF'
nginx:
pkg.installed: []
service.running:
- require:
- pkg: nginx
EOFRead the wrapper first. /srv/salt is the directory the master serves state files from, and it belongs to your lab account, so a plain redirect writes into it with no sudo at all. Everything between << 'EOF' and the closing EOF is the text being written. Salt commands still need sudo, because reading the master key is a root job; writing a file here is not.
Now read the state itself line by line, because every line is doing a job:
nginx: at the left margin is the state ID, your label for this block of work. When you do not say otherwise, Salt also uses the ID as the name of the thing being managed. So this ID names both the package and the service.pkg.installed: [] is the first function hanging off that ID. The [] is an empty list, meaning no options. Salt allows more than one function under a single ID, which is how one label can describe both the package and the service.service.running: is the second function under the same ID.- require: opens the requisite list, and - pkg: nginx is the one entry in it. Read that entry as a type and an ID: the pkg state whose ID is nginx.The whole file in one sentence: nginx must be installed, nginx must be running, and the installing comes first.
>>> It ties them together. A requisite is about success, not just order. Salt runs the required state first, and if that state fails, the state that requires it is not attempted and reports which requisite let it down. Without the requisite you would get an install failure followed by a second failure trying to start software that is not there, and two failures are harder to read than one. If you picked the first answer, two functions under one ID is legal Salt all by itself, no requisite needed. If you picked the third, requisites are the opposite of parallel: they are how you tell Salt that one thing has to settle before another begins.
Every Salt command keeps the same three-part shape you already know: sudo salt '<target>' <module>.<function>. Here the function is state.apply, and it takes one argument, the name of your file without the .sls on the end.
Before you press Enter, commit to two things. How many blocks come back, and what should sit under Changes: in each one. Apply the file to your one minion:
sudo salt 'minion1' state.apply websvc
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' state.apply websvc ||| sudo salt "minion1" state.apply websvc ||| sudo salt minion1 state.apply websvc output: minion1: ---------- ID: nginx Function: pkg.installed Result: True Comment: All specified packages are already installed Started: 09:10:05.560663 Duration: 21.204 ms Changes: ---------- ID: nginx Function: service.running Result: True Comment: Running in OFFLINE mode. Nothing to do Started: 09:10:05.583866 Duration: 8.007 ms Changes:
Summary for minion1 ------------ Succeeded: 2 Failed: 0 ------------ Total states run: 2 Total run time: 29.211 ms hint: The target is quoted, the function is state.apply, and the argument is the file name without .sls: sudo salt 'minion1' state.apply websvc
Two blocks, separated by a row of dashes, one block per state. Each block reports the same seven fields, and they are worth naming once:
ID is your label. Function is what Salt ran. Result is whether that function errored. Comment is the module explaining what it decided. Started and Duration are timing. Changes is what actually moved on the machine.
Now read the two Comments. The package state says All specified packages are already installed, so nginx was on this machine before the run and there was nothing to install. The service state says Running in OFFLINE mode. Nothing to do, which is the service module reporting that it did not act on this pass.
Then look under both Changes: fields. Empty. Two states, two Result: True lines, and nothing changed on the machine at all.
The summary at the bottom counts the same thing a different way: Succeeded: 2, Failed: 0, two states run. Hold on to that empty Changes field for the next screen.
This capture ran on a master where nginx had already been installed by earlier work, which is why the package state had nothing to do. On a fresh box the same command installs the package and says so, and that block takes several seconds instead of twenty milliseconds.
service.status is not a state. It is a plain function you call directly, the same way you called test.ping. It asks one minion a yes or no question about one service and hands back a boolean.
That makes it the perfect second opinion. The state run told you what Salt decided. This tells you what the machine says.
Before you run it, commit to an answer. The run you just did came back completely green. Ask the minion about nginx:
sudo salt 'minion1' service.status nginx
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' service.status nginx ||| sudo salt "minion1" service.status nginx ||| sudo salt minion1 service.status nginx output: minion1: False hint: The module is service and the function reports a status, then the service name: sudo salt 'minion1' service.status nginx
False. A green state run, and the service is not up.
Nothing here is broken and nothing here contradicts anything. Result: True means one thing only: that state ran without throwing an error. It is not a promise about the machine. The promise about the machine lives in the other two fields, and both of them told you the truth on the previous screen. Comment said nothing to do. Changes was empty.
The return shape is the one you already read a hundred times: the minion ID on its own line ending in a colon, then the value indented four spaces underneath. service.status answers True or False and nothing else, which is exactly what you want from a second opinion.
That is the single most valuable habit in this entire track, and you just built it in two commands.
Plenty of people run Salt for a year, read Result, see green, and walk away. From here on you read three fields in order. Result tells you whether the state errored. Comment tells you what the module decided. Changes tells you what actually moved.
The rest of this lesson gives that habit something to catch, because you are about to write a state file that does move something.
What you have so far is two thirds of the most common shape in Salt. Nearly every piece of software on a fleet is described the same way:
1. Install the package. pkg.installed makes sure the software is present. 2. Deploy the configuration. file.managed makes sure the config file has the right content. 3. Start the service. service.running makes sure the process is up.
Experienced Salt users call this pkg/file/service and build every new state around it. A database server? Install postgresql, deploy its config, start the postgresql service. A DNS server? Install bind, deploy named.conf, start the named service. The pattern does not change, only the three names inside it.
There is a hole in that pattern, though, and it is a famous one. Suppose you change the config file on the master and re-apply. Salt updates the file on the minion, and the running process carries on serving the old configuration it loaded into memory at startup. The file on disk is new. The behaviour is old.
>>> Nothing happens to it. require only ever asks two questions: did that other state run first, and did it succeed. It has no opinion about whether that state changed anything. So the service state looks at a service that is already up, decides there is nothing to do, and moves on. If you picked the second answer, that is the behaviour you want and it is exactly what watch is for, but you have to ask for it. If you picked the third, a changed file is not a failure. Salt reports it as a change and the run stays green, which is precisely why this gap catches people out.
In the combined file you are about to write, the service state gets its own ID, nginx_service, instead of sharing the nginx ID. Two reasons. The file reads as three separate jobs, and a requisite pointing at it is unambiguous.
That raises a question first, though. The service on the machine is still called nginx, and you just labelled the state something else.
- name: nginx line, because Salt only falls back to the ID when you do not give it a name.>>> It needs - name: nginx. The ID is a convenience: when you leave the name out, Salt borrows the ID and uses that. The moment your label stops being the real name of the thing, you have to say the real name yourself. If you picked the first answer, Salt does no guessing here at all and would go looking for a service literally called nginx_service. If you picked the third, requisites answer a different question entirely: they say what has to happen first, never what the thing is called.
Now the whole pattern, in one file, with the gap closed. Same plain redirect as before, because this file lands in the same directory your account owns. The source file it deploys is already on this master at /srv/salt/files/lab.conf:
cat > /srv/salt/req.sls << 'EOF'
nginx:
pkg.installed: []
/etc/nginx/conf.d/lab.conf:
file.managed:
- source: salt://files/lab.conf
- require:
- pkg: nginx
nginx_service:
service.running:
- name: nginx
- require:
- pkg: nginx
- watch:
- file: /etc/nginx/conf.d/lab.conf
EOFThree IDs, three jobs. The middle one uses the target path as its ID, which is a common habit with file states because the path is already unique.
The new lines are at the bottom. watch: opens a list in the same shape as require:, and each entry is a type and an ID. Here - file: /etc/nginx/conf.d/lab.conf means the file state whose ID is that path. Read the two requisites together as one sentence: run after the package, and if that config file ever changes, restart me.
watch: has to sit at the same indentation as the other options under service.running, and the - file: entry is indented two spaces further under it. YAML cares, and getting this wrong is the most common error in the whole states layer. If the run comes back complaining about the SLS file rather than about nginx, count your spaces before you doubt anything else.
Scaffolding off. No command is printed from here on.
Salt's service functions are not limited to services your states manage. Prove that to yourself before the big apply: ask about a service named cron, the scheduler, which no state of yours has ever touched. Ask the master to report on it, using the same function you used on nginx.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' service.status cron ||| sudo salt "minion1" service.status cron ||| sudo salt minion1 service.status cron output: minion1: False hint: The function you used to check nginx takes any service name you hand it.
One boolean, in the same shape as before. That is the point of the challenge: service.status is a question, not a change. It reports on any service name you give it, whether or not Salt has ever managed that service, and it never touches the machine.
This one came back False, so the minion did not report cron as up when it was asked. When a service you expected to be up answers False, treat that as a lead to chase on the machine itself rather than as a Salt fault. The query is doing its job either way.
Last one, still no command shown.
Your combined file is written and sitting on the master. Send it to the one minion this master trusts, the same way you sent the two-state file, with the new file name in place of the old one. Watch the third block when it comes back.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' state.apply req ||| sudo salt "minion1" state.apply req ||| sudo salt minion1 state.apply req output: minion1: ---------- ID: nginx Function: pkg.installed Result: True Comment: All specified packages are already installed Started: 09:12:03.469091 Duration: 22.189 ms Changes: ---------- ID: /etc/nginx/conf.d/lab.conf Function: file.managed Result: True Comment: File /etc/nginx/conf.d/lab.conf updated Started: 09:12:03.493606 Duration: 4.526 ms Changes: ---------- diff: New file ---------- ID: nginx_service Function: service.running Name: nginx Result: True Comment: Service started Started: 09:12:03.506200 Duration: 1029.798 ms Changes: ---------- nginx: True
Summary for minion1 hint: Same three-part shape as the earlier apply, and the argument is the new file name without .sls.
Three blocks now, in the order you wrote them, and this time two of them have something under Changes:.
The middle block is the config file. Comment: File /etc/nginx/conf.d/lab.conf updated, and under Changes a diff: reading New file. That state changed something, which is the exact condition your watch line exists to react to.
Now read the third block field by field, because it is the one you came for. ID: nginx_service is the label you chose. Name: nginx is a row you have not seen before. Salt prints it only when the ID and the real name differ, so it is Salt saying out loud that it acted on the nginx service and not on something called nginx_service. Comment: Service started. And under Changes:, the service name with True beneath it.
That Changes block is the whole lesson. Your first run left Changes empty and the machine was untouched. This run names what moved and what it became. Duration: 1029.798 ms backs it up: real work takes real time, and a second is a long time next to the four milliseconds the file state needed.
One word worth collecting. The comment reads Service started, not restarted. Started means it was down and Salt brought it up. Restarted is the word you get when a watch fires against a service that was already up. Reading which of the two you got tells you what state the machine was in before you arrived.
Applying that same file again is a boring, cheap thing to do, and that is by design rather than by luck.
A state file only ever acts on the difference between what you asked for and what is there. Once the machine already matches the description, a run has no difference left to act on. That property has a name worth carrying: a state file is idempotent, which means applying it once and applying it ten times leave the machine in the same place.
That is not a limitation, it is the entire point of describing a machine instead of scripting it. It is what makes a state file safe to apply on a schedule forever.
There is one more option worth adding before you trust this with anything real. Put - enable: True under service.running and you are asking for two promises instead of one: running now, and switched on at
Practice Service States in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.