Learn › Salt Configuration Management › State Files
Create top.sls to assign different states to different minions. Run state.highstate.
On the network behind this terminal there are two machines. minion1 is meant to be a web server. minion2 is meant to be a database server. Both are connected, both are trusted, and both will do anything this master tells them to.
On the master, in /srv/salt, there are instructions waiting for them. Nothing on this box says which instructions belong to which machine, so right now the master is holding two sets of orders and no delivery addresses.
You could send them by hand, one command per machine, and it would work today. Then the fleet grows to fifty machines and nine sets of orders and the hand-addressing falls apart. This lesson is about writing the addresses down once, in a file Salt reads for you, so that a single command sends the right software to the right machine every time.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do, so what you type here is what you will type there. Your own timestamps, durations and package versions will be your own. The lab user is saltops, the master is named saltmaster, and the two minions are minion1 and minion2. Your progress in the lab is tracked automatically, so type commands naturally.
Salt reads state files from one directory on the master, called the file root. On this box it is /srv/salt, and it was set up before you logged in. A state file is a plain text file, ending in .sls, that describes a condition a machine should be in.
Some of it was filled in for you before you logged in. Start by looking at the directory itself rather than guessing what is inside it. ls -l lists a directory one entry per line, with the permissions, owner, size and time on each line.
Before you run it, commit to a number: how many entries should come back?
ls -l /srv/salt
prompt: saltops@saltmaster:~$ answer: ls -l /srv/salt ||| ls -l /srv/salt/ output: total 12 -rw-r--r-- 1 root root 25 Jul 31 09:59 database.sls drwxr-xr-x 2 saltops saltops 4096 Jul 31 09:59 files -rw-r--r-- 1 root root 129 Jul 31 09:59 webserver.sls hint: The long listing of a directory is ls -l, then the path: ls -l /srv/salt
Three entries. Two state files, database.sls and webserver.sls, and one directory called files, the directory you shipped lab.conf out of in the service states lesson.
Read the first character of each line to tell them apart: - means an ordinary file, d means a directory. The two .sls files show root root for owner and group because the setup wrote them as root. You are only going to read those two. The directory itself belongs to your account, saltops, which matters in a minute when you write a file of your own into it.
Notice what is not here. There is no top.sls. That absence is the whole lesson.
ls told you the names. It told you nothing about the contents. cat prints a file to the screen, and a 129 byte file is small enough to read whole.
The name webserver.sls is a hint, not a promise. Read the file and find out what it actually asks for.
cat /srv/salt/webserver.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/salt/webserver.sls output: nginx: pkg.installed
nginx_service: service.running:
hint: Print a file to the screen with cat and the full path: cat /srv/salt/webserver.sls
This file holds two states, not one. A blank line separates them, and each one starts with a label at column 0.
The first is nginx:, the state ID, a label chosen by whoever wrote the file. Under it, pkg.installed says the nginx package should be present. The second is nginx_service:, and under it service.running says the nginx service should be running. Because that ID does not match the service name, the file spells the name out with - name: nginx.
Two more lines carry real weight. - enable: true asks for the service to come back after a reboot. - require: followed by - pkg: nginx is a requisite: it tells Salt that this state depends on the nginx state above it, so the package must be handled first. Salt reorders work to satisfy requisites rather than trusting the order you typed.
Remember the number two. It comes back on screen shortly.
Now the other one. It is 25 bytes, so expect something much shorter.
cat /srv/salt/database.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/salt/database.sls output: sqlite3: pkg.installed hint: Same tool, other file: cat /srv/salt/database.sls
One state, two lines. sqlite3: is the state ID and pkg.installed is the condition. sqlite3 is a small self-contained database engine, which makes it a quick stand-in for the heavier database software a real fleet would run.
Now read both files again in your head and notice what neither of them says. Not one line in either file names a machine. A state file describes a condition to be true, never who it is true for. webserver.sls says nginx should be installed and running; it does not say where. That separation is deliberate, and it is what lets one file serve a whole fleet.
So the master is holding two descriptions and no addresses. Knowing which file belongs to which machine, and writing that down, is the second thing your lab grades.
The top file is a single file at /srv/salt/top.sls that maps minions to the states they should have. It is an assignment sheet, and it holds lines that read like this:
Think of it like a class schedule at a school. Each person gets assigned specific classes. The schedule is the one document that keeps track of who gets what, and nobody has to remember it by heart.
Once that file exists, one command called state.highstate reads it and applies the correct states to every machine it reaches. You stop choosing what goes where at the moment you type. The file already chose.
Salt began as a remote execution bus, which is the salt command you have been typing since the first lesson. States came afterwards, and states needed somewhere to record who gets what. That record is the top file, and it sits at the top of the state tree in /srv/salt, which is where the name comes from.
The word highstate has the same root. When Salt takes every state a minion is assigned and compiles them into one complete picture of how that machine should look, the result is called that minion's highstate. state.highstate is the command that says: work out my highstate, and make me match it.
The first line of the top file is base:. That is an environment name. Salt uses environments to keep separate sets of states apart, so a team can run a dev tree and a prod tree side by side. base is the default environment and most setups never need a second one. On this box the master config at /etc/salt/master.d/lab.conf lists base under file_roots, pointing at /srv/salt.
You are about to write a sheet that names both machines and gives them different work. Before you make anything happen, take a position on what a single minion actually gets handed.
>>> Only database. When a minion is told to run a highstate, it reads the top file, finds the entries whose target matches its own minion ID, and compiles only those states. minion2 never applies the webserver states because nothing in the file assigns them to it. If you picked the first answer, that is the assumption worth unlearning early. The top file is a filter, not a broadcast list, and a fleet where every machine ran every state would be unmanageable. If you picked the third, note that a highstate is not a planning step. It compiles and applies in one pass, which is exactly what makes it a single command instead of two.
The file is short. Five lines do the whole job. A heredoc is the shell's way of typing several lines into a file in one go: everything between the << 'EOF' marker and the closing EOF becomes the content of the file.
/srv/salt belongs to your account on this box, so this is a plain redirect with no sudo in front of it:
cat > /srv/salt/top.sls << 'EOF'
base:
'minion1':
- webserver
'minion2':
- database
EOFFive lines, three jobs:
base: is the environment name, at column 0 with no indentation.'minion1': is a target pattern, indented 2 spaces. Here it is a plain minion ID. The quotes keep YAML from reading the value as something other than text.- webserver is the state file to apply, indented 4 spaces. It is webserver.sls without the extension. The leading - makes it a YAML list item, which is why a machine can be given several state files: add another - name line under the same target.The heredoc prints nothing when it works. Silence is success, which is unnerving the first time, so read the file back and check your own typing before you go near the minions.
cat /srv/salt/top.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/salt/top.sls output: base: 'minion1':
'minion2':
hint: Print your new file with cat and its full path: cat /srv/salt/top.sls
There it is on disk. Five lines, one environment, two targets, one state file each.
Read it as a staircase. base: sits flush left. Both machine names sit one step in at 2 spaces. Both assignments sit two steps in at 4 spaces. The depth is the meaning: - webserver belongs to 'minion1': because it is indented under it, and for no other reason.
In the lab, that this file exists at all is the first thing graded, and that it gives each machine its own state file is the second. cat is how you prove both to yourself, in one second, without waiting for a check to tell you.
YAML indentation is the whole grammar of this file. base: at column 0, targets at 2 spaces, state names at 4 spaces, and spaces only, never tabs. Get the depth wrong and Salt does not shout at you the way a compiler would. It reads a different structure than the one you meant, and you end up debugging a machine that quietly received nothing.
The command that reads this file is sudo salt '*' state.highstate. You have used '*' since the first lesson as the target that means every accepted minion. There are now two different selections happening in one line, and mixing them up is the classic first mistake with this command.
>>> It chooses the machines, not the states. The target is the same targeting you already know: '*' reaches every accepted minion, 'minion1' reaches one, and a glob like 'minion*' reaches a group. What each of those machines then applies comes from the top file, every time. If you picked the first answer, remember that the top file is the only thing in Salt that assigns states to machines; no target string can override it. If you picked the third, that is a reasonable guess from the shape of the command, but Salt never selects states by the target. The target selects who runs, and the file selects what runs.
One more position to take before you press Enter, and this one is about reading the screen rather than about targeting. Salt reports on every state it ran, one result block per state, with a per-machine summary underneath.
You read webserver.sls a few steps ago. Count what is in it, then answer.
>>> Two. webserver.sls holds two state IDs, nginx and nginx_service, and each one returns its own block with its own Result, Comment, Duration and Changes. The count is per state, not per file and not per machine, which is why the summary line at the bottom of a machine's output is called Total states run. If you picked the first answer, that would leave you unable to tell which half of a file failed, and Salt is built so a failure names itself. If you picked the third, note that Salt reports every state it ran whether or not it had work to do. A state that found nothing to change still tells you so, and that is often the result you most want to see.
The command is the ordinary three-part Salt shape you already use: sudo salt, a quoted target, then module.function. The function here is state.highstate and it takes no arguments, because the arguments are in the file you just wrote.
Keep the target quoted so your shell hands the star to Salt instead of expanding it into filenames. Expect this one to take a moment. Salt waits for a machine to finish before it prints that machine's block, and installing software takes seconds rather than milliseconds.
sudo salt '*' state.highstate
prompt: saltops@saltmaster:~$ answer: sudo salt '*' state.highstate ||| sudo salt "*" state.highstate output: minion2: ---------- ID: sqlite3 Function: pkg.installed Result: True Comment: The following packages were installed/updated: sqlite3 Started: 10:05:01.955637 Duration: 9666.24 ms Changes: ---------- sqlite3: ---------- new: 3.40.1-2+deb12u2 old:
Summary for minion2 ------------ Succeeded: 1 (changed=1) Failed: 0 ------------ Total states run: 1 Total run time: 9.666 s minion1: ---------- ID: nginx Function: pkg.installed Result: True Comment: The following packages were installed/updated: nginx Started: 10:05:01.945949 Duration: 10104.78 ms Changes: ---------- nginx: ---------- new: 1.22.1-9+deb12u9 old: nginx-common: ---------- new: 1.22.1-9+deb12u9 old: ---------- ID: nginx_service Function: service.running Name: nginx Result: True Comment: Running in OFFLINE mode. Nothing to do Started: 10:05:12.053116 Duration: 8.392 ms Changes:
Summary for minion1 ------------ Succeeded: 2 (changed=1) Failed: 0 ------------ Total states run: 2 Total run time: 10.113 s hint: Quote the target so the shell leaves the star alone, then ask for the highstate: sudo salt '*' state.highstate
One command went out. Two machines came back having done two different jobs, and neither one touched the other's software. You never named a state file and you never named a machine on the command line. The top file did both.
Find the two machine headings first, minion2: and minion1:, each flush left with a colon. Everything under a heading belongs to that machine until the next heading. Under minion2: there is one state, ID: sqlite3, and its summary says Total states run: 1. Under minion1: there are two, ID: nginx and ID: nginx_service, and its summary says Total states run: 2. That is your assignment sheet, executed.
Now look at the order. minion2 printed first, even though the command reads left to right and minion1 sorts first alphabetically. Salt prints each machine's block as that machine's answer arrives, and minion2 finished its work in 9.666 s while minion1 needed 10.113 s. On a fleet of two hundred, output order tells you who was quick, and nothing else. Never read meaning into it.
Getting a highstate to run at all is the third thing your lab grades.
Three states ran across the two machines and every one of them says Result: True. Two of them changed the machine and one of them did not. Learning to see that difference at a glance is the reading you will do for the rest of your Salt life.
Result is the verdict. Changes is the receipt. A state describes a condition, and Result: True means that condition holds now. It says nothing about whether Salt had to do work to get there. That is why every block carries a separate Changes: field.
Put the two package states side by side and they tell the same story twice. Both report Comment: The following packages were installed/updated: with a package name after it. Both carry a populated Changes block: the package name, then new: with the version that landed, and old: with nothing after it, because there was no previous version to replace. minion1's block lists two packages under Changes, nginx and nginx-common, because installing nginx pulls in its common files as a dependency and Salt reports everything that actually moved. Both took seconds: 9666.24 ms and 10104.78 ms.
Now the third block, nginx_service. Same Result: True, and everything else is different. Its Comment reads Running in OFFLINE mode. Nothing to do, its Changes: field is bare, and its Duration is 8.392 ms, under ten milliseconds against ten seconds. These lab minions are lightweight containers with no running init system for Salt to talk to, so Salt detected that, reported it plainly, and did not pretend to start anything. True with an empty receipt is Salt telling you it had nothing to do.
The summary lines carry the same story in one word each. minion2 reports Succeeded: 1 (changed=1) and minion1 reports Succeeded: 2 (changed=1). That changed= counter counts states that moved, not machines: minion1 ran two states and only one of them changed anything. When you scan a fleet-wide highstate, changed= is the number your eye should go to, because it is the list of what was not already correct.
There is a second thing hiding in the timestamps. nginx started at 10:05:01.945949 and ran for 10104.78 ms, which lands it just past 10:05:12. nginx_service started at 10:05:12.053116, right after. That is the - require: line from webserver.sls doing its job: the service state waited for the package state to finish. Requisites are not a suggestion about ordering, they are the ordering.
Count what changed. You arrived with two anonymous state files a
Practice Top File in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.