Learn › Salt Configuration Management › Multi-Minion Orchestration
Use salt-run state.orchestrate for coordinated multi-minion deployments.
There are two machines on this master today. minion1 is the web tier and minion2 is the database tier, and there is a state file waiting for each of them, written before you logged in.
You can apply both. What you cannot do, with anything you have learned so far, is say which one goes first and which one waits.
That sounds like a small gap. A tier that comes up before the tier it depends on is a tier that comes up broken, and nothing in state.apply gives you a way to say which one waits. In the plan you are about to write, the web tier goes first and the database tier is the one that waits for it.
By the end of this lesson the order will live in a file on the master, and one command will run the whole plan.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do, so your own timestamps and durations will be your own. The lab user is saltops, the master is named saltmaster, and there are two minions, minion1 and minion2. Both keys are already accepted for you. Your progress in the lab is tracked automatically, so type commands naturally.
/srv/salt is where every state file on this master lives, and there is one directory inside it that matters today.
Print the tree and that directory together. Naming two paths makes ls label each one before its contents, and -l gives you the long listing with owners and sizes.
ls -l /srv/salt /srv/salt/orch
prompt: saltops@saltmaster:~$ answer: ls -l /srv/salt /srv/salt/orch output: /srv/salt: total 16 -rw-r--r-- 1 saltops saltops 105 Aug 4 23:34 database.sls drwxr-xr-x 2 saltops saltops 4096 Aug 4 23:34 orch -rw-r--r-- 1 saltops saltops 63 Aug 4 23:34 top.sls -rw-r--r-- 1 saltops saltops 102 Aug 4 23:34 webserver.sls
/srv/salt/orch: total 0 hint: The long listing of two paths at once: ls -l /srv/salt /srv/salt/orch
Four entries, and one of them is not a file.
database.sls and webserver.sls are the two state files this lesson uses. One per tier.top.sls is the top file from earlier in this track. This lesson does not use it and does not change it.orch begins with a d in the permissions column, which is how ls says directory.Then the second listing: /srv/salt/orch: followed by total 0. That is not an error. total 0 is an empty directory, and it is empty because filling it is your job.
Read the owner column on every line: saltops saltops. This whole tree is yours, which is why every write into /srv/salt in this lesson is a plain redirect with no sudo in front of it.
Before you can order two pieces of work you have to know what each one does. Print the first state file.
cat /srv/salt/webserver.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/salt/webserver.sls output: web_ready: file.managed:
hint: Print the file: cat /srv/salt/webserver.sls
Four lines, and you have read this shape since the file states lesson.
web_ready: at the left margin is the ID, your label for this block of work. file.managed is the state function, and it means make this file exist with these contents. - name: is the path it manages, /tmp/web_deployed. - contents: is the text that has to be inside it.
One property of file.managed matters later, so hold on to it: it is idempotent. Running it a second time changes nothing, because the file already matches what was asked for. It does not fail and it does not write again. It looks, sees the file is already right, and reports that it did nothing.
Now the second one. Expect the same shape with different values.
cat /srv/salt/database.sls
prompt: saltops@saltmaster:~$ answer: cat /srv/salt/database.sls output: db_ready: file.managed:
hint: Print the file: cat /srv/salt/database.sls
The same four lines with a different ID, a different path and different contents. db_ready manages /tmp/db_deployed.
Both states are deliberately small. A real web state installs a package, writes a config and starts a service. None of that would teach you anything new here, because the hard part of this lesson is not what a tier does. It is when it does it. Two machines and two marker files are enough to prove an order happened.
Orchestration is a file that describes work across several machines, in a stated order, and a single command that runs it from the master.
It is written in an SLS file like the ones you already write, but the functions inside it are different, because the thing being described is different. A state file describes what one machine should look like. An orchestration describes what a fleet should do, and in what sequence.
Two functions to learn, plus one word you already know.
salt.state applies a state file to a target. It is state.apply written as a step in a plan.salt.function runs an execution function on a target. It is salt 'minion1' test.ping written as a step in a plan.require is the same requisite you met in the requisites lesson, doing the same job it always did: this step waits for that step.Two things here are easy to get wrong, and the rest of the lesson keeps returning to both.
The first is that salt.state and salt.function exist only inside orchestration files. Put them in an ordinary state file and Salt will not know what you mean.
The second is that an orchestration is run by salt-run, not by salt. salt-run executes a runner, which is work that happens on the master itself instead of being handed out to minions. The master reads the plan and the master issues each step. The minions never see the plan at all.
Before you build a plan, be honest about what you have without one. Take a position.
>>> There is no answer, and that is the whole problem. The master publishes one job to every matching minion at once, and each minion runs it alone and returns whenever it is done. If you picked the first option, the order the master matched targets in is not an execution order, it is only the order results happen to print in. If you picked the third, Salt has no idea what a database is. Nothing in webserver.sls or database.sls mentions the other tier. Order has to be written down or it does not exist.
Do the web tier the old way once. This is the command you have used all track: target a minion, apply a state by name.
sudo salt 'minion1' state.apply webserver
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' state.apply webserver ||| sudo salt "minion1" state.apply webserver ||| sudo salt minion1 state.apply webserver output: minion1: ---------- ID: web_ready Function: file.managed Name: /tmp/web_deployed Result: True Comment: File /tmp/web_deployed updated Started: 23:34:45.624497 Duration: 4.272 ms Changes: ---------- diff: New file
Summary for minion1 ------------ Succeeded: 1 (changed=1) Failed: 0 hint: Target the minion and apply the state by name: sudo salt 'minion1' state.apply webserver
Read this block carefully. Every screen for the rest of the lesson is made of blocks shaped like it.
| Field | What it is telling you |
|---|---|
minion1: | Whose result this is |
ID | The label from the state file, web_ready |
Function | The state function that ran, file.managed |
Name | What it acted on, /tmp/web_deployed |
Result | True for success, False for failure |
Comment | The human sentence explaining the result |
Duration | How long that one piece of work took |
Changes | What is different now, empty when nothing changed |
Changes here is not empty. Under it, diff: New file means the file did not exist and now it does. The summary agrees: one state succeeded and one of them changed something.
Now the point of the exercise. That was one machine and one state. To deploy both tiers in order you would type this, wait, read the result, decide it was good, then type the next one. The order was real, but it lived in your hands. Nothing on this master knows it was supposed to happen that way, and nothing would stop the next person doing it backwards.
Salt started with one direction of travel: the master publishes, the minions execute. That is why salt '*' something is the shape of almost everything in this track.
But some work is not about a minion at all. Reading the master's own configuration, listing which keys are accepted, asking which minions are up: those are questions about the master. Salt grew a second command for that class of work, salt-run, and the things it runs are called runners.
Once runners existed, orchestration was a natural fit. A plan across machines is not work for any one minion, it is work for the thing that can see all of them. So orchestration was written as one more runner, and it still answers to a shorter name, state.orch, alongside the full state.orchestrate.
Carry this sentence: a state file is what one machine should look like, and an orchestration is what the fleet should do next.
Now build the thing. Write it with a heredoc, the shell's way of sending several lines into a file in one go. cat > path << 'EOF' opens the file and everything goes in until a line reading exactly EOF. Quoting 'EOF' stops the shell touching anything inside.
No sudo here. You own this directory, as the owner column proved earlier.
cat > /srv/salt/orch/deploy.sls << 'EOF'
deploy_web:
salt.state:
- tgt: 'minion1'
- sls:
- webserver
deploy_db:
salt.state:
- tgt: 'minion2'
- sls:
- database
- require:
- salt: deploy_web
confirm_db:
salt.function:
- name: test.ping
- tgt: 'minion2'
- require:
- salt: deploy_db
EOFRead it back before you go on. Never trust a heredoc you have not printed.
prompt: saltops@saltmaster:~$ answer: cat /srv/salt/orch/deploy.sls output: deploy_web: salt.state:
deploy_db: salt.state:
confirm_db: salt.function:
hint: Print the file you just wrote: cat /srv/salt/orch/deploy.sls
Three blocks, and each one is a step in the plan.
deploy_web uses salt.state. Its tgt is the target expression, exactly what you would type after salt on the command line. Its sls is the state to apply, given as a list, which is why - webserver sits on its own indented line underneath. It has no require, so it is free to start immediately.
deploy_db is the same function pointed at the other machine and the other state, plus the line that makes this a plan instead of a list: - require: followed by - salt: deploy_web. Read that as this step waits for the step called deploy_web.
Look closely at that requisite. The word is salt, not state and not file. In an ordinary state file you wait on a file state with - file: some_id. In an orchestration every step is a salt.something step, so the requisite type is always salt and the value is the ID of the step you are waiting for.
confirm_db uses the other function. Where salt.state takes an sls, salt.function takes a - name:, which is the execution function to run, here test.ping. It waits on deploy_db.
So the chain is written down: web, then db, then the check. Not because of where the blocks sit in the file, but because each one names the one before it.
You already applied webserver to minion1 by hand a few steps ago, so /tmp/web_deployed is already there with the right contents. The plan you just wrote starts by applying that same state again.
>>> It succeeds with nothing to report. That is idempotence, and it is what makes a plan safe to run twice. If you picked the first option, an existing correct file is the goal, not a collision. file.managed fails when it cannot make the file right, never because it already is. If you picked the third, a change means something on the machine is different than it was a moment ago. Nothing is different, so the changes field has nothing to hold. Watch the Comment line on the next screen.
salt-run for a runner, state.orchestrate for the runner you want, then the file, named the way Salt names state files: dotted, relative to /srv/salt, without the .sls. Your file is /srv/salt/orch/deploy.sls, so it is orch.deploy.
sudo salt-run state.orchestrate orch.deploy
Three steps on two machines produce a long screen. The block below is the top of it, down through the middle of the second step.
prompt: saltops@saltmaster:~$ answer: sudo salt-run state.orchestrate orch.deploy ||| sudo salt-run state.orch orch.deploy output: saltmaster_master: ---------- ID: deploy_web Function: salt.state Result: True Comment: States ran successfully. No changes made to minion1. Started: 23:34:46.744241 Duration: 408.24 ms Changes: ---------- ID: deploy_db Function: salt.state Result: True Comment: States ran successfully. Updating minion2. Started: 23:34:47.152695 Duration: 378.545 ms Changes: minion2: ---------- ID: db_ready Function: file.managed Name: /tmp/db_deployed Result: True Comment: File /tmp/db_deployed updated hint: A runner, the orchestrate runner, and the dotted path to your file: sudo salt-run state.orchestrate orch.deploy
Start with the first line, because it is the one that catches people. saltmaster_master: is where a minion name usually goes. This return belongs to the master, because the thing that ran was a runner and runners run on the master. The minions did work, but the plan was not theirs.
Now deploy_web. Function: salt.state, Result: True, and the comment: States ran successfully. No changes made to minion1. There is your prediction, in Salt's own words. The state ran, the machine already matched, nothing changed, and Changes below it is empty.
Then deploy_db, whose comment reads differently: Updating minion2. Under it the Changes field is not empty, and what is inside it is the interesting part. Indented under minion2: is a complete state result block, with its own ID: db_ready, its own Function: file.managed, its own Name, Result and Comment. That is the minion's state return, nested inside the master's step return.
Read the indentation as a sentence: the master ran a step, and the step's result is what a minion did.
The best test of a deployment plan is running it again. Type the same command a second time.
sudo salt-run state.orchestrate orch.deploy
This time the block below picks up at the end of the second step, so the first four lines are the last four of deploy_db, and it runs to the bottom of the screen.
prompt: saltops@saltmaster:~$ answer: sudo salt-run state.orchestrate orch.deploy ||| sudo salt-run state.orch orch.deploy output: Comment: States ran successfully. No changes made to minion2. Started: 23:34:49.237859 Duration: 342.766 ms Changes: ---------- ID: confirm_db Function: salt.function Name: test.ping Result: True Comment: Function ran successfully. Function test.ping ran on minion2. Started: 23:34:49.580762 Duration: 162.606 ms Changes: ---------- ret: ---------- minion2: True
Summary for saltmaster_master ------------ Succeeded: 3 (changed=1) Failed: 0 ------------ Total states run: 3 Total run time: 870.259 ms hint: The same runner command as before, typed again.
Three things on this screen, in order.
The database step went quiet. No changes made to minion2. Last time it said Updating minion2. Nothing about the plan changed, the machine did. The file is already right, so the state has nothing to do, exactly as deploy_web reported on the first run. A plan you can run twice without fear is a plan you can run when you are not sure whether it already ran.
The third step finally shows itself. Function: salt.function, Name: test.ping, and under Changes a ret: block holding minion2: True. That is the ordinary return of test.ping, carried back inside the step. Where salt.state nested a whole state result, salt.function nests the function's return value.
The summary counts the master's steps, not the minions' work. Total states run: 3 is your three steps. Succeeded: 3 (changed=1) says all three worked and one of them had something to report, which on this screen is the only step with anything under Changes. Failed: 0, and the whole plan finished in under a second.
The steps ran in the order you wanted. That could be a coincidence of how the file is laid out, so go and look at what enforces it.
grep -B 1 -A 2 'require' /srv/salt/orch/deploy.sls
grep prints matching lines. -B 1 adds one line of context before each match and -A 2 adds two after, so you see which step each requirement belongs to and what it waits on. When grep prints more than one group of context lines, it separates the groups with a line reading --.
prompt: saltops@saltmaster:~$ answer: grep -B 1 -A 2 'require' /srv/salt/orch/deploy.sls ||| grep -B 1 -A 2 require /srv/salt/orch/deploy.sls ||| grep -A 2 -B 1 'require' /srv/salt/orch/deploy.sls output:
--
Practice Orchestration in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.