Learn › Salt Configuration Management › Multi-Minion Orchestration
Create reactor SLS files that respond automatically to events.
In the last lesson you fired events onto the bus and watched them go past. Every one of them arrived, and every one of them was ignored. You were the listener, and when you closed the listener the events kept arriving with nobody reading them.
There is a directory on this box called /srv/reactor. It is empty. Nothing in the master's configuration mentions it. It was created before you logged in and, as things stand, the master has no idea it exists.
By the end of this lesson an event fired from a minion will cause a file to appear on that minion, with no human in between. You will not run the command that creates the file. You will fire an event and the master will run it for you, because you will have told it which events to care about and what to do when they arrive.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do. Your own timestamps and file sizes will be your own. The lab user is saltops, the master is named saltmaster, and there is one minion called minion1. Its key is already accepted for you. Your progress in the lab is tracked automatically, so type commands naturally.
Two directories matter today. /srv/salt is where your state files have always lived. /srv/reactor is new, and it is where the work a reactor performs will be described.
Print both at once. The -l gives you the long listing, and naming two directories makes ls label each one before its contents.
ls -l /srv/reactor/ /srv/salt/
prompt: saltops@saltmaster:~$ answer: ls -l /srv/reactor/ /srv/salt/ ||| ls -l /srv/reactor /srv/salt ||| ls -l /srv/salt/ /srv/reactor/ output: /srv/reactor/: total 0
/srv/salt/: total 4 -rw-r--r-- 1 saltops saltops 89 Aug 4 20:38 motd.sls hint: The long listing of two directories at once: ls -l /srv/reactor/ /srv/salt/
/srv/reactor/ says total 0. That is not an error, it is an empty directory, and it is empty because writing what goes in it is your job in this lesson.
/srv/salt/ has one file, motd.sls, 89 bytes, owned by saltops. That is an ordinary state file and it is there so you have something real to point at later.
Notice the ownership on that line: saltops saltops. The init handed you these directories, which is why you will write into /srv/reactor with a plain redirect and no sudo at all.
The master reads every .conf file in /etc/salt/master.d/. That is a drop-in directory: instead of one enormous config file, you add small files and the master merges them at startup.
Before you look, commit to a number. How many files do you expect to find in there right now?
ls -l /etc/salt/master.d/
prompt: saltops@saltmaster:~$ answer: ls -l /etc/salt/master.d/ ||| ls -l /etc/salt/master.d output: total 4 -rw-r--r-- 1 root root 113 Aug 4 20:36 lab.conf hint: The long listing of the drop-in directory: ls -l /etc/salt/master.d/
One file, lab.conf, and read the owner column: root root. That is the difference between this directory and /srv/reactor. You own the reactor folder, root owns the master's config, and that single fact decides which of the two writes later in this lesson needs sudo.
One file could still contain anything. Search the whole directory for the word instead of trusting the file count.
grep -rs 'reactor' /etc/salt/master.d/ || echo '(none)'
grep -r searches recursively through a directory. -s silences the complaints grep would otherwise make about anything it cannot read. The || echo '(none)' is plain shell: grep exits non-zero when it finds nothing, so this prints a word instead of leaving you looking at a blank line wondering whether the command ran.
prompt: saltops@saltmaster:~$ answer: grep -rs 'reactor' /etc/salt/master.d/ || echo '(none)' ||| grep -rs reactor /etc/salt/master.d/ || echo '(none)' ||| grep -rs 'reactor' /etc/salt/master.d || echo '(none)' output: (none) hint: Recursive silent grep for the word, with a fallback echo so an empty result is visible: grep -rs 'reactor' /etc/salt/master.d/ || echo '(none)'
(none). The master has never been told to react to anything, which is the honest starting position for this lesson and the thing you are about to change.
This is also a habit worth taking with you. When you are about to add configuration, look first and confirm it is not already there. Salt merges drop-in files, and a second reactor: key in a second file does not add to the first one, it replaces it.
A reactor is two things joined together: an event tag to watch for, and a file describing work to do when a matching event arrives.
That is the whole idea. Everything confusing about reactors comes from expecting a third thing that is not there.
Three consequences follow, and each one costs people time when they discover it the hard way.
Those three are the exact order of the next three steps, and they are the three things the lab grades you on before you ever fire an event.
Before event-driven automation, keeping a fleet correct meant asking. A script on a timer would wake up, connect to each machine, look at it, and fix whatever had drifted. That is polling, and it has two costs that get worse as the fleet grows: you pay for every check whether anything changed or not, and you find out about a change only on the next tick.
Salt already had a message bus carrying everything the fleet did, which you watched in the last lesson. Once that bus exists, the reactor is a small idea: stop asking, and let the thing that already knows tell you.
Carry this sentence: a reactor turns a question you were asking on a timer into an answer that arrives on its own.
You are about to write a reactor file into /srv/reactor. Take a position before you do.
>>> No. /srv/reactor is a directory you chose, not a directory Salt watches. Salt has no rule that says files in a folder called reactor are reactors. If you picked the first answer, that is the most common wrong model of this whole subject, and the empty grep you just ran is the evidence against it: the master's config is the only place a binding can live. If you picked the third, filenames carry no meaning here either. You will bind lc/deploy/done to a file called notify.sls, and the mismatch is fine because the binding is explicit.
This file describes what should happen. Write it with a heredoc, which is the shell's way of sending several lines into a file in one go. cat > path << 'EOF' opens the file, everything you type goes in until a line reading exactly EOF, and quoting 'EOF' stops the shell from touching anything inside.
cat > /srv/reactor/notify.sls << 'EOF'
write_motd:
local.cmd.run:
- tgt: 'minion1'
- arg:
- echo 'Deployed by reactor' > /etc/motd_reactor
EOFRead it back before you go on. Never trust a heredoc you have not printed.
prompt: saltops@saltmaster:~$ answer: cat /srv/reactor/notify.sls ||| cat /srv/reactor/notify.sls output: write_motd: local.cmd.run:
hint: Print the file you just wrote: cat /srv/reactor/notify.sls
Five lines, and every one of them is doing a job.
write_motd: at the left margin is the ID. It is your label for this block of work and it can be anything. It is not the event tag and it is not a filename.local.cmd.run: is the work. Read it as two halves. local means run this through the master's own client, the same path a salt command on the command line takes. cmd.run is the execution module and function you already know.- tgt: 'minion1' is the target, exactly as if you had typed salt 'minion1' yourself.- arg: opens the argument list, and the line under it is the single argument passed to cmd.run: an ordinary shell command that writes a line into a file.The word to hold on to is local. It is doing more work than it looks like, and the last challenge in this lesson is entirely about what happens when you forget it.
Now the binding, and this one goes in the master's configuration, which root owns. That is why this write uses sudo tee instead of a plain redirect: sudo cat > file would not work, because the shell opens the file as you before sudo ever runs. tee is a program that writes what it reads to a file, so it can be the thing that runs as root.
sudo tee /etc/salt/master.d/reactor.conf > /dev/null << 'EOF'
reactor:
- 'lc/deploy/done':
- /srv/reactor/notify.sls
EOFThe > /dev/null is there because tee also prints what it writes, and you do not need to see it twice.
prompt: saltops@saltmaster:~$ answer: cat /etc/salt/master.d/reactor.conf ||| sudo cat /etc/salt/master.d/reactor.conf output: reactor:
hint: Print the config file you just wrote: cat /etc/salt/master.d/reactor.conf
Three lines, and the shape is worth reading slowly because it catches people.
reactor: is a list. Each entry in that list is a single-key mapping: the key is the event tag, and the value is the list of files to run. That is why there is a dash in front of the tag and another dash in front of the path, at different indents.
| Line | Indent | Meaning |
|---|---|---|
reactor: | 0 spaces | The setting itself |
- 'lc/deploy/done': | 2 spaces, list item | An event tag to watch for |
- /srv/reactor/notify.sls | 4 spaces, list item | A file to run when it matches |
The tag is quoted because it contains slashes. The path is a full path, because the master resolves it as a real file on disk and not as a state name.
The file is written and it is in the directory the master reads. Take a position on what that means.
>>> No. A drop-in directory describes where the master looks, not how often. Configuration is read once, at startup. If you picked the first answer, that is the assumption that produces the most frustrating half hour in this whole subject: a correct file, a correct binding, a fired event, and nothing happening, because the process making the decision is still running on the configuration it read before you edited anything. If you picked the third, the minions are not part of this at all. The reactor mapping is master-side, and no minion needs to know it exists.
Restarting the master drops every minion connection and they reconnect on their own. It takes a few seconds and it is silent when it works.
sudo systemctl restart salt-masterGive it a moment, then confirm the minion is back before you trust anything you see next. A reactor that appears not to fire because the minion had not reconnected yet is a fault you can spend a long time not finding.
sudo salt 'minion1' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' test.ping ||| sudo salt "minion1" test.ping ||| sudo salt minion1 test.ping output: minion1: True hint: Ping the one minion by name: sudo salt 'minion1' test.ping
True from minion1. The master came back up and the minion found it again without anyone touching the minion.
If you get No response or Minion did not return, the answer is almost always that you were faster than the reconnect. Wait a few seconds and ask again before you go looking for a real fault.
You have read your own file back, but that only proves what you wrote. This asks the master what it believes, which is a different question and the more useful one.
sudo salt-run config.get reactor
salt-run runs a runner, which is a command that executes on the master itself rather than being sent to a minion. config.get asks the master for one of its own settings.
prompt: saltops@saltmaster:~$ answer: sudo salt-run config.get reactor ||| sudo salt-run config.get 'reactor' output:
| _ |
|---|
---------- lc/deploy/done:
hint: A runner asks the master about itself: sudo salt-run config.get reactor
This is the moment the two halves meet. Your file said one thing; the master now says the same thing back.
|_ is how Salt prints a list item in this output format, and the ---------- under it opens the mapping inside that item. Then lc/deploy/done: is the tag and the indented - /srv/reactor/notify.sls is its file list.
If this had come back empty after a restart, the fault would be in the YAML of reactor.conf rather than anywhere else, and you would go back and compare your indentation against the table in the previous reveal.
Everything is in place. Fire the exact tag the binding names, from the minion, the same way you fired events in the last lesson.
sudo salt 'minion1' event.send 'lc/deploy/done'
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' event.send 'lc/deploy/done' ||| sudo salt "minion1" event.send "lc/deploy/done" ||| sudo salt minion1 event.send 'lc/deploy/done' output: minion1: True hint: The same event.send you used in the event bus lesson, with the tag the binding names: sudo salt 'minion1' event.send 'lc/deploy/done'
True, and that is all you get. Read what this return means carefully, because it is easy to over-read.
True means the event was sent. It says nothing at all about the reactor. The minion fired the event onto the bus and reported success at doing that, and the story ends there as far as this command is concerned. Whether anything was listening, whether a reactor matched, whether the work succeeded or failed, none of it appears here.
That separation is deliberate and it is the point of an event system. The thing firing the event does not know or care who reacts. It also means you cannot tell from this screen whether your reactor worked, which is why the next step exists.
The reactor was supposed to write /etc/motd_reactor on minion1. Read that file from here, without logging into the minion.
sudo salt 'minion1' cmd.run 'cat /etc/motd_reactor'
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' cmd.run 'cat /etc/motd_reactor' ||| sudo salt "minion1" cmd.run "cat /etc/motd_reactor" ||| sudo salt 'minion1' cmd.run "cat /etc/motd_reactor" ||| sudo salt minion1 cmd.run 'cat /etc/motd_reactor' output: minion1: Deployed by reactor hint: Run cat on the minion and read the output back: sudo salt 'minion1' cmd.run 'cat /etc/motd_reactor'
Deployed by reactor, on a machine you did not touch.
Trace the whole path once, because this is the thing you built. You ran event.send on minion1. The minion put an event on the bus with the tag lc/deploy/done. The master, which is always watching that bus, matched the tag against its reactor mapping and found /srv/reactor/notify.sls. It read that file, saw local.cmd.run targeting minion1, and issued that command. The minion ran it and the file appeared.
Nobody typed echo 'Deployed by reactor' > /etc/motd_reactor. You typed a file, a binding, and a tag, and the fleet did the rest.
Stop and take stock of what changed on this box.
You began with an empty directory and a master whose configuration did not contain the word reactor. You now have a master that watches for one specific event tag and performs a specific piece of work whenever it sees it, indefinitely, without anyone being logged in.
That is the whole shape of event-driven automation, and everything more elaborate is this same three-part structure with more tags, more files, and more careful work inside them.
The rest of this lesson is about the two ways it silently does nothing.
>>> Nothing at all. The event is real, it is on the bus, a listener would see it, and no reactor matched so no work was done. If you picked the first answer, an unmatched event is not an error condition. Most events on a busy master match nothing, including all the job events you saw in the previous lesson. If you picked the third, this one is worth being firm about: matching is not fuzzy. A tag either matches a binding or it does not, and a reactor that nearly matches is a reactor that does not run.
Scaffolding off. No command is printed from here on.
You committed to an answer. Now watch it happen. Fire an event with the tag lc/nothing/listens, which appears in no binding on this master. Use the same command you used to fire the bound one.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' event.send 'lc/nothing/listens' ||| sudo salt "minion1" event.send "lc/nothing/listens" ||| sudo salt minion1 event.send 'lc/nothing/listens' output: minion1: True hint: The same firing command as before, with a tag that appears in no binding.
True again, and it is the identical screen you got from the tag that did work.
This is the honest and slightly uncomfortable part of event-driven systems. The success of firing tells you nothing about the success of reacting, so the two screens are indistinguishable. Whether your automatio
Practice Reactors in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.