Learn › Salt Configuration Management › Multi-Minion Orchestration
Monitor and fire events on the Salt event bus.
This box looks idle. One minion is connected, no deployment is running, and nobody else is logged in.
It is also, right now, carrying a stream of messages between its own parts. Salt does not work by opening a connection to a machine and waiting for the reply. Every piece of Salt announces what it is doing on a shared channel, and the other pieces listen. A job is created: announcement. A minion returns a result: announcement. A key is accepted: announcement.
That channel is the event bus, and it has been running since this master started. Nothing has been recording it.
In this lesson you will attach yourself to the bus, send what goes past into a file, then fire two events of your own: one from the minion, one from the master. At the end you will count the lines that landed in the file and compare that number with the number of events you actually asked for. What the comparison shows is the reason the next lesson exists.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do. Your own timestamps, job ids and process ids will be your own. The lab user is saltops, the master is named saltmaster, and there is one minion called minion1. Its key is already accepted for you. Your progress in the lab is tracked automatically, so type commands naturally.
The event bus is not a service of its own. There is no salt-event to start and no separate port you have to open. The bus lives inside the master process, so the question of whether the bus is up is the question of whether salt-master is up.
systemctl is-active asks the service manager for one word about one service, and prints nothing else. It is the fastest health question you can ask a Linux box.
sudo systemctl is-active salt-master
prompt: saltops@saltmaster:~$ answer: sudo systemctl is-active salt-master ||| sudo systemctl is-active 'salt-master' ||| sudo systemctl is-active "salt-master" output: active hint: One word back about one service: sudo systemctl is-active salt-master
active. One word, no banner, no table. That is the whole answer, and it is the answer you want.
Two other words show up in this position often enough to know by sight. inactive means the service is not running and was not asked to be. failed means it tried and stopped, which is the one that sends you to the logs.
Because the bus is part of this process, everything in this lesson depends on that one word. A master that is not active has no bus, and a listener attached to nothing sits there quietly telling you nothing is happening.
test.ping publishes a tiny job to a minion and waits for the answer. It proves the connection, the accepted key and the round trip in one line, which is why it is the first thing anyone runs on a master.
Quote the target so the shell hands the name to Salt untouched, then the module and function.
sudo salt 'minion1' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' test.ping ||| sudo salt "minion1" test.ping ||| sudo salt minion1 test.ping output: minion1: True hint: Quote the target, then the module and function: sudo salt 'minion1' test.ping
One block. The minion id, a colon, and the result indented under it. True means the minion is there and answering.
Now think about what that single word cost. The master had to create a job, publish it, wait for a machine that was not asked directly, and collect a return. You were shown the last line of that story and nothing else.
Every one of those steps was announced on the bus while it happened. The salt command you typed is itself a listener: it attaches, waits for the return event it cares about, prints it, and detaches. Everything else it heard, it threw away.
That is the gap this lesson closes. You are about to attach a listener that throws nothing away.
An event is two things joined by a tab character.
custom/deploy is a tag. So is salt/job/20260804223456961340/ret/minion1.The tag says what kind of thing this is. The data says the details. Anything reacting to events matches on the tag first and reads the data second. That is why tags are built out of slash-separated pieces: the pieces let a matcher be as broad or as narrow as it needs.
The bus itself works by publish and subscribe. Nothing addresses an event to anyone. The publisher puts it on the bus and immediately forgets about it. Any number of listeners may be attached, or none at all, and the publisher cannot tell the difference and does not slow down either way.
Three consequences follow, and every one of them costs somebody time when they meet it the hard way.
Salt appeared in 2011, written by Thomas Hatch, and the design decision that made it different was made on day one. The tools of the time managed a fleet by connecting to each machine in turn, usually over SSH, running something, and reading the result. That works, and it stops working somewhere around the point where connecting to everything takes longer than the work does.
Salt turned the shape inside out. Minions connect to the master and stay connected. The master publishes a job once, and every minion that matches the target picks it up at the same moment. The transport underneath is ZeroMQ, a messaging library built for exactly this pattern. The bus you are about to listen to is that same transport, carrying Salt's own announcements alongside the job traffic.
The useful accident is that the announcements were made public. Salt could have kept its internal chatter private. Instead any process on the master with permission may attach and read the lot, which is how one design decision about speed turned into a platform for automation that nobody had to design separately.
Carry this sentence: the bus is a live stream, not a log. Everything surprising about events comes from expecting a recording where there is only a broadcast.
You are about to attach a listener to a master that has been running for a while and has already carried plenty of traffic, including the ping you just ran. Take a position before you attach it.
>>> No. Attaching to a bus is like tuning a radio to a frequency: you hear what is being said from now on. If you picked the first answer, that is the single most common wrong model here, and it matters because it changes what you do when you are investigating a fault. There is nothing to go back and read, so the listener has to be running before the thing you want to see happens. If you picked the third, there is no per-job replay either. A job that returns while nothing is attached returns to the caller and its events pass unheard, exactly like every other event.
salt-run runs a runner, which is a command that executes on the master itself instead of being sent out to minions. The runner you want is state.event, and it does one thing: attach to the bus and print every event that goes past, one per line.
It never exits on its own. Left in the foreground it would hold your terminal for as long as you let it, so send it to the background and give it somewhere to write.
sudo salt-run state.event > ~/events.log 2>&1 &Read that line in three pieces. > ~/events.log sends everything the listener prints into a file in your home directory. 2>&1 sends its error output down the same pipe so a complaint lands in the file rather than on your screen. The & at the end puts the whole thing in the background and hands your prompt straight back.
One question to settle before you look at the result: the listener runs as root, so who is going to own the file?
Now look at the file. ls -l gives you the long listing, and the columns you care about are the owner and the size.
ls -l ~/events.log
prompt: saltops@saltmaster:~$ answer: ls -l ~/events.log ||| ls -l /home/saltops/events.log ||| ls -l $HOME/events.log output: -rw-r--r-- 1 saltops saltops 0 Aug 4 22:34 /home/saltops/events.log hint: The long listing of one file: ls -l ~/events.log
The file exists, and the size column reads 0.
Both halves of that are worth a sentence. The file exists because the redirect created it the instant you pressed Enter, before the listener had produced anything at all. That is what a > redirect does: it opens the file first and hands the open file to the command second.
The size is 0 because nothing has been published since you attached. That is the answer you committed to in the last step, sitting on the screen as a number. A listener on a quiet master writes nothing, and a file of zero bytes is not a broken listener.
Read the owner column too: saltops saltops, your account, even though the process behind it is running as root. The redirect is performed by your shell, as you, before sudo runs anything at all. Your shell opened the file, so your shell's identity is on it. Your own timestamp will differ from the one here.
That file is now the record for everything that follows, and having it non-empty is one of the five things your lab grades.
The listener is attached and the file is empty. The next thing you do is publish exactly one custom event from minion1. Take a position on what lands in that file.
>>> Several. Asking a minion to publish an event is itself a job, and jobs are the thing the bus was built to announce. Your custom event travels in company. If you picked the first answer, you are thinking of the bus as a channel you are using, when it is a channel Salt is already using and has let you join. If you picked the third, nothing is batched or held back. Events are published the moment they happen. That is why a listener has to be running at the time, and why it sees the start of a job before the end of it.
event.send is an execution module function, so it runs on the minion, the same way test.ping and cmd.run do. It takes a tag, and optionally a chunk of data to attach.
The tag is yours to choose. The habit worth forming is a prefix of your own, custom/ or a company name, so that your tags can never collide with the salt/ ones Salt reserves for itself.
The data goes on as a second argument in JSON, quoted so the shell hands it over as one piece.
sudo salt 'minion1' event.send 'custom/deploy' '{"app": "web", "version": "2"}'
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' event.send 'custom/deploy' '{"app": "web", "version": "2"}' ||| sudo salt "minion1" event.send "custom/deploy" '{"app": "web", "version": "2"}' ||| sudo salt minion1 event.send 'custom/deploy' '{"app": "web", "version": "2"}' output: minion1: True hint: The same target-then-function shape as test.ping, with a tag and a JSON payload after it: sudo salt 'minion1' event.send 'custom/deploy' '{"app": "web", "version": "2"}'
True from minion1, and it is the identical shape test.ping gave you.
Read what that True claims, because it is narrower than it looks. It says the minion received the instruction and published the event. It says nothing about whether anything heard it, whether the tag was spelled the way you meant, or whether any consequence followed.
That separation is the point of an event system rather than a flaw in it. The publisher is deliberately ignorant of its audience. It also means this screen can never tell you whether your automation worked, which is why the rest of this lesson is about going and looking.
Firing a custom event from a minion is the second of the five things your lab grades.
Minions are not the only source of events. The master can publish directly, without involving any machine at all, and that is how scripts, timers and outside systems get their news onto the bus.
The function has the same name, but it is reached through salt-run rather than salt, because this time it is a runner executing on the master rather than a job sent to a minion.
Watch the shape of what comes back and compare it with the last screen.
sudo salt-run event.send 'custom/from-master' '{"who": "master"}'
prompt: saltops@saltmaster:~$ answer: sudo salt-run event.send 'custom/from-master' '{"who": "master"}' ||| sudo salt-run event.send "custom/from-master" '{"who": "master"}' ||| sudo salt-run event.send custom/from-master '{"who": "master"}' output: True hint: A runner acts on the master, so it is salt-run, then the same tag and payload: sudo salt-run event.send 'custom/from-master' '{"who": "master"}'
True again, but look at what is missing: there is no minion id above it and no indentation under one.
That is the visible difference between a job and a runner. A job is published to machines, so the return is organised by machine, one block each. A runner ran here, on the master, so there is nobody to attribute the answer to and the answer is a bare value.
You now have two events of your own on the bus, published from two different places by two different mechanisms. Firing one from the master is the fourth of the five things your lab grades.
The listener has been attached the whole time. Two events of yours have gone past. Print the file and read it.
The lines are long and your terminal will wrap them, which is normal and worth knowing before it surprises you. Each line is one event: the tag, a tab, then the JSON.
cat ~/events.log
prompt: saltops@saltmaster:~$ answer: cat ~/events.log ||| cat /home/saltops/events.log ||| cat $HOME/events.log output: salt/job/20260804223456961340/new {"jid": "20260804223456961340", "tgt_type": "glob", "tgt": "minion1", "user": "sudo_saltops", "fun": "event.send", "arg": ["custom/deploy", {"app": "web", "version": "2"}], "minions": ["minion1"], "missing": [], "_stamp": "2026-08-04T22:34:56.963661+00:00"} salt/job/20260804223456961340/publish {"fun": "event.send", "arg": ["custom/deploy", {"app": "web", "version": "2"}], "tgt": "minion1", "jid": "20260804223456961340", "ret": "", "tgt_type": "glob", "user": "sudo_saltops", "_stamp": "2026-08-04T22:34:56.968928+00:00"} custom/deploy {"id": "minion1", "cmd": "_minion_event", "pretag": null, "data": {"__pub_fun": "event.send", "__pub_arg": ["custom/deploy", {"app": "web", "version": "2"}], "__pub_tgt": "minion1", "__pub_jid": "20260804223456961340", "__pub_ret": "", "__pub_tgt_type": "glob", "__pub_user": "sudo_saltops", "__pub_resource_targets": [], "__pub_pure_resource_target": false, "__pub_minion_is_target": true, "app": "web", "version": "2"}, "tag": "custom/deploy", "ts": 1785882897, "_stamp": "2026-08-04T22:34:57.049920+00:00"} salt/job/20260804223456961340/ret/minion1 {"cmd": "_return", "id": "minion1", "success": true, "return": true, "retcode": 0, "jid": "20260804223456961340", "fun": "event.send", "fun_args": ["custom/deploy", {"app": "web", "version": "2"}], "user": "sudo_saltops", "_stamp": "2026-08-04T22:34:57.060867+00:00", "ts": 1785882897} salt/run/20260804223500708197/new {"fun": "runner.event.send", "jid": "20260804223500708197", "user": "salt", "fun_args": ["custom/from-master", {"who": "master"}], "_stamp": "2026-08-04T22:35:00.961994+00:00"} custom/from-master {"who": "master", "_stamp": "2026-08-04T22:35:00.965623+00:00"} salt/run/20260804223500708197/ret {"fun": "runner.event.send", "jid": "20260804223500708197", "user": "salt", "fun_args": ["custom/from-master", {"who": "master", "_stamp": "2026-08-04T22:35:00.965623+00:00"}], "_stamp": "2026-08-04T22:35:00.984296+00:00", "return": true, "success": true} hint: Print the whole file with cat and its path: cat ~/events.log
Take it slowly. This screen is the lesson.
First the shape. Every line is tag, a tab, then a JSON object. That tab is the reason this format is worth putting up with. The tag is always the first tab-separated field, so any tool that splits on tabs can pull the tags out without understanding a word of the JSON. You will use that in a moment.
Now the two events you asked for.
custom/deploy carries "app": "web" and "version": "2", your own data, alongside "cmd": "_minion_event". That key is how the master records that a minion published this, rather than the master. It is also exactly what your lab greps for when it scores the minion event.custom/from-master is almost bare: "who": "master" and a stamp. A runner publishing directly has no job wrapped around the payload, so nothing else is attached.Then the ones you did not ask for.
salt/job/<jid>/new announces that a job now exists. Read its data and you have the whole request: "fun": "event.send", the target, the arguments, and "minions": ["minion1"], the list of machines expected to answer.salt/job/<jid>/publish announces that the job went out over the wire.salt/job/<jid>/ret/minion1 is the return, and the tag itself ends in the machine name. Inside: "retcode": 0 and "success": true.salt/run/<jid>/new and salt/run/<jid>/ret are the same pair for a runner. Their "fun" is runner.event.send, which is the string your lab greps for to score the master event.Two more details worth carrying away. Every single line contains _stamp, a timestamp Salt adds to every event without exception, which is why a file containing _stamp is proof that a listener really ran. And the "user" field differs between the two halves: sudo_saltops on the job you published through sudo salt, and salt on the runner, because the master ran that one as itself.
The JSON is where the detail lives. The tags are where the shape lives, and the shape is easier to see with the detail taken away.
cut -f1 prints the first tab-separated field of every line, which on this file is exactly the tag and nothing else. Send that into sort so the families group together instead of arriving in the order they happened.
Before you press Enter, commit to a number. You published two events
Practice The Event Bus in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.