Learn › Salt Configuration Management › Salt Fundamentals
Install salt-master, start the service, and accept your first minion key.
On the network behind this terminal there is a machine named minion1. It is awake. Every few seconds it looks for a Salt master, fails to find one, and tries again. It has been doing that on a loop since this lab booted.
Your box is the machine it is looking for. Your box is not a Salt master yet, so nothing answers. And here is the part that surprises people: once you do put a master on this box, minion1 still will not do a thing you ask. There is a second gate after the connection, and that gate is the reason Salt is safe to run on a network you do not fully control.
This lesson closes both gaps. You will end with a running master, two open ports, an accepted key, and a machine that answers when you call it by name.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output shown was captured from a real Salt master with one real minion attached, which is exactly the machine your lab boots. The lab user is saltops, the master is named saltmaster, and there is exactly one minion, named minion1.
Salt is a tool for managing many servers from one place. Imagine you run 50 web servers. Without Salt you log into each one, type the same command 50 times, and hope you did not fat-finger server 37. Salt lets you type one command on one machine and reach all 50 at once.
It has two halves, and the names carry the whole mental model:
The master publishes commands. The minions carry them out and send results back. That traffic rides two TCP ports: 4505, the publish port where the master broadcasts, and 4506, the return port where minions send results and files back.
Notice the direction. Minions dial out to the master. The master never dials in to them. That is why a minion behind a firewall still works, and it is also why a minion can start knocking long before anyone on the master has agreed to listen.
Thomas S. Hatch released Salt in 2011. The tools of the day opened a fresh SSH connection to every server in turn, which is fine for ten machines and painful for a thousand. He wanted answers from a whole fleet in under a second, so he built Salt on a persistent message bus instead. Every minion holds an open connection to the master and listens. Publishing a command to one minion costs about the same as publishing it to five hundred.
That always-connected design is what makes the key system necessary. If any machine on the network can open a connection and subscribe to your command bus, then any machine on the network can hear what you tell your production servers to do. So Salt makes the connection cheap and makes the trust expensive. Connecting is automatic. Being trusted is a decision a human makes.
This machine ships without the master software on purpose. Installing it is the first real task, and there is a helper on the box that does it cleanly:
sudo install-salt-masterThe helper wraps the official Salt bootstrap script with two flags: -M, which means install the master, and -X, which means do not start any daemons yet. Starting the service is the next step and you will do it yourself.
It prints a stream of progress lines from the bootstrap script and takes a minute or two on a fresh box. When it finishes it prints a short Done message naming the command to run next. Installing gives you two things: the salt-master daemon that listens for minions, and the salt command-line tool you type to send work out.
Reach for the helper rather than apt-get. On this lab box the sudo rules grant passwordless root for a short list of commands: the helper, systemctl, salt, salt-key, salt-run and salt-call. Package tools are not on that list, so sudo apt-get install salt-master will not go through.
Installing a package does not necessarily start it, and here the bootstrap was told explicitly not to. A service is a program that runs in the background. The salt-master service is the thing that holds ports 4505 and 4506 open and waits for minions.
Start it, and set it to come back on its own after a reboot:
sudo systemctl start salt-master
sudo systemctl enable salt-mastersystemctl start prints nothing at all when it works. Silence is success, which is unnerving the first time. So ask a direct question instead of trusting the silence. systemctl is-active answers with a single word and nothing else.
Before you run it, decide what a healthy answer looks like. One word, not a paragraph. Ask the service how it is doing:
systemctl is-active salt-master
prompt: saltops@saltmaster:~$ answer: systemctl is-active salt-master ||| sudo systemctl is-active salt-master output: active hint: The one-word health question is systemctl is-active, then the service name: systemctl is-active salt-master
One word: active. The service is running right now. That single word is the whole answer, which is why this is the fastest service check there is. Any other word means you do not have a master. inactive means it was never started or it stopped cleanly. failed means it tried and died. If you see either one, run sudo systemctl start salt-master again and read what it prints, because a failing start reports the reason on the spot. This first check is also the first thing your lab grades.
A running service and a listening socket are not the same claim. The service can be up while the ports are bound to the wrong address, or not bound at all. ss is the tool that shows sockets, and four flags make it readable:
-t shows TCP only.-l shows listening sockets only.-n prints port numbers instead of guessing at service names.-p shows which process owns each socket.That still prints every listening socket on the box, so pipe it through grep 450 to keep only the Salt ports.
Before you run it, commit to a number: how many lines should come back? Check the ports:
ss -tlnp | grep 450
prompt: saltops@saltmaster:~$ answer: ss -tlnp | grep 450 output: LISTEN 0 1000 0.0.0.0:4506 0.0.0.0:* users:(("/opt/saltstack/",pid=1874,fd=42)) LISTEN 0 1000 0.0.0.0:4505 0.0.0.0:* users:(("/opt/saltstack/",pid=1867,fd=27)) hint: Socket statistics, TCP, listening, numeric, with process: ss -tlnp | grep 450
Two rows, both starting with LISTEN. That is the proof the master is reachable. Read each row left to right. LISTEN is the socket state. 0.0.0.0:4506 and 0.0.0.0:4505 are the local address and port, and 0.0.0.0 means every network interface on this box, not just loopback. A minion on another machine can therefore reach it.
Two details worth noticing. The 4506 row printed first even though 4505 is the lower number, because ss prints sockets in the order the kernel hands them over, not sorted. And the process column reads /opt/saltstack/ rather than a tidy salt-master, because this package installs into /opt/saltstack and the kernel keeps only the first 15 characters of a process name. If your screen shows the two LISTEN rows with no users: column at all, that is a privilege detail of ss -p, not a Salt fault. The LISTEN rows are what the lab grades.
Your master is up and both ports are open, so minion1 has stopped failing and found it. Before you touch the keyboard again, take a position. This one question is the entire security model of Salt.
>>> Nothing runs. Being connected and being trusted are two different states, and Salt keeps them apart on purpose. When a minion first reaches a master it sends its public key, one half of a cryptographic pair that proves it is the same machine next time. The master files that key under Unaccepted and does nothing else with it. Until a human accepts that key, commands aimed at that minion go nowhere. If you picked the first answer, this is the single most common wrong assumption about Salt: connection is not authorization. If you picked the third, note that the master never refuses and never hangs up. It keeps the key pending and the minion keeps retrying, patiently, for as long as it takes you to look.
The command that shows you pending keys is salt-key. Run it with -L (the long form is --list-all) and it prints every key the master knows about, sorted into four buckets: Accepted, Denied, Unaccepted, and Rejected.
Two of those buckets matter today. Unaccepted is the waiting room: a minion said hello and nobody has decided about it yet. Accepted is under management: the master will publish commands to it.
You have accepted nothing yet, so predict the screen before you run it. How many of the four buckets will have a name under them? List the keys:
sudo salt-key -L
prompt: saltops@saltmaster:~$ answer: sudo salt-key -L ||| sudo salt-key --list-all output: Accepted Keys: Denied Keys: Unaccepted Keys: minion1 Rejected Keys: hint: The key manager is salt-key, and the flag that lists all four buckets is -L: sudo salt-key -L
Four headings, and only one of them has anything under it. Accepted Keys: is empty, which is exactly why your commands would land nowhere. minion1 sits under Unaccepted Keys:, heard, filed, ignored. Denied Keys: and Rejected Keys: are empty too, and you want them that way. Denied means the master saw a key that conflicts with one it already holds under that name. Rejected means a human said no on purpose. Read this screen top to bottom every time you wonder why a minion is silent, because the bucket a name sits in is the answer. Getting a key listed here is the third thing your lab grades.
If minion1 is missing from every bucket, it never reached the master at all. That is a network or service problem, not a key problem, and no amount of salt-key will fix it. Wait ten seconds and list again first, because minions retry on their own schedule and the master may have only just come up.
Accepting is one command. -a takes a minion ID and accepts that one key. -y answers the confirmation prompt for you so it does not stop and wait. There is also a capital -A, which accepts every key currently pending at once, handy in a lab and reckless in production where you may not know who is knocking.
You know this machine because you booted it, so accept it by name. Watch the shape of what comes back: salt-key states what it is about to do first, then reports what it did.
sudo salt-key -a minion1 -y
prompt: saltops@saltmaster:~$ answer: sudo salt-key -a minion1 -y ||| sudo salt-key -a minion1 --yes output: The following keys are going to be accepted: Unaccepted Keys: minion1 Key for minion minion1 accepted. hint: Same tool as before, lowercase -a with the minion ID, and -y to skip the prompt: sudo salt-key -a minion1 -y
Read it as two halves. The top half, The following keys are going to be accepted: followed by the Unaccepted bucket, is the confirmation you would have been asked to approve. -y answered it for you, which is why it scrolls past instead of waiting. The bottom half is the receipt: Key for minion minion1 accepted. One receipt line per key accepted. If a name you expected has no receipt line, it was not pending when you ran the command. Nothing here says the minion is working. It says only that the master now trusts it.
This is the step where the lesson lands. You ran salt-key -L a minute ago and you are about to run the identical command again. Nothing about the command changes. Everything about the answer does.
Before you run it, picture the four headings and decide where the name sits now. List the keys again:
sudo salt-key -L
prompt: saltops@saltmaster:~$ answer: sudo salt-key -L ||| sudo salt-key --list-all output: Accepted Keys: minion1 Denied Keys: Unaccepted Keys: Rejected Keys: hint: You are re-running the list command from earlier, unchanged: sudo salt-key -L
The name moved up. minion1 now sits under Accepted Keys:, and Unaccepted Keys: is bare. The four headings are identical to the first run, in the same order, which is the point. salt-key -L is not a status report you skim. It is a two-column question: name, and bucket. Same command, same headings, different bucket, different world. Anything under Accepted will receive what you publish. Anything under Unaccepted will not. That one fact explains most of the moments where Salt seems to be ignoring you, and an accepted key is the fourth thing your lab grades.
Stop for a second and count what changed. Twenty minutes ago this box had no master software on it. It now runs the salt-master service, holds two ports open to the network, and keeps a signed record of exactly one machine it is willing to command.
That is the hard part of Salt and you are already through it. Everything else in this track, remote commands, grains, states, pillar, orchestration, rides on the two things you just built: a listening master and an accepted key. The rest of this lesson is proving it works and learning to read what comes back.
Trust is recorded on the master. The minion, though, has been talking to a master that did not trust it, and it does not learn about the change instantly.
>>> It may not answer yet. Acceptance changes the master's records instantly, but the minion has to reconnect and re-authenticate before it can receive published work. That reconnect takes a few seconds, sometimes longer on a busy box, and a command published inside that window can come back with no answer from a minion that is perfectly healthy. If you picked the second answer, resist that instinct: re-accepting an already accepted key fixes nothing and teaches you nothing. If you picked the third, that is the most reasonable wrong answer here, because plenty of tools do queue. Salt publishing does not. A published job goes out to whoever is listening at that instant, and a minion that was not listening simply misses it.
Here is that exact window, captured on this fixture seconds after the key was accepted, with a command aimed at every minion:
ERROR: Minions returned with non-zero exit code
minion1:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20260731050048327226Nothing is broken here. Minion did not return. [No response] means the master published the job and nobody answered before the timeout expired. Salt then hands you the job ID, that long number, so you can look up any late results with salt-run jobs.lookup_jid. Every published job gets an ID like that, and the results are stored on the master whether or not you were still watching.
The fix for this particular case is to wait a few seconds and run the command again. Learn the shape of this screen now, because you will meet it again for real reasons: a minion that is powered off, a service that died, a firewall that dropped port 4505.
The traditional first command against a new minion is test.ping. The name is borrowed from a much older tool, and the borrowing misleads almost everyone.
>>> A tiny job over the Salt bus. test.ping asks the minion process, are you there and are you listening to me, and a healthy minion returns True. That makes it a far better test than it first appears. An ICMP ping proves only that a machine has a working network stack. test.ping proves the salt-minion service is running, its key is accepted, and the full request and response path works in both directions. If you picked ICMP, notice the consequence: a machine can answer an ordinary ping while its salt-minion service is stone dead. If you picked SSH, that is how the previous generation of tools worked, and moving off it is the reason Salt exists.
Every Salt command has the same three-part shape:
sudo salt '<target>' <module>.<function>
The target picks which minions run it. A minion ID like 'minion1' picks that one machine. A '*' picks every accepted minion. The quotes are not decoration: they stop your shell from expanding * into filenames before Salt ever sees it, so quote the target every time and it will never bite you.
The module.function half is what to run. Salt groups its built-in abilities into modules, and test is the module for checking that the plumbing works. ping is its simplest function. The test module holds a few more small probes, including one that reports the Salt version running on the minion.
Call your one minion by name:
sudo salt 'minion1' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' test.ping ||| sudo salt "minion1" test.ping ||| sudo salt minion1 test.ping output: minion1: True hint: Target the minion by name in quotes, then the module and function: sudo salt 'minion1' test.ping
True, and you now command a machine you never logged into. This is the standard Salt return shape and you will read thousands of these: the minion ID on its own line ending in a colon, then its result indented four spaces underneath. One block per minion that answered.
That indentation is not cosmetic. Salt is telling you which machine said what, and once you target twenty minions the indented block is the only thing keeping the answers apart. Read the ID first, then the value under it. It is also the one shape that separates a return from a key listing: salt-key -L prints key names flush left under their heading, while a salt return value is indented four spaces under the minion ID. A True here is the fifth and last thing your lab grades, and it is the moment the fleet bec
Practice Meet the Salt Master in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.