Learn › Salt Configuration Management › Salt Fundamentals
Execute commands on remote minions using Salt execution modules.
Your minion is connected. Now you want to know something simple: is it alive?
On any other system you would open an SSH session, wait for the login, type a command, read the answer, and log out. Then you would repeat the whole dance for the next machine, and the next.
Salt does not work that way. You type one line on the master, and an answer comes back with the name of the machine that produced it printed above it. No session. No password. No script copied anywhere.
That reply has a shape, and the shape is the point. It is built to survive being one answer out of five hundred. By the end of this lesson you will be reading it the way an operator reads it.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output was captured from a real Salt topology, a Debian 12 master named saltmaster with one minion container called minion1. That is exactly the machine your lab boots. Two of the outputs later on, disk.usage and grains.items, are long enough to fill several screens, so what you see is the top of the real capture, ending exactly where the capture ends.
Remote execution means running a command on other machines from one place, without logging into any of them. It is the first thing Salt does and the foundation for everything after it. You sit at the master. The work happens elsewhere. The results come to you.
Every Salt command you will ever type follows the same three-part shape:
sudo salt '<target>' <module.function> [arguments]minion1 means that one machine. A quoted * means every minion whose key you have accepted.test module holds testing functions. The cmd module runs shell commands. The disk module reports storage. The grains module reports facts about the machine.Wrap the target in quotes every time. Without quotes, your shell sees a * first and tries to expand it into filenames in your current directory, and Salt never receives the star you meant.
Every Salt command in this lab is typed with sudo in front of it. Reading the master's keys and publishing jobs on its bus are root-level actions, and the lab account saltops is granted exactly those four Salt programs through sudo and nothing else. Leave the sudo off and you get a permissions error, not a result.
Salt was written by Thomas Hatch and released in 2011. He was managing servers the SSH way, one connection per machine per command, and it did not scale. So he built something different: the master and every minion hold a persistent, encrypted connection to a shared message bus, and jobs travel over that bus.
That one design choice is why Salt keeps up on a large fleet where the SSH way does not. Nothing is dialed. Nothing is logged into. The connection is already there, waiting.
The two ports from the first lesson, 4505 and 4506, are that bus. Port 4505 is the publish channel: the master announces a job once, and every connected minion hears it at the same moment. Port 4506 is the return channel: each minion sends its own result back independently, on its own schedule.
The simplest function in Salt is test.ping. Before you run it, commit to an answer. Getting this wrong is the single most common misunderstanding on day one.
>>> It is a job on the bus. The name is the trap. If it were an ICMP echo, a good result would only prove the machine has a working network stack. That tells you nothing about whether salt-minion is running, or whether its key was ever accepted. If it were SSH, there would be a login, a key and a shell involved, and Salt uses none of those here. What test.ping proves is much stronger than either, and you are about to read it off the answer itself.
Now run it. Name the machine as the target, then the module and function joined by a dot.
Before you press Enter, decide what a healthy answer looks like.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' test.ping ||| sudo salt "minion1" test.ping ||| sudo salt minion1 test.ping output: minion1: True hint: The three-part shape is target, then module.function: sudo salt 'minion1' test.ping
Two lines, and that is the shape of every Salt result you will ever read. The machine name sits at the left margin with a colon after it. Its result is indented four spaces underneath. When five hundred machines answer, you get five hundred of these little blocks, and the name is how you tell them apart.
True here is not a generic success message. It means the minion process is running, its key is accepted by this master, and it reached back to the master on the return port. A minion that is down does not answer False. It simply does not return, and you are about to see exactly what that looks like.
Naming a machine is the precise form. The other form names none: a quoted * targets every minion whose key this master has accepted, whether that is one machine or a thousand.
The command is identical on a fleet of five hundred. Only the size of the reply changes.
prompt: saltops@saltmaster:~$ answer: sudo salt '*' test.ping ||| sudo salt "*" test.ping output: minion1: True hint: The target for every accepted minion is a quoted star: sudo salt '*' test.ping
One job, one answer, because this master has one accepted key. Nothing about the command grows when the fleet does. That is the whole promise of the message bus: the master publishes the job once, and every minion listening picks it up at the same moment.
Because each minion returns on its own, the blocks arrive in whatever order the machines finish, not in the order you would expect. On a bigger fleet the order can change between two runs of the same command. Read Salt output by the machine NAME, never by position. Any script that assumes the first block belongs to a particular machine will work perfectly until the day it does not.
There is one failure you are very likely to meet in the lab, and it is not a broken lab. It happens when you accept a minion key and immediately ask the minion something. The key is accepted, but the minion has not finished reconnecting to the master yet, so it hears nothing.
This was captured seconds after a real key acceptance:
ERROR: Minions returned with non-zero exit code
minion1:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20260731050048327226Read it carefully, because it is telling the truth in an unfriendly way. Minion did not return. [No response] means the master published the job and nobody answered before it stopped waiting. It does NOT mean the minion said no, and it does not mean the key was rejected.
The cure is to wait a few seconds and run the command again. That long number at the end is the job ID, and Salt is offering you a way to collect the answer later if it turns up after the timeout.
This is why a health check is worth running before anything else. If test.ping does not come back clean, no other command will either, and you will waste time debugging the wrong thing.
So far you have asked Salt about Salt. The cmd module is different: cmd.run takes any shell command you like and runs it. This is the function that makes people nervous the first time, and the nerves are healthy.
>>> On minion1 only, and the numbers belong to minion1. The first option has the mental model backwards: your terminal is where you type, never where the work lands. The third option describes the SSH-and-copy-a-script workflow Salt exists to replace, and nothing is copied anywhere here. The job is published on the bus, the minion runs the command itself, and the minion sends back what the command printed. Change that target to a quoted star and the same command would run on every accepted minion at once, which is the power and the danger in one keystroke.
Name the machine, then cmd.run, then the shell command in quotes. Whatever you put in those quotes is executed on the remote machine, exactly as if you had logged in and typed it.
uptime is a good first choice. It reports how long a machine has been running, so its answer could not possibly have come from anywhere else.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' cmd.run 'uptime' ||| sudo salt "minion1" cmd.run "uptime" ||| sudo salt minion1 cmd.run 'uptime' output: minion1: 05:02:04 up 49 days, 23:52, 0 user, load average: 0.55, 0.39, 0.26 hint: Name the machine, then the module.function, then the shell command in quotes: sudo salt 'minion1' cmd.run 'uptime'
Read the line: 05:02:04 is the clock on minion1, and up 49 days, 23:52 is how long it has been running. Then 0 user is how many people are logged in, and the three load averages cover the last 1, 5 and 15 minutes.
Now compare this result line with the True you saw earlier. This one sits one space deeper. That extra space is not a Salt quirk. It belongs to uptime, whose own output begins with a space. Salt indented the result by four and left the text completely alone.
That is the whole contract of cmd.run: it does not interpret, it does not tidy, it hands you exactly the bytes the remote machine printed. Which is also why it is a last resort. cmd.run returns text a human has to read. Salt's dedicated modules return structured data a script can use, and they behave the same way across different distributions. Reach for a real module when one exists.
The salt-minion process runs as root, so cmd.run runs your command as root on the remote machine. There is no confirmation prompt and no undo. The target selects who does it: swap minion1 for a quoted star and a careless command lands on every accepted minion at the same instant. Read the target before you read the command.
Stop for a second, because you crossed the line that matters.
You ran a shell command on a machine you never logged into. No SSH session, no password prompt, no copied script, no second terminal window. You named a target and a function, and another computer did the work and reported back.
Nothing about that changes when the fleet is five hundred machines instead of one. The target grows. The command does not. That is the entire promise of configuration management, and you have now used it by hand.
cmd.run 'df' would work, and it would hand you a wall of text to squint at. Salt has a disk module instead, and disk.usage is its report. Commit to what comes back before you run it.
>>> A nested structure. The first option describes cmd.run 'df', which is the thing this module exists to replace: forwarded text is fine for a human and useless to a program. The third option is too small to be true, because a machine has more than one mounted filesystem and each one has its own size, used and available figures. disk.usage returns a labelled entry per mount point, which a script can read by key instead of by counting columns. That is the difference between a dedicated module and cmd.run, and it is the reason to prefer modules.
No arguments needed. Name the machine and ask the disk module for its usage report.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' disk.usage ||| sudo salt "minion1" disk.usage ||| sudo salt minion1 disk.usage output: minion1: ---------- /: ---------- 1K-blocks: 73334784 available: 20369244 capacity: 73% filesystem: overlay used: 52965540 /dev: ---------- 1K-blocks: 65536 hint: The module is disk and the function is usage, with no arguments: sudo salt 'minion1' disk.usage
Learn to read this layout once and you can read every structured Salt result. A row of dashes, ----------, opens a group. Indenting one level deeper means being inside that group. So the first ---------- opens the minion's whole report, /: names the first mount point, and its own ---------- opens the numbers for that mount.
Inside the entry for / you get four facts. 1K-blocks: 73334784 is the total size counted in 1-kilobyte blocks, so around 70 gigabytes. available: 20369244 and used: 52965540 are the same units. capacity: 73% is the number a human actually wants.
filesystem: overlay is worth a second look. Your minion runs as a container, and a container's root filesystem is assembled from stacked layers by a driver called overlay. On a plain server that field would name a device such as /dev/sda1 instead. Salt reported the truth about the machine it ran on.
The real output keeps going with an entry for every mounted filesystem, so what you see above stops partway through /dev, exactly where the capture stops. That is normal for this command. The first entry, /, is almost always the one you came for.
Every minion inspects itself when it starts and stores what it finds as grains: key and value facts about the machine. The operating system name, the kernel, the BIOS, the CPU flags, the addresses. Think of grains as the machine's profile card, filled in by the machine.
grains.items dumps the whole card. It is a long answer, and you are about to see why that matters in a moment.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.items ||| sudo salt "minion1" grains.items ||| sudo salt minion1 grains.items output: minion1: ---------- biosreleasedate: 12/12/2023 biosvendor: Dell Inc. biosversion: 2.19.0 boardname: 0599V5 ca_truststore: certifi cpu_flags:
hint: The module is grains and the function is items, with no arguments: sudo salt 'minion1' grains.items
Same layout as disk.usage, because it is the same structure: ---------- opens a group, and each key sits above its indented value. Some values are single, like biosvendor. Some are lists, and a list is printed as - items, which is why cpu_flags runs down the screen.
You are looking at the first thirty lines of a much longer dump, ending exactly where the capture ends. That length is the point of this step, not an accident. Scrolled back far enough you would find os, osrelease, kernel, num_cpus, mem_total, fqdn and the addresses.
Notice that the BIOS facts name real Dell hardware. The minion is a container running on a physical machine, and the hardware facts it can see belong to that machine. Grains report what the minion can actually observe, which is not always what you assumed.
Grains get a lesson of their own next, including how to read one fact instead of all of them, and how to target commands by fact rather than by name.
That grains dump was far longer than one screen. It scrolled off the top, and now you cannot compare it to anything.
Salt writes its results to standard output like any other program, which means the ordinary shell redirect works. The > character sends everything the command prints into a file instead of onto your screen, creating the file if it does not exist and replacing it if it does. ~ is shorthand for your home directory, so ~/output.txt is a file called output.txt in your own home.
Nothing about the Salt part of the command changes. The redirect is bolted onto the end.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' grains.items > ~/output.txt ||| sudo salt "minion1" grains.items > ~/output.txt ||| sudo salt minion1 grains.items > ~/output.txt output: hint: Run the same grains.items command, then add a > and the path: sudo salt 'minion1' grains.items > ~/output.txt
Your screen stayed empty, and that is the success. Every line that printed a moment ago went into ~/output.txt instead. Read it back with cat ~/output.txt and you get exactly the dump you just saw, starting at minion1:, still in the same order.
Two details worth keeping. The redirect belongs to your shell, not to Salt, so the file is written by your own account and is yours to read. And > replaces the file every time, so running the same command twice does not double the contents. Use >> when you want to add to the end instead of replacing.
This is how a routine record is kept. Capture a fleet's facts to a file today, capture them again after a change, and the difference between the two files is the evidence of what changed.
Scaffolding off. No command is shown from here.
A colleague asks which Salt release is actually installed on the minion, and you are not going to log into it to find out. The same module you used for the health check has a second function that reports it. Same three-part shape, one machine named, no arguments.
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' test.version ||| sudo salt "minion1" test.version ||| sudo salt minion1 test.version output: minion1: 3008.1 hint: Same test module as the health check, different function: the one that reports a version.
sudo salt 'minion1' test.version reports the version of the Salt installed ON the minion, not the version of the salt command you typed. That is the rule for every execution function:
Practice Remote Execution in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.