Learn › Salt Configuration Management › Production and Capstone

Salt-SSH

Use salt-ssh for agentless management without requiring a minion daemon.

A machine with nothing of ours on it

Every lesson in this track so far has needed something installed on the far end. A minion daemon, running, holding a key the master accepted, listening on the Salt bus.

Today you manage a machine without it. No daemon, no key, no bus. You connect the way a person connects: plain ssh, with a username and a password. Salt does the rest and leaves nothing behind.

Agentless is not free, though, and the bill arrives in two forms. One will appear on your screen within a few minutes, and working out what it means is most of this lesson. The other is a number you measure at the end.

Start by confirming the tool is here. which asks the shell where a program lives.

The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen was captured by booting this exact lab, so your own timestamps and durations will differ. The lab user is saltops, the master is saltmaster, and two machines sit on the lab network: minion1 at 10.100.1.2 and minion2. Both accept ssh as root with the password saltlab, and the helpers minion-ssh and minion-ssh2 run one command on each. Your progress is tracked automatically, so type naturally.

prompt: saltops@saltmaster:~$ answer: which salt-ssh ||| command -v salt-ssh output: /usr/bin/salt-ssh hint: Ask the shell where a program lives: which salt-ssh

/usr/bin/salt-ssh. A separate program from salt, shipped in its own package, sitting right beside the commands you have been using all along.

That separation is the thing to hold on to. salt publishes a job to daemons that are already connected. salt-ssh connects to machines itself. They share state files, pillar data and execution modules, and almost nothing else. Confirming the tool exists is the first thing your lab grades.

Read the file that decides which machines exist

With the agent, minions announce themselves. They connect, offer a key, and once you accept it they are part of the fleet. Nobody types a list of machines anywhere.

Agentless has no such thing. Nothing calls in, so you have to say where to go. That list is a file called the roster, and its default home is /etc/salt/roster. The salt-ssh package installs one. Read it before you touch it.

cat /etc/salt/roster

prompt: saltops@saltmaster:~$ answer: cat /etc/salt/roster ||| cat '/etc/salt/roster' output:

Sample salt-ssh config file

#web1:

host: 192.168.42.1 # The IP addr or DNS hostname

user: fred # Remote executions will be executed as user fred

passwd: foobarbaz # The password to use for login, if omitted, keys are used

hint: Print the file with cat and its full path: cat /etc/salt/roster

Five lines, and every single one starts with #. A # at the start of a line in YAML is a comment, so this file is documentation. As far as salt-ssh is concerned it is empty.

Read the comments anyway. They are a worked example and they name the fields you need:

web1: at the left margin is the target ID: the name you type on the command line. It is yours to choose and it has nothing to do with the machine's real hostname.
host: is where to connect. An IP address or a DNS name.
user: is the ssh account to log in as.
passwd: is that account's password. The comment names the better option: leave it out and keys are used instead.

One more thing. This file belongs to your account, which the setup arranged deliberately, so every write you do to it is a plain redirect with no sudo.

What agentless actually means

It does not mean nothing runs on the target. Something has to run there, or nothing could be done there. It means nothing is installed and nothing is left behind.

Here is the mechanism, worth knowing before you meet it in an error message. Salt builds a small tarball called the thin: a stripped-down copy of Salt, Python code, a few hundred kilobytes. It opens an ssh connection, copies the thin across, unpacks it into a temporary directory, runs the one thing you asked for, sends the answer back and disconnects. The unpacked thin stays cached so the next call skips the upload. No service is started and nothing joins the package database.

Two consequences follow, and both of them are this lesson.

The thin is Python source code. Something on the target has to run Python 3, or the thin cannot start.
Every command pays for a fresh ssh connection. The agent pays that once and keeps the channel open.

The minion daemon came first. Salt-ssh was added for the fleets nobody could put a daemon on: machines under a change freeze, appliances with a locked package manager, a brand new server with nothing on it. The design rule was that the same state files had to work either way. They do. The state you apply later is the file an agent would have run, unchanged.

Commit: is a login enough?

You are about to write your first roster entry: an address, a user, a password. Those three fields are all the sample shows, and ssh to that machine genuinely works.

[ ] Yes. Agentless means nothing has to exist on the target except a working sshd.
[x] Not necessarily. The thin is Python, so the target also needs a Python 3 that the ssh session can find.
[ ] Yes, because salt-ssh copies its own interpreter across with the thin.

>>> Not necessarily. Getting in is one problem, running something once you are in is another. If you picked the first answer, that is the common reading of agentless and it is half right: nothing is installed, but something is still executed, and executing Python needs a Python. If you picked the third, the thin holds Salt's own Python code and no interpreter at all. A CPython build is tens of megabytes, and shipping one on every call would defeat the tool.

Write your first roster entry

Replace the sample with a real entry. Use a heredoc: everything between << 'EOF' and a line reading exactly EOF becomes the content of the file, and the quotes around 'EOF' stop the shell touching anything inside.

cat > /etc/salt/roster << 'EOF'
minion1:
  host: 10.100.1.2
  user: root
  passwd: saltlab
EOF

No sudo on that write. The file is yours, and sudo in front of a redirect would not help anyway: the shell opens the output file as you, before sudo ever runs.

A heredoc prints nothing when it works, so read it back with cat /etc/salt/roster.

prompt: saltops@saltmaster:~$ answer: cat /etc/salt/roster ||| cat '/etc/salt/roster' output: minion1: host: 10.100.1.2 user: root passwd: saltlab hint: Print the file you just wrote: cat /etc/salt/roster

Four lines, and the comments are gone because > truncates the file before writing. The shape is the sample's: the ID at the left margin, the fields indented two spaces underneath.

minion1: is a label you chose. It matches the machine's name in the other lessons, which makes the commands read the same, but salt-ssh never checks that. It reads host: and connects there.

A roster file that exists and holds an uncommented host: line is the second thing your lab grades. You have that now.

That file holds a root password in plain text. Fine here, wrong in real work: use priv: with the path to an ssh private key instead of passwd:, and keep the roster at chmod 600. A roster is an inventory of your machines and how to get into them, which makes it one of the most valuable files on the box.

Ask the machine to answer

test.ping asks the far end to prove it is alive and answer True. Through the agent it goes over the Salt bus. Through salt-ssh the whole conversation is one ssh session.

The command shape is the one you know, with one flag added:

sudo salt-ssh -i 'minion1' test.ping

-i means ignore host keys. The first time ssh meets a machine it wants a human to confirm the fingerprint, and a Salt run has no human to ask, so it would stop. -i tells salt-ssh to accept the key and record it. Use it on every salt-ssh command in this lesson.

Before you press Enter, commit: one line saying True, or something longer?

prompt: saltops@saltmaster:~$ answer: sudo salt-ssh -i 'minion1' test.ping ||| sudo salt-ssh -i "minion1" test.ping ||| sudo salt-ssh -i minion1 test.ping output: minion1: ---------- _error: The command resulted in a non-zero exit code parsed: None retcode: 10 stderr: Warning: Permanently added '10.100.1.2' (ED25519) to the list of known hosts. ERROR: Unable to locate appropriate python command stdout: ERROR: Python version error. Recommendation(s) follow:

Install Python 3 on the target machine(s)
You can use ssh_pre_flight or raw shell (-r) to install Python 3

hint: The agentless command, the ignore-host-keys flag, the target in quotes, then the function: sudo salt-ssh -i 'minion1' test.ping

Not True. Read this screen field by field, because every line is useful.

_error: The command resulted in a non-zero exit code is Salt's summary: something ran on the far end and came back unhappy. parsed: None means it got no answer it could understand, because it expects structured data from the thin. retcode: 10 is the specific one: salt-ssh's code for a Python problem on the target, and you will recognise it instantly after today.

stderr: has two lines. The first is the ssh warning your -i flag caused: the host key was new, so it was added rather than refused. That is normal and will not appear again for this machine. The second is the fault: Unable to locate appropriate python command.

stdout: is the thin's own bootstrap script talking. It could not find an interpreter, so it printed the error and two suggestions rather than failing silently.

Now read what this screen does not say. It does not say the connection failed, and it does not say the password was wrong. Your roster is correct and the login worked. Salt got onto the machine and then could not run anything there.

Go and look for Python yourself

Salt says the target has no usable Python. Never take that on trust when you can check in one command. The setup left a helper, minion-ssh, which runs whatever you hand it on minion1 over ssh as root and prints the result here.

which prints nothing when it finds nothing, so add a fallback that says so out loud. || is plain shell for run the second thing only if the first failed.

minion-ssh 'which python3 || echo "no python3 on PATH"'

The single quotes keep your own shell off the command, so the whole string arrives at the far end intact.

prompt: saltops@saltmaster:~$ answer: minion-ssh 'which python3 || echo "no python3 on PATH"' ||| minion-ssh "which python3 || echo 'no python3 on PATH'" output: no python3 on PATH hint: Run which on the far end through the helper, with a fallback echo so an empty answer is visible: minion-ssh 'which python3 || echo "no python3 on PATH"'

There is no python3 on that machine's PATH. Salt was telling the truth.

That is stranger than it sounds. This machine runs a Salt minion, so it has Salt installed, and Salt is written in Python. There is certainly a Python on that box somewhere. The next step is finding where it went.

Find the interpreter Salt brought with it

Modern Salt packages are onedir builds. Instead of depending on whatever Python the operating system ships, the package carries its own complete CPython and installs it under /opt/saltstack/salt. It behaves identically on every distribution and a system Python upgrade cannot break it.

The cost is the one you just hit: the package needs no system python3, so it installs none, and on a machine whose only Python came from Salt there is nothing on the normal PATH.

Look inside the directory the package owns. ls -l on one file gives the long listing for it and nothing else.

minion-ssh 'ls -l /opt/saltstack/salt/bin/python3'

prompt: saltops@saltmaster:~$ answer: minion-ssh 'ls -l /opt/saltstack/salt/bin/python3' ||| minion-ssh "ls -l /opt/saltstack/salt/bin/python3" output: lrwxrwxrwx. 1 salt salt 10 Jul 1 14:44 /opt/saltstack/salt/bin/python3 -> python3.14 hint: The long listing of one file, run on the far end through the helper: minion-ssh 'ls -l /opt/saltstack/salt/bin/python3'

There it is. Read the line from the left.

The first character is l, so this is a symbolic link: a name that points at another name. The arrow says what it points at, python3.14, in the same directory. The owner columns say salt salt, because this tree belongs to the Salt package rather than to the operating system.

So the machine has a working Python 3, in a directory nobody looks in. One question is left: why did the thin not find it?

Read the PATH your command actually gets

When you run a command over ssh without opening an interactive shell, you do not get the PATH you would see if you logged in and looked around. You get a short default one, set by sshd, that skips most of what a login shell would load.

Ask the far end to print it. The single quotes do the same job as before: they stop your own shell expanding $PATH here, so the far end expands its own.

minion-ssh 'echo $PATH'

prompt: saltops@saltmaster:~$ answer: minion-ssh 'echo $PATH' ||| minion-ssh 'printf "%s\n" "$PATH"' output: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hint: Echo the PATH variable on the far end, in single quotes so your own shell leaves it alone: minion-ssh 'echo $PATH'

Six directories, separated by colons, and /opt/saltstack/salt/bin is not one of them.

That is the whole fault, explained. The interpreter exists. The command that needs it cannot see it, because the directory holding it is not on the list of places the shell searches, and which python3 failed for the same reason.

Nothing here is broken. A default PATH and a package that installs outside it are both behaving as designed, and the two designs do not meet.

Commit: where does the fix belong?

You have all three facts: no python3 on PATH, a working interpreter at /opt/saltstack/salt/bin/python3, and a PATH without that directory in it. Decide what to do before you read on.

[ ] Install a system python3 on the target with apt, and on every target after it.
[x] Add one line to the roster entry telling salt-ssh to extend PATH for this machine.
[ ] Nothing on the master can help. PATH belongs to the target, so the target must be changed.

>>> One line in the roster. The roster is not only an address book: it carries per-machine options for how salt-ssh behaves once it gets there, and the PATH is one of them. If you picked the first answer, it would work, and the error message suggests it, but look at the cost: a package on every machine you manage, on a fleet you chose salt-ssh for because you cannot install things on it. If you picked the third, salt-ssh sets the PATH for the commands it runs, so telling it what to set leaves no footprint on the machine.

Add the one line that fixes it

The roster option is set_path, and its value is the PATH salt-ssh uses on that machine. Write the file again with the new line at the bottom:

cat > /etc/salt/roster << 'EOF'
minion1:
  host: 10.100.1.2
  user: root
  passwd: saltlab
  set_path: '$PATH:/opt/saltstack/salt/bin'
EOF

Two pieces of quoting are doing real work. The 'EOF' in quotes keeps your own shell out of the heredoc, so $PATH lands in the file as those five characters rather than your master's PATH. The single quotes around the value do the same job inside YAML.

The value reads as: whatever PATH would have been, then this directory on the end. Salt-ssh expands it on the far end, so nothing is lost and one directory is gained. Read it back with cat /etc/salt/roster.

prompt: saltops@saltmaster:~$ answer: cat /etc/salt/roster ||| cat '/etc/salt/roster' output: minion1: host: 10.100.1.2 user: root passwd: saltlab set_path: '$PATH:/opt/saltstack/salt/bin' hint: Print the file again and check the new line landed: cat /etc/salt/roster

Five lines now, and the fifth is indented two spaces like the others, so it belongs to minion1 and nothing else. That matters on a real roster: every machine can have its own set_path, because every machine can have Python somewhere different.

Notice what you did not do. You did not log into the target, install anything on it, or edit a file on it. The machine is exactly as you found it. What changed is what the master will say when it connects.

Run the same command again

Nothing else has changed. Same target, same function, same flag, same machine. Run the identical command you ran before, and watch how many lines come back this time.

sudo salt-ssh -i 'minion1' test.ping

True, from a machine running no agent of yours, reached over ordinary ssh, fixed by one line on this master.

A lot happened in those two seconds. Salt opened an ssh connection, copied the thin across, unpacked it into a temporary directory, ran it with the interpreter your set_path line made reachable, collected the answer and disconnected. What is left there is the unpacked thin, cached so the next call does not resend it.

That cached directory is the evidence your lab grades. The check for this task reaches into minion1 and looks for the thin. A ping that printed the retcode 10 error deployed no thin and does not count, which is the honest way round.

Apply a real state to a machine with no daemon

A ping proves the path works. Configuration is the point. This master already has /srv/salt/webserver.sls, the shape you have been writing since the states lessons:

nginx:
  pkg.installed: []
  service.running:
    - enable: True
    - require:
      - pkg: nginx

One ID, nginx, with two state functions under it. Nothing in that file mentions ssh, rosters or salt-ssh. It was written for the agent and is about to be used unchanged.

The command is the one you know, with salt-ssh -i in front instead of salt:

sudo salt-ssh -i 'minion1' state.apply webserver

This one is slower, because installing a package means the far end talking to its mirrors. Before you press Enter, commit: how many state blocks come back from a file with one ID and two functions?

prompt: saltops@saltmaster:~$ answer: sudo salt-ssh -i 'minion1' state.apply webserver ||| sudo salt-ssh -i "minion1" state.apply webserver ||| sudo salt-ssh -i minion1 state.apply webserver output: minion1: ---------- ID: nginx Function: pkg.installed Result: True Comment: The following packages were installed/updated: nginx

Practice Salt-SSH in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.