Learn › Salt Configuration Management › Production and Capstone
Debug common Salt issues: key mismatches, broken YAML, test mode, and manage.status.
This master is not down. The package is installed, the service is up, the machine it manages is powered on, and the salt-minion process over there is running.
And the fleet does not answer.
That is the hardest kind of fault to walk into. Every reassuring check comes back green and none of them touch the thing that is wrong. No crash, no stack trace, no service to restart.
Two faults were planted on this box before you logged in, and neither announces itself. By the end of this lesson you will have found both, fixed both, and written down the order you worked in.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and doing exactly what you are about to do. Your own timestamps, durations and package versions will differ. The lab user is saltops, the master is named saltmaster, and there is one minion machine. Your progress in the lab is tracked automatically, so type commands naturally.
The first rule of this subject is that you check the cheapest thing first, and the cheapest thing is always on the machine you are sitting on.
systemctl is-active asks systemd for a one-word verdict on a service. Quicker to read than status, quicker to type, and it answers exactly one question with no noise around it.
sudo systemctl is-active salt-master
prompt: saltops@saltmaster:~$ answer: sudo systemctl is-active salt-master ||| systemctl is-active salt-master ||| sudo systemctl is-active 'salt-master' output: active hint: The one-word service verdict from systemd: sudo systemctl is-active salt-master
active. One word, and it is the honest answer to the question you asked.
Be precise about what it rules out: a master that failed to start, crashed, or was never enabled. It says nothing about whether the master can reach a minion. A running process with a wrong picture of the world is still a running process, and systemd has no opinion on the picture.
test.ping is the cheapest question you can put to a minion. It does not ping in the network sense. It publishes a tiny job over the Salt bus, and a healthy minion answers True, which proves the connection, the accepted key and the round trip together. The '*' is quoted so the shell hands the star to Salt instead of expanding it against your current directory.
Before you press Enter, commit. One machine is out there with its minion process running. How many blocks do you expect back, and what name do you expect on them?
sudo salt '*' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt '*' test.ping ||| sudo salt "*" test.ping output: ERROR: Minions returned with non-zero exit code bad-minion: Minion did not return. [No response] The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
hint: Quote the star so the shell leaves it alone, then the module and function: sudo salt '*' test.ping
Read this screen from the second line, not the first.
bad-minion: is a name you have never seen. The master printed it because it believes that is one of the minions it manages. Minion did not return. [No response] is what it says when it published a job to a name, waited, and got nothing back. That is not an error from the minion. It is the absence of a minion.
Now read what is missing, which is the more useful half. There is no block for the machine you actually have. The master did not try it and fail. It never mentioned it.
ERROR: Minions returned with non-zero exit code on the first line is the headline, not the diagnosis. It sits above many different faults, and the useful line is always further down.
It feels like a fault could be anywhere. In a Salt setup there are exactly three places for one to hide.
The order matters, and it is the order of this lesson. Check who is talking before you check what they are saying.
The screen you just read is a fault in the first place. Here is why that place exists at all.
Salt could have used a shared password, as early tools did, and that works until the day one of four hundred machines is stolen. So Salt uses a key pair instead: two matching halves of a mathematical lock. The minion keeps the private half and sends the public half to the master, asking to be let in.
What matters today is where the master files that public half: a directory of accepted keys, in a file named after the minion id. Not a database, not an index, just a filename. So the filename is the identity, and changing it changes who the master thinks it is talking to without touching a byte of the key.
Carry this sentence: the master does not know your minions, it knows a directory of filenames. Almost every key fault is a disagreement between a name in that directory and a name a machine calls itself.
Take a position before you touch anything.
>>> The list is wrong. The master published a job to every name it holds, and the only name it holds is one nothing answers to. If you picked the first answer, a dead network is a reasonable guess and the most expensive one to chase, because you cannot prove it from here without help. Check the cheap local facts before you ask anyone about switches. If you picked the third, look again at what the screen did not say. A stopped minion service leaves the right name in the list, accepted and silent. Here the right name is absent, and no service on the minion can add or remove a filename in the master's key directory.
salt-key manages that directory of names. With --list-all it prints every name the master holds, sorted into the four states a key can be in. It changes nothing, and it is the single most useful command in this lesson.
sudo salt-key --list-all
prompt: saltops@saltmaster:~$ answer: sudo salt-key --list-all ||| sudo salt-key -L ||| sudo salt-key --list-all --out=txt output: Accepted Keys: bad-minion Denied Keys: Unaccepted Keys: minion1 Rejected Keys: hint: The key manager, asked to list everything it holds: sudo salt-key --list-all
There is the whole fault on one screen, and it took one read to find. Four headings, and each is a state a key can be in:
| Heading | What it means |
|---|---|
Accepted Keys | The master will talk to these names |
Denied Keys | A key arrived claiming a name the master holds a different key for |
Unaccepted Keys | A machine asked to join and nobody has answered yet |
Rejected Keys | An operator refused this one |
bad-minion is accepted. That is the name that answered nothing, because no machine here calls itself that. minion1 is unaccepted. That is the real machine, asking to join this whole time, into a queue nobody was reading.
Both halves of the fault are visible at once: the master holds a name that is not real, and refuses to talk to the name that is. Everything from here is repair.
salt-key -d deletes a key by name. The -y answers the confirmation prompt in advance, which is fine here and worth thinking twice about anywhere else.
sudo salt-key -d bad-minion -y
prompt: saltops@saltmaster:~$ answer: sudo salt-key -d bad-minion -y ||| sudo salt-key -d 'bad-minion' -y ||| sudo salt-key -y -d bad-minion ||| sudo salt-key --delete=bad-minion -y output: The following keys are going to be deleted: Accepted Keys: bad-minion Key for minion bad-minion deleted. hint: The key manager, deleting one name, with the confirmation answered in advance: sudo salt-key -d bad-minion -y
Salt shows you what it is about to remove, then confirms it is gone. Both lines are worth reading, because -y means you agreed to the first without seeing it.
The accepted list is now empty. A fleet of nothing is briefly worse than what you started with, and it is the correct middle of this repair.
Deleting a key on a real fleet cuts that machine off until its key is accepted again. If nobody is watching the unaccepted queue for it, you have taken a working server out of management. Delete a key when you know why it is wrong, never as a first move.
salt-key -a accepts a key by name, moving it out of the unaccepted queue into the accepted list. Same -y.
sudo salt-key -a minion1 -y
prompt: saltops@saltmaster:~$ answer: sudo salt-key -a minion1 -y ||| sudo salt-key -a 'minion1' -y ||| sudo salt-key -a "minion1" -y ||| sudo salt-key -y -a minion1 ||| sudo salt-key --accept=minion1 -y output: The following keys are going to be accepted: Unaccepted Keys: minion1 Key for minion minion1 accepted. hint: The key manager again, this time accepting the name that was waiting: sudo salt-key -a minion1 -y
Same two-part shape as the delete: here is what I am about to do, and here is confirmation that I did it.
Nothing was sent to the machine. No service was restarted and nobody logged into it. A file moved from one directory to another on this master, and that file is now the reason the two of them can speak.
Never trust a repair you have not re-read. Ask the same question you asked before, and compare the two screens.
sudo salt-key --list-all
prompt: saltops@saltmaster:~$ answer: sudo salt-key --list-all ||| sudo salt-key -L output: Accepted Keys: minion1 Denied Keys: Unaccepted Keys: Rejected Keys: hint: The same list command you ran before the repair: sudo salt-key --list-all
One name, accepted, and it is the right one. The unaccepted queue is empty because nothing is waiting any more.
Put this screen beside the first list and the diff is the whole first half of this lesson. Two salt-key calls, no restarts, nothing touched on the minion.
This is the moment the repair is either real or not. Run the exact command that failed at the start of this lesson and read what comes back.
sudo salt '*' test.ping
prompt: saltops@saltmaster:~$ answer: sudo salt '*' test.ping ||| sudo salt "*" test.ping output: minion1: True hint: The same fleet-wide ping you ran at the start: sudo salt '*' test.ping
minion1: True. The star means what it always meant. What changed is what the star resolves to.
That is the diff worth holding on to. The command was identical both times. The failure was never in the command, never in the network, never on the minion. It was a filename in a directory on this master.
test.ping proves a machine answered. It does not give you a roll call, and on a real fleet a roll call is what you want first.
salt-run runs a runner, a command that executes on the master itself rather than being sent to a minion. manage.status splits every accepted name into two lists: the ones that answered and the ones that did not.
sudo salt-run manage.status
prompt: saltops@saltmaster:~$ answer: sudo salt-run manage.status output: down: up:
hint: A runner asks the master about itself, and the manage runner reports on the fleet: sudo salt-run manage.status
down: with nothing under it, and minion1 under up:. Read the empty heading as carefully as the full one.
Reach for this first when somebody says the fleet is broken, because it turns a vague complaint into two lists. One machine under down: gives you a machine to look at. Every machine under down: means the fault is on the master.
One warning. manage.status only reports on names that are accepted. The machine you just repaired would not have appeared under down: an hour ago, it would not have appeared at all. That is why the key list comes first, always.
Stop and count what you did. Every obvious check said healthy and the fleet was silent. You read one list, saw two names in the wrong states, ran two commands, and the fleet came back.
The method is the part to keep, not the commands. You checked the local service, asked the fleet, read the state the master holds, and only then changed anything. All of it cheap, none of it able to make things worse.
The second fault lives somewhere else entirely. There is a state file on this master called /srv/salt/broken.sls, written before you logged in and named after its own problem.
In YAML, indentation carries the meaning, and it must be made of spaces. A tab is one character drawn several columns wide, so a file with tabs looks aligned to you and is a different shape to the parser.
>>> The run stops. A minion renders the state files into a plan before it changes anything, and a file it cannot parse means there is no plan to run. If you picked the first answer, plenty of tools do tolerate tabs, which is exactly why this catches people. The YAML specification forbids them for indentation, and Salt does not soften that. If you picked the third, partial application would be far more dangerous than failing. A half-applied plan leaves a machine in a state nobody described. Salt would rather change nothing.
Apply it the ordinary way. state.apply takes the state name without the .sls extension, so /srv/salt/broken.sls is applied as broken. Read every line of what comes back.
sudo salt 'minion1' state.apply broken
prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' state.apply broken ||| sudo salt "minion1" state.apply broken ||| sudo salt minion1 state.apply broken output: ERROR: Minions returned with non-zero exit code minion1: Data failed to compile: ---------- Rendering SLS 'base:broken' failed: Illegal tab character; line 5
--- install_packages: pkg.installed:
hint: The ordinary apply, with the state name and no .sls on the end: sudo salt 'minion1' state.apply broken
This is one of the most helpful error screens Salt produces. Take it apart line by line.
Data failed to compile: is the stage that failed. Compiling means turning the state files into a plan. That happens before any change is made, so this run touched nothing on the machine.
Rendering SLS 'base:broken' failed names the file in two parts: base is the environment it searched, broken is the state name inside it. Illegal tab character; line 5 is the diagnosis and the address. Not a category, not a guess, a line number.
Then Salt prints the file back with <====================== pointing at the offending line. Look at the wget line and then at the two around it. They do not start at the same column. You are looking straight at the fault and still cannot see the character causing it, because that character is invisible.
You cannot debug whitespace by looking at it. cat -A prints a file with every non-printing character shown as a visible symbol. Three of them matter here:
| Symbol | The character it stands for |
|---|---|
$ | The end of the line. Anything between it and the last visible character is trailing whitespace |
^I | A tab |
^M | A carriage return, which is what a file written on Windows leaves behind |
This file is yours. The setup handed /srv/salt to your account, so no sudo is needed to read it or, in a moment, to rewrite it.
cat -A /srv/salt/broken.sls
prompt: saltops@saltmaster:~$ answer: cat -A /srv/salt/broken.sls ||| cat --show-all /srv/salt/broken.sls ||| cat -vET /srv/salt/broken.sls output: install_packages:$ pkg.installed:$
^I - wget$
hint: The cat flag that shows all non-printing characters is -A: cat -A /srv/salt/broken.sls
There it is. ^I - wget$ on line 5, and every other line begins with plain spaces.
Read the broken line against the two good ones. - curl$ starts with six spaces, and so does - jq$. The wget line starts with a tab and three spaces, drawn at roughly the same width and read by YAML as something else entirely.
Every $ sits immediately after the last visible character, so there is no trailing whitespace here either. Keep cat -A in your hands permanently. Any time a file looks right and behaves wrong, it is the first thing to reach for.
You could hunt the tab down in an editor and hope you replaced it with the right number of spaces. On a six-line file it is faster and more certain to write the whole thing again.
A heredoc is the shell's way of typing several lines into a file at once. cat > path << 'EOF' opens the file, everything you type goes in until a line reading exactly EOF, and quoting 'EOF' stops the shell touching anything inside. Only the whitespace on the wget line changes:
cat > /srv/salt/broken.sls << 'EOF'
install_packages:
pkg.installed:
- pkgs:
- curl
- wget
- jq
EOFNo sudo on that write, because the directory is yours. A heredoc prints nothing when it works, so prove it with the same tool that found the fault.
cat -A /srv/salt/broken.sls
prompt: saltops@saltmaster:~$ answer: cat -A /srv/salt/broken.sls ||| cat --show-all /srv/salt/broken.sls ||| cat -vET /srv/salt/broken.sls output: install_packages:$ pkg.installed:$
hint: The same flag that exposed the tab, run on the file you just rewrote.
Six lines, six $ symbols, and no ^I anywhere. Every list item under - pkgs: now starts at the same six-space indent.
Compare this screen with the previous one. One character changed, in one column, on one line. That was the entire second fault, and it was enough to stop a package installation on another
Practice Troubleshooting Salt in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.