Learn › Salt Configuration Management › Production and Capstone

Capstone: Full Stack Deploy

Deploy a complete web + database stack across two minions using everything you have learned.

Four empty places, and one command that will fill them

Almost everything on this master is a placeholder. The formulas directory is empty. The orchestration directory is empty. The top file hands every machine a state called placeholder, and the pillar tree holds one value naming the project.

Two machines are waiting: minion1 and minion2. Both keys are already accepted. Neither has been told what it is for.

By the end of this lesson, one command typed here will put a database on one machine and a web server on the other, in that order, each configured from data only it can see.

Nothing here is new. Every piece is something you have used on its own, and today they have to agree.

The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen here was captured by booting this exact lab, so your own timestamps and package versions will differ. The lab user is saltops, the master is saltmaster, and the minions are minion1 and minion2, both keys already accepted. Progress is tracked automatically, so type commands naturally.

Look at everything you have been handed

Four directories carry this lesson. /srv/formulas will hold the formulas, /srv/pillar the per-machine data, /srv/salt is the state tree you have used all track, and /srv/salt/orch inside it will hold the plan.

Naming several directories makes ls label each one before listing its contents.

ls /srv/formulas/ /srv/pillar/ /srv/salt/ /srv/salt/orch/

prompt: saltops@saltmaster:~$ answer: ls /srv/formulas/ /srv/pillar/ /srv/salt/ /srv/salt/orch/ output: /srv/formulas/:

/srv/pillar/: defaults.sls top.sls

/srv/salt/: orch placeholder.sls top.sls

/srv/salt/orch/: hint: One ls, four directories: ls /srv/formulas/ /srv/pillar/ /srv/salt/ /srv/salt/orch/

Read the empty ones first. /srv/formulas/: has a heading and nothing under it, and so does /srv/salt/orch/:. Those are not errors. Filling them is the work of this lesson.

/srv/pillar/ holds defaults.sls and top.sls. /srv/salt/ holds orch, plus placeholder.sls and top.sls.

Four directories, five files, two machines. That is the entire starting position.

Read the three files the init left behind

Each of these three files is a slot you are about to fill.

Start with the state top file, which decides what states a machine gets.

cat /srv/salt/top.sls

prompt: saltops@saltmaster:~$ answer: cat /srv/salt/top.sls output:

Top file: assign states to minions

The member builds this out during the capstone

base: '*':

placeholder

hint: Print the state top file: cat /srv/salt/top.sls

Two comments, then the top file. base: names the environment, '*': matches every machine, and - placeholder is the state each matched machine gets, referring to placeholder.sls next door.

Hold on to that shape: environment, target, list of names. The pillar version looks identical.

Now the pillar top file. Same syntax, different question: which data does each machine get.

cat /srv/pillar/top.sls

prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/top.sls output:

Pillar top file: assign pillar data to minions

The member builds per-server pillar data here

base: '*':

defaults

hint: Print the pillar top file: cat /srv/pillar/top.sls

Same structure, different meaning. Every machine matched by '*' gets defaults.sls from the same directory.

That '*' is the first thing this lesson changes. Right now both machines get identical data, which is the opposite of what a two-tier stack needs.

And the data itself. One file, and it is short.

cat /srv/pillar/defaults.sls

prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/defaults.sls output:

Default pillar values

project_name: capstone hint: Print the default pillar file: cat /srv/pillar/defaults.sls

One key, one value, and that is the sum of what this master knows about either machine.

Nothing here says which machine is the web tier, or gives either a port or a database name. Every fact that makes minion1 different from minion2 is one you are about to write.

What a full stack deploy is actually made of

A deployment across more than one machine is four decisions, and Salt keeps them in four places on purpose.

The formula says *how*. A state file describing the work: install this package, write this file. No machine names, no values that differ between machines.
The pillar says *what, for this machine*. A port number, a database name. Data only, no logic.
The top files say *who gets which*. One for states, one for pillar, both with the syntax you just read.
The orchestration says *in what order, across the fleet*. It runs on the master and issues state runs in sequence.

Keeping them apart is what makes one formula usable on ten machines needing ten different ports. Mix them and you get a state file with minion1 inside it.

Why the master runs the plan

Salt was written in 2011 by Thomas Hatch, who wanted to ask a thousand machines a question and get the answers back in under a second. That message bus is why test.ping returns while you are still watching.

The speed created a new problem. A database that comes up after the application needing it is a fleet that is fast and wrong, and a minion cannot decide the ordering, because it knows nothing about the others.

So the ordering lives where the knowledge is. The master talks to everything, so the master is where a plan can say: this target, then that one, and only if the first worked.

Carry this sentence into the lab: a state file orders work inside one machine, and an orchestration orders work between machines.

Commit: who can see which pillar value?

You are about to write two pillar files and a top file giving web to minion1 and db to minion2. Take a position before you type it.

[ ] The value from db.sls. Both pillar files sit in the same directory, so both machines can read both.
[x] Nothing. The pillar top file decides which files a machine is given, and minion1 was never given db.sls.
[ ] An error, because that key does not exist for that machine.

>>> Nothing comes back at all. Pillar is compiled per machine: the master reads the top file, works out which files apply, and builds a private set of data. If you picked the first answer, sitting in the same directory grants nothing, and that per-machine compilation is what keeps one machine's secrets off another. If you picked the third, an unassigned key is not an error. It is a normal question with an empty answer, which you will see later.

Give each machine its own data

Two files, one value each. Pillar is data, and anything clever belongs elsewhere. These directories belong to your account, so every write in this lesson is a plain heredoc: no sudo, no tee.

cat > /srv/pillar/web.sls << 'EOF'
web_port: 8080
EOF

cat > /srv/pillar/db.sls << 'EOF'
db_name: capstone_db
EOF

Now the assignment, replacing the placeholder top file you read earlier.

cat > /srv/pillar/top.sls << 'EOF'
base:
  'minion1':
    - web
  'minion2':
    - db
EOF

Read it back. Never trust a heredoc you have not printed.

prompt: saltops@saltmaster:~$ answer: cat /srv/pillar/top.sls output: base: 'minion1':

'minion2':

hint: Print the file you just replaced: cat /srv/pillar/top.sls

The '*' is gone and two machine names have taken its place. Read it as two sentences: minion1 gets web, minion2 gets db.

Two things are worth noticing. defaults is no longer assigned to anything, so project_name no longer reaches either machine: a top file is a complete statement, not an addition to the last one. And web and db are filenames without the .sls, resolved inside /srv/pillar.

Make the machines fetch their new data

A minion caches its pillar. Writing files on the master does not update a running minion, and that is the most common reason a correct pillar file appears to do nothing.

saltutil.refresh_pillar tells the minions to recompile. Send it to both at once.

sudo salt '*' saltutil.refresh_pillar

prompt: saltops@saltmaster:~$ answer: sudo salt '*' saltutil.refresh_pillar ||| sudo salt "*" saltutil.refresh_pillar output: minion2: True minion1: True hint: Target both machines and refresh: sudo salt '*' saltutil.refresh_pillar

True from both, and minion2 answered first. Returns arrive in whatever order the machines answer: a '*' command is many conversations at once, not a list worked through.

True means the refresh was accepted, not that the data is what you intended. That is the next question.

Ask each machine what it now knows

Ask each one for its own key, one machine at a time. The interesting part is the difference between the answers.

Start with the web tier.

sudo salt 'minion1' pillar.get web_port

prompt: saltops@saltmaster:~$ answer: sudo salt 'minion1' pillar.get web_port ||| sudo salt "minion1" pillar.get web_port ||| sudo salt minion1 pillar.get web_port output: minion1: 8080 hint: Ask one machine for one pillar key: sudo salt 'minion1' pillar.get web_port

8080, printed plainly. The value you wrote into web.sls reached the machine the top file assigned it to.

pillar.get takes one key and returns its value for one machine. It answers the question every pillar problem reduces to.

Now the database tier, same command shape, its own key.

sudo salt 'minion2' pillar.get db_name

prompt: saltops@saltmaster:~$ answer: sudo salt 'minion2' pillar.get db_name ||| sudo salt "minion2" pillar.get db_name ||| sudo salt minion2 pillar.get db_name output: minion2: ********** hint: Same shape, other machine, other key: sudo salt 'minion2' pillar.get db_name

Ten asterisks. Not an error, not an empty value, and not a sign that anything went wrong.

Salt masks some pillar values on their way to your terminal. The machine has the value and the master compiled it. The printing is what was suppressed. One screen ago web_port came back as 8080 in the clear, so whatever the rule is, it is not the same for every key.

That leaves a question hanging: if you cannot read it, can a state file use it? The formulas you write next interpolate this exact key into a file on that machine, and you will read that file back later.

Write both formulas

A formula here is a directory with an init.sls in it. Name the directory db and Salt resolves the state name db to db/init.sls.

The database tier first, because that is the order the deployment runs in. These directories are yours, so no sudo.

mkdir -p /srv/formulas/db
cat > /srv/formulas/db/init.sls << 'EOF'
db_package:
  pkg.installed:
    - name: sqlite3

db_marker:
  file.managed:
    - name: /etc/db_ready
    - contents: 'database tier ready, schema {{ salt['pillar.get']('db_name', 'unset') }}'
    - require:
      - pkg: db_package
EOF

prompt: saltops@saltmaster:~$ answer: cat /srv/formulas/db/init.sls output: db_package: pkg.installed:

name: sqlite3

db_marker: file.managed:

name: /etc/db_ready
contents: 'database tier ready, schema {{ salt'pillar.get' }}'
require:
pkg: db_package

hint: Print the formula you just wrote: cat /srv/formulas/db/init.sls

Two states, with a dependency between them.

db_package installs sqlite3.
db_marker writes /etc/db_ready, and require holds it back until the package state has succeeded.
The {{ ... }} in the contents is Jinja, and it runs before the YAML is parsed. By then the expression is a plain value, which is why the quotes inside it do not break the quotes around it.
salt'pillar.get' is pillar.get called from a template. The second argument is the fallback: a missing key writes the word unset, turning a silent misconfiguration into a visible one.

There is no service.running here and there will not be one in the next formula. These minions are containers with salt-minion as process 1 and no init system, so nothing on them can be started as a service. Every claim this lesson makes is checkable: a file on disk, and a package in a list.

The web tier is the same shape with its own names.

mkdir -p /srv/formulas/web
cat > /srv/formulas/web/init.sls << 'EOF'
web_package:
  pkg.installed:
    - name: nginx

web_marker:
  file.managed:
    - name: /etc/web_ready
    - contents: 'web tier ready on port {{ salt['pillar.get']('web_port', 'unset') }}'
    - require:
      - pkg: web_package
EOF

prompt: saltops@saltmaster:~$ answer: cat /srv/formulas/web/init.sls output: web_package: pkg.installed:

name: nginx

web_marker: file.managed:

name: /etc/web_ready
contents: 'web tier ready on port {{ salt'pillar.get' }}'
require:
pkg: web_package

hint: Print the other formula you just wrote: cat /srv/formulas/web/init.sls

Put the two files side by side in your head. The structure is identical: a package state, a marker state, a require between them. Only names differ.

That is the property worth taking away. Neither file mentions a machine, a port or a database name. Both would work unchanged on a machine you have not built, as long as it is assigned the right pillar.

Commit: what makes a directory of formulas reachable?

Two formulas now exist on disk under /srv/formulas. Take a position on what a master needs before it will hand one out.

[ ] Nothing more. The file is under /srv, which is where Salt looks.
[x] file_roots has to name that directory, and the master has to have read that setting when it started.
[ ] The minion has to refresh its file list, the way it refreshed its pillar.

>>> file_roots has to name the directory, and the master reads it at startup. A tree of state files is just files until the master is told to serve it. If you picked the first answer, /srv carries no special meaning: those paths are conventional, and a formula in a directory nobody declared is one nobody can reach. If you picked the third, minions do not search for state files. They ask the master by name.

Confirm the master is pointed at the formulas

The master reads every .conf in /etc/salt/master.d/ when it starts. That is a drop-in directory: small files merged at startup, instead of one enormous file.

On this master that setting is already in place. The lab's own lab.conf in that directory declares file_roots with both /srv/salt and /srv/formulas, so the formulas you just wrote are already being served. Write the setting out yourself anyway, into the empty /etc/salt/master.d/capstone.conf that belongs to your account: on a master you set up, this is the step that makes a new directory reachable, and this is the shape it takes.

cat > /etc/salt/master.d/capstone.conf << 'EOF'
file_roots:
  base:
    - /srv/salt
    - /srv/formulas
EOF

prompt: saltops@saltmaster:~$ answer: cat /etc/salt/master.d/capstone.conf output: file_roots: base:

/srv/salt
/srv/formulas

hint: Print the drop-in config you just wrote: cat /etc/salt/master.d/capstone.conf

Four lines. file_roots: is the setting, base: is the environment, and under it is an ordered list of directories the master serves. It is a search order: Salt looks in /srv/salt first, and the first match wins.

A master reads its configuration when it starts and not again, so a drop-in added afterwards does nothing until it restarts. Restart it, then give the minions a few seconds to reconnect.

sudo systemctl restart salt-master

Confirm the fleet came back before trusting anything next. A deployment that appears to fail because a minion had not reconnected is a fault you can spend a long time not finding.

sudo salt '*' test.ping

prompt: saltops@saltmaster:~$ answer: sudo salt '*' test.ping ||| sudo salt "*" test.ping output: minion1: True minion2: True hint: Ping everything: sudo salt '*' test.ping

Both machines answered. The master restarted, read your drop-in, and the minions found it again without anyone touching them.

If either is missing, or you get Minion did not return, you were almost certainly faster than the reconnect. Wait a few seconds and ask again before looking for a real fault.

Write the plan that runs both tiers

Everything the deployment needs exists, and nothing has been deployed. The sequence is missing.

An orchestration lives in the state tree like any other file. salt.state is what makes it different: it issues a state run at a target, from the master, as one step of a plan.

cat > /srv/salt/orch/deploy.sls << 'EOF'
deploy_database:
  salt.state:
    - tgt: 'minion2'
    - sls:
      - db

deploy_web:
  salt.state:
    - tgt: 'minion1'
    - sls:
      - web
    - require:
      - salt: deploy_database
EOF

prompt: saltops@saltmaster:~$ answer: cat /srv/salt/orch/deploy.sls output: deploy_database: salt.state:

tgt: 'minion2'
sls:
db

deploy_web: salt.state:

tgt: 'minion1'
sls:
web
require:
salt: deploy_database

hint: Print the orchestration you just wrote: cat /srv/salt/orch/deploy.sls

Two steps, and every line is doing a job.

Line	Meaning
`deploy_database:`	The step ID, your label for this piece of the plan
`salt.state:`	Run a state on a target, from the master
`- tgt: 'minion2'`	Which machine, as if typed after `salt`
`- sls:` then `- db`	Which state to apply, resolved through `file_roots`
`- require:` then `- salt: deploy_database`	This step waits for that step

Read the requisite carefully, because the direction is the point. It sits on deploy_web and names deploy_database: the step that depends is the step that carries the requirement. salt: before the ID is the state type being required, the same way you write - pkg: db_package inside a formula.

The database goes first because the tier other things depend on comes up first.

Commit: what is the require actually buying you?

The two steps appear in the order they should run, and one of them also carries a require. Take a position on what that requisite adds.

[ ] Nothing. The steps are already written in the order they should run.
[x] The web step would no longer wait for the database step, and it would run even if the database step had failed.
[ ] The whole orchestration would refuse to run, because orchestration steps must be linked.

>>> The requisite turns two steps into a sequence with a condition attached. If you picked

Practice Capstone: Full Stack Deploy in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.