Learn › Mental Models › Everything is a Queue

Everything is a Queue

uptime - a hands-on Linux lab on a real virtual machine.

The most universal model in systems: every latency spike has a queue behind it. The real M/M/1 math on an interactive chart, the 80 percent knee, utilization vs saturation, and the same model read live on CPU run queues and socket backlogs.

It is 09:12 and your team just shipped a deploy. Within minutes, the pager fires: checkout requests that normally finish in 200 milliseconds are taking 9 seconds. You pull up the dashboards. CPU: 40 percent. Memory: fine. Network: fine. Disk: fine. Every graph is green.

Nine seconds, and nothing is busy.

Most people stare at that contradiction for an hour. It feels impossible: how can everything be slow when nothing is working hard? This lesson gives you the one idea that dissolves the contradiction, and it is the same idea behind almost every latency mystery you will ever meet.

This is the first lesson of the Mental Models track. A mental model is a pattern that repeats across many different systems. You will learn the pattern once, with its real math, then see it running on a real Linux machine in two different places. The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output in them is a real capture from a real 4-core Linux machine.

A queue is a waiting line. Requests arrive, they wait their turn, one server works them off, they leave. Three numbers describe every queue in existence:

Two more words you will use forever. Utilization is the fraction of time the server is busy. Saturation is when work is waiting in line. They are not the same thing, and the difference is the whole lesson.

Here is the model: every latency problem is a queue you have not found yet. The time a request takes is its service time plus its time in line. When latency explodes, the service did not get slower. The line got longer. CPU cores pull work from a run queue. Network cards drain packet rings. Disks work off an I/O queue. Databases hand out connections from a pool queue. Locks keep a wait list. Your 9-second checkout was standing in one of those lines. The dashboards were green because dashboards show utilization, and the pain lives in saturation.

In 1909, a Danish engineer named Agner Erlang worked out the mathematics of waiting lines for the Copenhagen telephone exchange, and his formulas still run the internet you are using right now.

The simplest queueing result describes a single server with randomly arriving work (the model is called M/M/1: random arrivals, random service times, one server). It says:

Average time in the system = service time divided by (1 minus utilization).

In symbols: W = 1 / (1 - rho), measured in multiples of the service time, where rho is utilization from 0 to 1. Real systems are messier than the ideal model, so the exact constants vary, but the SHAPE of this curve holds everywhere, and the shape is the lesson.

Before you touch the chart, commit to a guess. A server sits at 50 percent utilization and requests take about twice their bare service time. Traffic grows until utilization hits 95 percent. Does the wait roughly double, or does something worse happen? Decide now, then drag the slider to 95 percent and check yourself.

{ "reading": "Wait time does not climb steadily. It hugs the floor, bends near 80 percent utilization, then explodes.", "caption": "W = 1 / (1 - rho) for an M/M/1 queue. Real systems vary in constants, never in shape.", "xLabel": "utilization", "xUnit": "%", "yLabel": "time in system", "yUnit": "x service time", "xDomain": [0, 0.99], "yDomain": [0, 20], "xFormat": "percent", "curves": [ { "id": "W", "label": "W(rho)", "generator": "mm1_wait" } ], "params": [ { "id": "rho", "label": "Utilization", "min": 0.05, "max": 0.98, "step": 0.01, "value": 0.5, "display": "percent" } ], "probe": "rho", "annotations": [ { "kind": "vline", "x": 0.8, "label": "the knee" } ], "readout": "At {rho} utilization, a request spends about {W} times its own service time in the system." }

If you guessed "roughly doubles," you just met the trap this lesson removes. At 50 percent utilization the multiplier is 2. At 95 percent it is 20: ten times worse, from less than double the traffic. The curve is a cliff, not a ramp. From 50 to 70 percent, almost nothing changes. From 90 to 95, the wait doubles. From 95 to 99, it quadruples. That is why systems feel fine right up until they fall over, and why engineers treat 80 percent utilization on any resource as the line you do not cross quietly. One more famous result hides here, called Little's Law: the number of jobs in the system equals the arrival rate times the time each spends inside. The line you can count is the wait you will feel.

Enough theory. The machine below is a real 4-core Linux box. First question for any queue: how many servers are working the line? For the CPU queue, that is the core count. Type this command exactly as shown:

nproc

prompt: student@linux:~$ answer: nproc output: 4 hint: Type nproc and press Enter. It prints a single number: the CPU cores this machine has.

Four cores means four servers working the run queue.

Now read the line length. Linux publishes it as the load average: the average number of processes running or waiting to run. Type this command exactly as shown:

uptime

prompt: student@linux:~$ answer: uptime output: 23:16:57 up 0 min, 0 user, load average: 0.22, 0.05, 0.02 hint: Type uptime and press Enter. The three numbers after "load average:" are 1, 5, and 15 minute averages.

The three numbers are the queue depth averaged over 1, 5, and 15 minutes. Read them against the core count you just measured: on 4 cores, a load of 0.22 means the queue is essentially empty and everyone gets a core the moment they ask. Load 4.0 would mean all four servers busy with nobody waiting: 100 percent utilization, the edge of the cliff. Anything much above 4 means processes are standing in line.

The chart below answers the question the cliff chart cannot: what happens OVER TIME when arrivals exceed capacity? Slide the offered load past 100 percent of capacity and watch the backlog. This is the simple fluid view of a queue (steady overload, no randomness), which is exactly how a backlog behaves once you are past the cliff.

{ "reading": "At or below capacity the backlog stays flat at zero. One step past capacity, the line tilts up and never comes back down.", "caption": "Backlog = (load - 1) x seconds, for offered load as a multiple of capacity. The fluid view of steady overload.", "xLabel": "time", "xUnit": "seconds", "yLabel": "jobs waiting", "xDomain": [0, 60], "yDomain": [0, 32], "curves": [ { "id": "backlog", "label": "backlog", "generator": "queue_growth" } ], "params": [ { "id": "load", "label": "Offered load", "min": 0.5, "max": 1.5, "step": 0.05, "value": 0.9, "display": "percent" } ], "readout": "At {load} of capacity, the backlog after 60 seconds is {backlog} jobs." }

That tilting line is not an abstraction. It is literally what the load average does when demand passes the core count. We proved it on this machine: we started 4 CPU-hungry processes on the 4 cores, waited two minutes, and captured the result. Then we made it 16.

Here are the two real captures, side by side. Same machine, same command, two very different situations.

With 4 hogs on 4 cores (the machine exactly full):

 23:19:08 up 2 min,  0 user,  load average: 3.57, 1.45, 0.54

With 16 hogs on 4 cores (12 processes always waiting in line):

 23:21:39 up 4 min,  0 user,  load average: 14.99, 7.20, 2.85

One honest detail before the comparison: the 1-minute number is an average, so it lags. The first capture reads 3.57, still climbing toward 4.0 when we took it; the second reads 14.99, climbing toward 16. The lag is why you always read all three numbers: 3.57 then 1.45 then 0.54 tells you this load is NEW.

Now the comparison. The first machine is at 100 percent utilization: every core busy, almost nobody waiting. It feels a little slow. The second machine is deep into saturation: the same four busy cores, but the queue is nearly four times the core count. Every new process waits through more than three full turns before it runs. It feels broken.

This is the discrimination that matters. Utilization tells you the servers are busy. Saturation tells you that YOU will wait. Dashboards almost always graph the first. Users only ever feel the second. That is how every graph stays green while checkout takes 9 seconds.

If the queue model only described CPUs, it would be a fact, not a mental model. So find it somewhere else. Every listening network socket on Linux keeps its own waiting line: connections that have arrived but that no program has accepted yet. Type this command exactly as shown:

ss -ltn

prompt: student@linux:~$ answer: ss -ltn output: State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::]:22 [::]:* hint: ss lists sockets. The flags: -l listening, -t TCP, -n numeric ports.

Map the model onto what you see. Recv-Q is the queue depth: connections waiting in line right now. Send-Q is the capacity: the longest line this socket will tolerate before it starts refusing arrivals. Arrival rate is connection attempts; the server is your program calling accept. A slow program with a full Recv-Q is the checkout mystery in miniature: the CPU can be idle while every new connection stands in a line nobody is draining. Same three numbers, completely different subsystem. That is what makes it a model.

A model becomes yours when you can state it without the lesson in front of you.

answer: queue|||queueing|||queuing|||waiting line|||line hint: The waiting line the dashboards do not graph.

Now do the one thing we cannot check: take paper and draw the model twice. Arrivals, a line, a server, a leaving job. Label the three numbers. Draw it once for the CPU run queue and once for the socket queue you just read. Nobody grades this drawing, and the research on how experts form transferable patterns says the drawing is where the model actually moves into your head. Two minutes. It matters.

Three incidents, one per step. In each one, name the line before you name the fix.

CPU at 25 percent rules out the run queue: the servers are not even busy. Something else has a line. Disks keep a queue exactly like cores do, and iostat's await column is its uptime. More cores and more memory speed up a server that is not waiting; they do nothing for a request standing in a disk line.

Pending IS the queue made visible. The scheduler is the server, pods are arrivals, and node capacity sets the service rate. Healthy nodes with a growing Pending line means arrivals exceed placeable capacity: the cluster's version of load 16 on 4 cores.

The monitor answers "is it up, is it busy" and both answers look fine. Load 19 on 4 cores means your shell waits behind roughly five full rotations of the run queue for every keystroke. The box is alive, busy, and unusable, all at once. Utilization is not saturation.

You can now read three different queues with the same four questions. This table is the whole lesson in one place.

And the model does not stop here. In the networking track, switch buffers and priority flow control are this exact model at the fabric layer. In the GPU track, NCCL channels queue collective operations between GPUs. In Kubernetes, you just met the scheduler's line. Retry storms, message brokers, lock contention: one model, everywhere. When any of those lessons arrive, you will already know the shape.

This lesson taught the model with one real capture set from one real machine. The hands-on labs for this module are being built now: you will get a live 4-core box of your own, generate real load, watch the run queue climb in real time with runqlat, and fill a socket backlog on purpose to watch connections queue. This lesson stands complete on its own, and those labs will appear right here in the module when they land.

If the command line itself still feels new, the Linux Foundations track sharpens exactly the tools this lesson leaned on. Otherwise: the next model awaits. Everything is a queue. Now you can see them.

Practice Everything is a Queue in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.