GPU Operations & NCCL Troubleshooting

Real GPUs. Real diagnostics. Real break/fix scenarios.

The only hands-on GPU operations course that gives you actual GPU virtual machines. Learn nvidia-smi, DCGM diagnostics, NCCL collective operations, and production troubleshooting on real NVIDIA hardware. From RTX 4000 workstation GPUs to A100 datacenter GPUs with NVLink. Labs run on real NVIDIA GPUs in cloud environments. Single-GPU and diagnostics labs use RTX workstation GPUs. NCCL, MIG, and advanced labs use NVIDIA A100 datacenter GPUs with NVLink. The skills and commands transfer directly to production DGX and HGX clusters. Covers NCP-AII certification domains: system bring-up, control plane, cluster testing, and troubleshooting. Approximately 69% hands-on, 31% theory and log analysis (the theory portion covers physical hardware that cannot be replicated in cloud).

8 hands-on labs across 3 modules, about 4 hours of work. Every lab runs on a real virtual machine in your browser, and checks your work on the machine itself.

Modules and labs

nvidia-smi Mastery

The single most important command in GPU operations. Learn every flag, every output field, and every monitoring mode. After this module, nvidia-smi output tells you a story instead of showing you numbers. 5 labs on RTX 4000 Ada. Estimated GPU cost: ~$3.63.

First Contact: GPU Detected, Driver Loaded - Your first GPU command. Run nvidia-smi and read the whole cockpit: driver and CUDA versions, temperature, power, memory, utilization, and the process list, on a
The GPU That Moved - Set CUDA_VISIBLE_DEVICES on a real two GPU machine and watch nvidia-smi ignore it completely, because nvidia-smi is an NVML tool and not a CUDA application. The
The Card That Slowed Down - A slow GPU is almost never a mystery, and it is almost never heat. Read power, clocks and the Clocks Event Reasons field on real hardware, then cap a card to it

NCCL Fundamentals

How GPUs communicate during distributed training. Collective operations, nccl-tests, topology detection, environment variable tuning, and debug log interpretation on real A100 hardware with NVLink. 6 labs on dual A100 NVLink. Estimated GPU cost: ~$23.95.

Reading NCCL's Mind - A training job finishes cleanly and runs slow. Switch on the NCCL debug log, find the transport selection line, reproduce the loss with NCCL_P2P_DISABLE=1, and
Reading the Fabric - Two GPUs in one machine can be neighbors or strangers. Read nvidia-smi topo -m and the P2P capability matrix on real hardware, decode PHB, PIX, PXB, NODE, SYS a
Your First All-Reduce - Run your first all-reduce on two real GPUs and learn to read every column nccl-tests prints: size, count, type, redop, time, algbw, busbw, and the correctness c

NCCL Troubleshooting

Break/fix scenarios on real multi-GPU systems. Diagnose NCCL timeouts, bandwidth bottlenecks, wrong interface selection, shared memory exhaustion, and straggler GPUs. The debugging methodology used by production AI infrastructure teams. 5 labs on dual A100 NVLink. Estimated GPU cost: ~$21.96.

The Job That Never Starts - Diagnose NCCL bootstrap failures, the class of error where a job dies before any collective runs. Read the warning, clear the hardware, find the interface NCCL
The All-Reduce That Got Slower - A training job silently lost a fifth of its throughput overnight with no error and no alert. Measure the all-reduce on two real GPUs, read the NCCL transport li

Frequently asked questions

How many labs are in GPU Operations & NCCL Troubleshooting?

GPU Operations & NCCL Troubleshooting has 8 hands-on labs, roughly 4 hours of guided work, spread across 3 modules.

Do I need my own server or hardware to do the GPU Operations & NCCL Troubleshooting labs?

No. Every lab boots a real virtual machine in your browser, with a real Linux shell. There is nothing to install, and nothing on your own machine to break.

What level is GPU Operations & NCCL Troubleshooting?

Labs in this track are rated beginner, intermediate, advanced. The track starts from the beginning, so no prior experience is assumed.