Learning notes: GPU/HPC infrastructure, and how it differs from ordinary cloud instances

These are learning notes. I have not run GPU or HPC infrastructure in production; this is my map of the area as I learn it. Where I am unsure, I say so.

My background is Linux, AWS and latency-sensitive systems. I wanted to know which of those habits carry over to GPU clusters and which ones don’t.

1. You don’t just launch one

With an ordinary instance, I assume capacity exists. With large GPU instances that assumption breaks: the instance types are in high demand and available only in some regions and zones. AWS offers Capacity Blocks for ML, which reserve GPU instances for a fixed future window. According to the docs, a block can start up to eight weeks ahead, cancellations aren’t allowed, and instances in a block are placed close together in EC2 UltraClusters. There are also On-Demand Capacity Reservations.

The ops lesson: capacity planning goes back to being a calendar exercise, a bit like booking hardware in a data centre.

2. Drivers, CUDA and the version matrix

A GPU node has more layers that must agree than a normal host: the kernel, the NVIDIA kernel driver, the CUDA toolkit or runtime the application was built against, libraries such as cuDNN and NCCL, and the framework on top. NVIDIA publishes a CUDA compatibility guide explaining which driver versions support which CUDA versions.

What I take from it: pin the whole stack together, build images rather than patching hosts, and treat a kernel update as a change that can break the driver. This is the same change discipline as anywhere else, just with a longer dependency chain.

3. NCCL and the interconnects

Distributed training spends a lot of time in collective operations, where every GPU exchanges data with every other GPU. NCCL is NVIDIA’s library for this. It uses whatever paths exist:

NVLink / NVSwitch between GPUs inside one server.
InfiniBand (or RDMA over Ethernet) between servers in on-prem clusters.
EFA (Elastic Fabric Adapter) on AWS, which provides OS-bypass networking for HPC and ML. It is used with the aws-ofi-nccl plugin.

For my world this is the interesting part: it is a network performance problem, and the slowest link sets the pace for the whole job.

4. Topology, placement and NUMA

On AWS, cluster placement groups pack instances close together for low-latency networking. Inside the box, the same ideas I already use for latency tuning apply: which CPU socket a GPU and NIC hang off, and whether traffic crosses the inter-socket link. nvidia-smi topo -m prints the GPU/NIC/CPU affinity matrix. Slurm’s gres.conf can bind GPUs to specific cores for the same reason.

This is where my Linux background seems most directly useful. NUMA locality, IRQ placement and PCIe paths matter here for the same reasons they matter for low-latency trading hosts. I want to verify how far that analogy really holds.

5. Storage: feeding the GPUs and saving checkpoints

Two patterns, as I understand them:

Reading training data needs sustained throughput across many nodes. If storage can’t keep up, expensive GPUs sit waiting.
Checkpoints are large periodic writes from all nodes at once. They are the recovery point when a job fails.

On AWS the documented options include Amazon FSx for Lustre, often linked to S3, plus local NVMe for scratch space. I don’t yet have a feel for sizing any of this.

6. Schedulers: Slurm or Kubernetes

Slurm is the traditional HPC scheduler. It treats GPUs as generic resources (GRES) and is built around batch jobs that need many nodes at the same time.
Kubernetes exposes GPUs through device plugins, requested as a resource like nvidia.com/gpu. The NVIDIA GPU Operator installs the driver, container toolkit, device plugin and DCGM exporter as managed components.

My current reading: plain Kubernetes scheduling doesn’t natively do “all N pods start together or none do”, so batch and gang-scheduling add-ons exist. Slurm does this by design.

7. Monitoring: busy is not the same as useful

DCGM is NVIDIA’s management and monitoring suite, and dcgm-exporter feeds Prometheus. The trap I keep reading about: the classic “GPU utilisation” figure only says a kernel was running during the sample period, not how much of the GPU was doing work. DCGM exposes finer profiling metrics, such as SM activity and tensor-pipe activity, that say more.

That matches a lesson I already know from CPUs: a busy metric is not a throughput metric. Measure the job’s useful output too.

8. Failure modes

Xid errors are the NVIDIA driver’s error reports in the kernel log. NVIDIA’s Xid documentation says to grep for NVRM: Xid and use nvidia-smi, DCGM diagnostics and nvidia-bug-report.sh to gather evidence.
ECC memory errors are visible through nvidia-smi and DCGM.
One bad node can stall everyone. In a synchronous training job, a single slow or failed GPU or link holds up all the others. So health checks before a job starts, and quickly draining bad nodes, matter more than in a stateless web fleet, where you just replace the instance.

9. The cost model

I won’t quote prices; they change and vary by region. The structure is what’s different. GPU nodes are expensive per hour, so idle time is the biggest waste. Reserved capacity is paid whether or not you use it. Failed jobs cost the compute time since the last checkpoint. The levers seem to be: keep GPUs fed (storage, data loading), checkpoint at a sensible interval, catch bad nodes early, and release capacity you aren’t using.

What I want to try next

Launch a single small GPU instance, install the driver and CUDA by hand, then redo it with a pre-built image, and note what breaks.
Run nvidia-smi topo -m and compare it with what lscpu and lstopo say about the same machine.
Run dcgm-exporter into Prometheus and compare “GPU utilisation” with the profiling metrics on a toy workload.
Read the NCCL tests and EFA setup guides end to end before touching a multi-node setup.
Set up a tiny Slurm cluster and a Kubernetes cluster with the GPU Operator, and compare how each handles a job that needs all its GPUs at once.

I’ll update this note as I go.