How to design an AI Cluster

1. 1. Workload: why training, fine-tuning and inference need different clusters
2. 2. Facility: how much power and cooling an AI cluster rack needs
3. 3. Compute: how to choose GPUs and NICs for an AI cluster
4. 4. Network: designing the back-end AI fabric, endpoint-scheduled vs fabric-scheduled
5. 5. Storage: how to size checkpoint and dataset throughput per GPU
6. 6. Validation: write AI cluster acceptance tests before hardware arrives
7. 7. Operations: the part that decides year two
8. What changes if you are building for inference
9. The short version: the seven decisions that set AI cluster performance

1. Workload: why training, fine-tuning and inference need different clusters

Decide this before you price anything! Training, fine-tuning and inference are three different clusters wearing the same name. Training is bursty, synchronized, intolerant of failure, and gated by collective performance. Fine-tuning is a smaller version of the same machine. Inference is request/response, stateless, and gated by model load time and KV-cache behavior rather than by collectives. Sizing one for the other is the most expensive mistake on this list.

Building an inference cluster? The facility, compute, and operations sections still apply. Only the fabric and storage needs change. A short note at the end explains how.

Pin down: parameter count, precision, parallelism strategy (TP/PP/DP/EP), target time-to-train, tenancy model.

Rule of thumb: size the network for the collective, the storage for the checkpoint, and the power for the sustained load, never the idle draw or the peak spec.

2. Facility: how much power and cooling an AI cluster rack needs

Facility is the constraint that is actually binding! Why? Most AI workloads are gated by power and cooling long before accelerator supply. Rack density has stepped from ~5–10 kW in traditional DC design to 30–60 kW for GPU racks, and 100–150 kW for the newest liquid-cooled designs. Air is practical to roughly 20 kW/rack and marginal to ~35 kW, so the cooling method is a big decision, not just another line item. Design for synchronized power swings too: thousands of GPUs entering and leaving a collective together create step loads your UPS and generators must ride through.

Cabling is an architecture decision, not just a purchase. DAC/AEC, AOC and optics differ by an order of magnitude in cost, power per port and failure rate, so switch placement that minimizes optics count is worth real money, and pre-terminated, labelled trunks separate a two-week bring-up from a ten-week one. Three physical decisions you cannot cheaply reverse: rack power density, cooling medium, cable architecture.

3. Compute: how to choose GPUs and NICs for an AI cluster

Compute is bought by the rack, not the node. Pick compute based on the specs that matter, not the brand. Look at memory bandwidth and capacity, interconnect bandwidth per GPU, power envelope, software-stack maturity, and lead time. Then get the boring details right, because they fail silently:

• One NIC per GPU, rail-aligned, with PCIe and NUMA topology verified rather than assumed. A mis-mapped NIC costs 20–30% of fabric performance and reports nothing.
• Firmware, BIOS and driver uniformity as a hard requirement. One drifted node degrades the whole collective.
• An out-of-band management network: notionally optional, never optional in practice.

4. Network: designing the back-end AI fabric, endpoint-scheduled vs fabric-scheduled

Network is where clusters are won or lost. You are building five networks, not one. Scale-up, scale-out back-end (the collectives), front-end, storage, and out-of-band. Oversubscription is acceptable on the front-end and storage fabrics, but on the back-end it must be explicit, tested and owned.

AI traffic breaks the assumptions classic DC fabrics were built on: few flows, enormous flows, perfectly synchronized. ECMP hashing collapses under exactly that profile, which is why load balancing, not port speed, is the real design variable.

There are two main network architectures, and they are not interchangeable:

Network Architecture	Characteristics
Endpoint-scheduled	NIC-driven packet spraying, path-aware congestion control, out-of-order receive with selective retransmit. Flowlet-style, typically with PFC disabled.
Fabric-scheduled	Ingress virtual output queues, credit-based end-to-end scheduling, cell spraying, no dependence on NIC capability. Per-packet spray of this kind requires PFC end-to-end.

Running one family’s flow-control configuration underneath the other family’s spraying is an expensive misconfiguration, and a common one. “We enabled PFC” is not a design.

The choice is commercial as well as technical: endpoint scheduling makes fabric behavior a property of the NIC you bought, while fabric scheduling keeps it a property of the network. DriveNets AI Fabric supports both approaches, using cell-based spraying across distributed white boxes operated as one logical switch, so there is no ECMP polarization to tune and no dependence on endpoint capability. Either way, choose deliberately rather than inheriting the design of your front-end network.

Plan tier count as a scale threshold rather than a preference: two tiers below roughly 8,000 GPUs, three above. And when one building runs out of power and cooling, the cluster spans sites. Scale-across changes buffer requirements at the DCI leaf and belongs in the design from day one.

The fabric and the job scheduler must also agree on topology, or every other decision here is wasted.

Measure tails, not averages: p99/p99.9 latency, jitter, and JCT variance across repeated identical runs.

5. Storage: how to size checkpoint and dataset throughput per GPU

Storage is sized for the burst, not the average.

Storage serves two workloads pulling in opposite directions: dataset reads (many small random reads, cacheable) and checkpoint writes (enormous synchronized bursts that stall the cluster until they complete).

Reference points: roughly 0.6 GB/s per GPU sustained for training and the data pipeline, rising to 1–4 GB/s per GPU where KV-cache offload is in play. Back-calculate aggregate GB/s from your checkpoint window as a percentage of step time, then tier deliberately: local NVMe scratch → parallel file system → object capacity → archive. Metadata performance is the silent killer for small-file datasets.

On convergence, the honest rule: keep storage traffic off the back-end fabric unless the fabric can demonstrate isolation under test. Scheduled fabrics with ingress VOQs generally can, and where they do, you remove an entire duplicate network, and the TCO case is real.

6. Validation: write AI cluster acceptance tests before hardware arrives

Define your pass/fail before hardware arrives. Acceptance should be contractual, and you need to sequence it:

Phase	Acceptance Requirements
Physical & Link	Verify sustained-load power draw, thermal soak, and leak test. Every link must be at full rate with clean error counters, optics DOM in spec, and no flapping over 24 hours.
Node & Fabric	Check firmware uniformity, GPU health, NUMA/rail verification, and microbenchmarks. Verify pairwise bandwidth matrix, all-reduce bus bandwidth versus theoretical, and congestion under load.
Isolation & Resilience	Run the same collective with and without background RDMA noise to ensure overlapping curves. Maintain sustained load for ten minutes while flapping ~5% of interfaces for five seconds each. Watch degradation and recovery time.
Application	Execute a real training run, repeated. JCT variance is the acceptance criterion.

 

7. Operations: the part that decides year two

AI cluster operations require correlating GPU, host, fabric, power and cooling telemetry on one timeline, streamed rather than polled. The requirement is answering “why was this run 12% slower?” in minutes, not days, so watch per-queue congestion and PFC counters, ECN marks, link errors, per-step collective completion time, and GPU busy-but-waiting rather than GPU utilization.

Enforce configuration against a reference design rather than merely storing it as code. Isolate tenants properly, with VLAN-over-VXLAN as the baseline and EVPN Type-5 with per-tenant VRFs where isolation must be real. And plan for inserting a new accelerator generation without redesigning the fabric: silicon turns over faster than DC depreciation.

What changes if you are building for inference

Facility, compute and operations are almost unchanged. Three things are different:

• Fabric priority shifts from lossless collective bandwidth to latency and tenant isolation, so oversubscription that would be indefensible on a training back-end is often perfectly reasonable here.
• Failure tolerance inverts. Replicas are stateless, so losing a node costs a few in-flight requests rather than a restart from the last checkpoint. That changes what redundancy is worth paying for.
• Storage is sized by the cache, not the checkpoint. Model load time and KV-cache offload (HBM to CPU memory to SSD to network) set the requirement, and a central cache beats a node-local one on capacity ceiling, survivability across node failure, and sessions that reschedule onto a different server.

The short version: the seven decisions that set AI cluster performance

An AI cluster is a single machine, not a collection of servers. Design it as one system, or the weakest component sets job completion time (JCT), the only score that matters. Seven decisions make it one system:

1. Pick the workload before pricing anything. Training, fine-tuning and inference are three different clusters wearing the same name, and sizing one for the other is the most expensive mistake on this list.
2. Facility is the binding constraint. Most AI workloads are gated by power and cooling long before accelerator supply: GPU racks draw 30–60 kW against a traditional data center’s 5–10 kW, and the newest liquid-cooled designs 100–150 kW. Rack power density, cooling medium and cable architecture are the three physical decisions you cannot cheaply reverse.
3. Buy compute by the rack, not the node. One NIC per GPU, rail-aligned, with PCIe and NUMA topology verified rather than assumed, because a mis-mapped NIC costs 20–30% of fabric performance and reports nothing.
4. On the back-end fabric, load balancing is the real design variable, not port speed. AI traffic is few flows, enormous, and perfectly synchronized, which is the exact profile that collapses ECMP hashing. Endpoint-scheduled and fabric-scheduled designs both work, but running one family’s flow control underneath the other family’s spraying is an expensive and common misconfiguration.
5. Size storage for the checkpoint burst, not the average. Roughly 0.6 GB/s per GPU sustained for training and the data pipeline, rising to 1–4 GB/s per GPU where KV-cache offload is in play, back-calculated from the checkpoint window as a percentage of step time.
6. Define pass/fail before the hardware arrives. Acceptance runs physical, link, node, fabric, isolation, resilience, application, and JCT variance across repeated identical runs is the criterion.
7. Instrument for year two. Correlate GPU, host, fabric, power and cooling telemetry on one timeline, streamed rather than polled, so “why was this run 12% slower?” is answered in minutes, not days.

Performance depends more on how the parts are integrated than on which parts you pick. Each component can be strong on its own, yet the cluster still underperforms if the parts do not work together as one system. That is why component-by-component procurement can result in an underutilized cluster.

Working through this list for a build of your own? Talk to DriveNets Infrastructure Services (DIS).

Frequently Asked Questions

What are the power and cooling requirements for an AI cluster?

Designing an AI cluster requires planning for extreme rack density and synchronized power swings. Traditional data center design supports 5–10 kW, but modern GPU racks demand 30–60 kW, reaching 100–150 kW for the newest liquid-cooled designs. Air cooling remains practical up to 20 kW per rack but becomes marginal around 35 kW.

How do endpoint-scheduled and fabric-scheduled AI networks differ?

Endpoint-scheduled networks rely on NIC-driven packet spraying and path-aware congestion control. In contrast, fabric-scheduled networks use ingress virtual output queues, credit-based end-to-end scheduling, and per-packet cell spraying that requires Priority Flow Control (PFC). Mixing flow-control configurations between these distinct architectures is a common and highly expensive misconfiguration.

How should storage be sized for AI training workloads?

Storage in an AI cluster must be sized for massive, synchronized checkpoint bursts rather than average dataset reads. Training workloads require roughly 0.6 GB/s per GPU sustained, which increases to 1–4 GB/s per GPU when KV-cache offload is active. Storage traffic should stay off the back-end fabric unless strict isolation is demonstrated.