{a.num}
{a.kicker}
{a.title}
{a.desc}
{a.tags.map((t) => {t})}
{a.readTime}
Read article
This reference architecture defines an 8-rail, non-blocking RoCEv2 compute fabric for NVIDIA H200 and B200 GPU servers, using 400G/OSFP end-to-end with a two-tier Leaf/Spine topology of 128×400G platform switches. The design is rail-aligned: each of a server's 8 GPU NICs lands on a dedicated TOR ("rail") so that same-rank collectives traverse a single switch hop, eliminating cross-rail oversubscription for NCCL ring/tree primitives at intra-pod scale.
The architecture scales linearly from a 128-GPU pod (single TOR per rail) to a 8,192-GPU AI factory (16 pods × 8 rails) under a single addressing and routing domain, with deterministic optics and power budgets at every step. Total fabric power tracks at ~1.33 kW per GPU node-equivalent across the scale band.
Each Server Group consists of 64 GPU servers (512 GPUs) front-ended by 8 TOR switches, one per rail. NIC on every server in the group connects only to TORi — the rail-aligned plane. The 8 TORs of a group fan up north to the Leaf layer in a full mesh, with up to 16 groups (MAX) per pod under the 64-leaf footprint shown below.
The table below codifies the deterministic BOM for every scale tier. All rows derive from three invariants: 8 GPUs/server, 8 NICs/server (1×400G each), and 128-port 400G platform switches at both TOR and Leaf. Optics quantities are split by segment (NIC↔TOR vs TOR↔Leaf) and by media type.
| Type | Component | GPU scale |
|---|---|---|
|
{fmt(p.gpus)}
GPUs
|
)}
||
| Compute | GPU chips (8 per server) | {SCALE_PROFILES.map((p) =>{fmt(p.gpus)} | )}
| AI servers (8×H200 / 8×B200) | {SCALE_PROFILES.map((p) =>{fmt(p.servers)} | )}|
| Network cards (8 × 1×400G per server) | {SCALE_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| Switching 128 × 400G |
Leaf | {SCALE_PROFILES.map((p) =>{p.leaf === 0 ? '—' : fmt(p.leaf)} | )}
| TOR | {SCALE_PROFILES.map((p) =>{fmt(p.tor)} | )}|
| Cabinet | GPU server rack (42U) | {SCALE_PROFILES.map((p) =>{fmt(p.gpuRack)} | )}
| Switch rack (42U) | {SCALE_PROFILES.map((p) =>{fmt(p.swRack)} | )}|
| Optics & jumpers |
NIC ↔ TOR 400G-OSFP-SR4 | {SCALE_PROFILES.map((p) =>{fmt(p.nics)} | )}
| NIC ↔ TOR MPO-MPO-OM4 | {SCALE_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| NIC ↔ TOR 400G-QSFP112-SR4 | {SCALE_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| TOR ↔ Leaf 400G-QSFP112-SR4 | {SCALE_PROFILES.map((p) =>{p.leaf === 0 ? '—' : fmt(p.nics)} | )}|
| TOR ↔ Leaf MPO-MPO-OM4 | {SCALE_PROFILES.map((p) =>{p.leaf === 0 ? '—' : fmt(p.nics)} | )}|
| TOR ↔ Leaf 400G-QSFP112-SR4 | {SCALE_PROFILES.map((p) =>{p.leaf === 0 ? '—' : fmt(p.nics)} | )}|
| Power | Total (server + switch) kW | {SCALE_PROFILES.map((p) =>{p.power.toLocaleString('en-US', { minimumFractionDigits: 1, maximumFractionDigits: 1 })} | )}
8 × H200 or B200 SXM GPUs per chassis with NVLink/NVSwitch for intra-node collectives. Each GPU is paired 1:1 with a ConnectX-7 / BlueField-3 400G NIC, BDF-affined to the GPU's NUMA / PCIe root complex. GPUDirect RDMA is mandatory; PCIe Gen5 ×16 per NIC.
128-port 400G platform (e.g. Spectrum-4 SN5600 class). 64 ports south to 64 servers, 64 ports north to Leaf. RoCEv2 with PFC + ECN, DCQCN tuned, lossless on the GPU traffic class.
Same 128 × 400G platform used as Leaf. Within a 16-group pod, 64 Leaves × 128 ports = 8,192 downlinks, exactly matching 128 TOR × 64 uplinks. No oversubscription, no port stranding.
400G-OSFP-SR4 on the NIC side, 400G-QSFP112-SR4 on the switch side, joined by MPO-12 OM4 trunks. Reach: 100 m on OM4. Substitute DR4 / LR4 for cross-hall runs.
Per-rack envelope sized for ~10.2 kW/server at sustained load. 8 servers per 42U GPU rack drives ≥ 80–100 kW rack density — liquid-assist or rear-door heat exchangers assumed at and above the 512-GPU tier.
Separate 1G/10G management plane (NMIX / BCM / NetQ) on dedicated OOB switches in the switch rack — never folded into the compute fabric. BMC, switch console, and PDU telemetry isolated from RoCE traffic.
This reference architecture defines a 4-rail, non-blocking RoCEv2 compute fabric for the NVIDIA GB200 NVL72 rack-scale platform. Each NVL72 is treated as a single compute unit — 72 Blackwell GPUs interconnected via a 5th-gen NVLink Switch fabric at 1.8 TB/s per GPU — and front-ended on the scale-out fabric by 72 × 400G RoCE NICs (one NIC per GPU) arranged in 4 rail planes of 18 NICs each.
The design scales from a 144-GPU pair (2 NVL72 racks, one rail-spine pair) to a 6,912-GPU AI factory (96 NVL72 racks). At full scale, the compute fabric comprises 128 TOR + 54 Leaf switches with ~12.1 MW of compute load behind a single addressing and routing domain. Per-rack density lands at ~126 kW — liquid cooling is non-optional from the first rack.
With NVL72, the scale-up domain expands from 8 GPUs (HGX) to 72 GPUs (rack). That shifts the boundary between NVLink (intra-rack, all-to-all at 1.8 TB/s) and RoCE (inter-rack, scale-out). Three consequences for fabric design:
Each NVL72 rack exposes 72 NICs split across 4 rail planes (NICs 1–18 → Rail 1, 19–36 → Rail 2, …). Within a pod, each rail plane is served by its own set of TORs that aggregate up to the Leaf layer in a full mesh. Cross-rail traffic is the exception — addressed by the Leaf layer when it occurs.
The table codifies the deterministic BOM for every scale tier. All rows derive from three invariants: 72 GPUs per NVL72, 72 × 400G NICs per NVL72, and 128-port 400G platform switches at both TOR and Leaf. Optics counts are reported per segment (NIC↔TOR vs TOR↔Leaf) and per media type.
| Type | Component | GPU scale |
|---|---|---|
|
{fmt(p.gpus)}
GPUs
|
)}
||
| Compute | GPU chips (72 per NVL72 rack) | {GB200_PROFILES.map((p) =>{fmt(p.gpus)} | )}
| AI servers (NVL72 sets) | {GB200_PROFILES.map((p) =>{fmt(p.servers)} | )}|
| Network cards (72 × 1×400G per NVL72) | {GB200_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| Switching 128 × 400G |
Leaf | {GB200_PROFILES.map((p) =>{fmt(p.leaf)} | )}
| TOR | {GB200_PROFILES.map((p) =>{fmt(p.tor)} | )}|
| Cabinet | GPU server rack (NVL72) | {GB200_PROFILES.map((p) =>{fmt(p.gpuRack)} | )}
| Switch rack (42U) | {GB200_PROFILES.map((p) =>{fmt(p.swRack)} | )}|
| Optics & jumpers |
NIC ↔ TOR 400G-OSFP-SR4 | {GB200_PROFILES.map((p) =>{fmt(p.nics)} | )}
| NIC ↔ TOR MPO-MPO-OM4 | {GB200_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| NIC ↔ TOR 400G-QSFP112-SR4 | {GB200_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| TOR ↔ Leaf 400G-QSFP112-SR4 | {GB200_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| TOR ↔ Leaf MPO-MPO-OM4 | {GB200_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| TOR ↔ Leaf 400G-QSFP112-SR4 | {GB200_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| Power | Total (server + switch) kW | {GB200_PROFILES.map((p) =>{p.power.toLocaleString('en-US', { minimumFractionDigits: 1, maximumFractionDigits: 1 })} | )}
72 Blackwell GPUs (36 GB200 superchips × 2 GPUs) interconnected by 9 NVSwitch trays. Intra-rack NVLink bandwidth is 1.8 TB/s per GPU, 130 TB/s rack aggregate. Liquid cooling (CDU + manifold) is integrated; air-cooled NVL72 variants are not in scope.
72 × ConnectX-7 / BlueField-3 NICs at 400G — one per GPU, PCIe Gen5 ×16. NICs are partitioned into 4 rail planes (18 NICs per rail). Each NICi bonds to TORrail(i) on its rail plane. GPUDirect RDMA enabled per-NIC.
128-port 400G platform (Spectrum-4 SN5600 class) operating in rail-pinned mode. 64 ports south to NVL72 racks (rail-aligned), 64 north to Leaf. RoCEv2 with PFC + ECN; DCQCN tuned for high-radix bursts typical of MoE routing.
Same 128 × 400G platform reused at Leaf. Inter-rail traffic (rare under rail-aligned collectives) is resolved one Leaf hop above the rail plane. At 54 Leaves × 128 ports, a 6,912-GPU pod fits in a single addressing / routing domain.
400G-OSFP-SR4 NIC-side, 400G-QSFP112-SR4 switch-side, joined by MPO-12 OM4 trunks. With 72 NICs per rack, structured cabling is racked into the NVL72 itself — no top-rack jumper farms. Cable bundle counts dominate the deployment plan.
NVL72 rack envelope at ~126 kW; deployment requires liquid-cooled CDU capacity at the row, redundant 415V/3-phase feeds, and structural floor loading review. Adjacent switch racks designed as air-cooled, hot-aisle-contained at 6–10 kW each.
This reference architecture defines a two-tier Spine–Leaf RoCEv2 fabric built on Broadcom Tomahawk 5 (TH5) 64×800G platforms, sized for NVIDIA GB300 NVL72 rack-scale systems. Every GPU is fronted by its own 800G OSFP NIC — doubling per-GPU scale-out bandwidth vs the 400G (GB200) reference — with optics, switches, and structured cabling sized to a single deterministic BOM at every scale tier.
The design scales from a 144-GPU pair (2 NVL72 racks, 3 spines / 6 leaves) to a 1,728-GPU pod (24 NVL72 racks, 24 spines / 48 leaves) under a single non-blocking fabric domain. TH5 switches operate at 51.2 Tb/s per box, collapsing the rail layer and enabling a flat Spine–Leaf topology with full 1:1 oversubscription end-to-end.
Two technology shifts let this design drop the rail/TOR layer used in 400G references and run a clean Spine–Leaf fabric:
Each NVL72 rack exposes 72 OSFP NICs at 800G — one per Blackwell GPU. NICs land on a set of 800G Leaf switches; Leaves fan north to a smaller set of 800G Spine switches in a full mesh. Within the pod, the fabric is 1:1 non-blocking end-to-end.
The table codifies the deterministic BOM for every scale tier. All rows derive from three invariants: 72 GPUs per NVL72, 72 × 800G OSFP NICs per NVL72, and 64-port 800G TH5 platform switches at both Leaf and Spine. Per‑link optics are 2× MPO-12 OM4 trunks + 2× 800G-OSFP transceivers, which is why MPO counts run at 2× the per-link count.
| Type | COMPONENT | GPU scale |
|---|---|---|
|
{fmt(p.gpus)}
GPUs
|
)}
||
| Compute | GPU chips (72 per NVL72 rack) | {GB300_PROFILES.map((p) =>{fmt(p.gpus)} | )}
| AI servers (NVL72 sets) | {GB300_PROFILES.map((p) =>{fmt(p.servers)} | )}|
| RoCE network cards (72 × 1×800G per NVL72) | {GB300_PROFILES.map((p) =>{fmt(p.nics)} | )}|
| Switching 64 × 800G TH5 |
Spine | {GB300_PROFILES.map((p) =>{fmt(p.spine)} | )}
| Leaf | {GB300_PROFILES.map((p) =>{fmt(p.leaf)} | )}|
| Transceivers & cables |
NIC ↔ Leaf 800G-OSFP | {GB300_PROFILES.map((p) =>{fmt(p.oOsfp)} | )}
| NIC ↔ Leaf MPO-MPO-OM4 (2× per link) | {GB300_PROFILES.map((p) =>{fmt(p.oMpo)} | )}|
| NIC ↔ Leaf 800G-OSFP (switch-side) | {GB300_PROFILES.map((p) =>{fmt(p.oOsfp2)} | )}|
| Leaf ↔ Spine 800G-OSFP | {GB300_PROFILES.map((p) =>{fmt(p.uOsfp)} | )}|
| Leaf ↔ Spine MPO-MPO-OM4 (2× per link) | {GB300_PROFILES.map((p) =>{fmt(p.uMpo)} | )}|
| Leaf ↔ Spine 800G-OSFP (spine-side) | {GB300_PROFILES.map((p) =>{fmt(p.uOsfp2)} | )}
72 Blackwell Ultra GPUs interconnected by NVSwitch trays. NVLink 5 delivers 1.8 TB/s per GPU intra-rack, 130 TB/s rack aggregate. Integrated liquid cooling (CDU + manifold) is mandatory; per-rack envelope tracks ≥ 130 kW.
72 × ConnectX-8 / BlueField-3 NICs at 1 × 800G OSFP, one per GPU on PCIe Gen6 ×16. GPUDirect RDMA, per-NIC PFC/ECN, and SR-IOV partitioning enabled. NIC BDF aligned to GPU NUMA / NVSwitch domain.
64 × 800G TH5 (Tomahawk 5, 51.2 Tb/s). 32 ports south to NVL72 NICs, 32 ports north to Spine. RoCEv2 with PFC + ECN; DCQCN tuned for the 800G loss‑less class.
Same 64 × 800G TH5 platform reused at Spine. Full mesh to Leaves; ECMP across N Spines provides bisection. At 24 Spines, the 1,728 GPU pod sits in a single fabric domain.
800G-OSFP on both ends, joined by 2 × MPO-12 OM4 trunks per link (each carries one 400G half). SR8 reach ≤ 100 m on OM4. Cross-hall runs use 2× 400G-DR4 or 800G-DR8 single-mode.
Dedicated OOB management plane (BMC, BCM, NetQ) on a separate 1G/10G fabric. Storage fabric on a dedicated DPU-attached plane — never folded into the compute RoCE plane.
This reference architecture defines a dual-plane, dual-port 400G RoCEv2 compute fabric for NVIDIA B300 GPU servers equipped with ConnectX-8 (CX8) dual-port 400G NICs. The design splits the scale-out fabric into two physically and logically independent planes (Plane 1, Plane 2) with a Cross-Plane (CP) tier bridging the two when bisection bandwidth is required. Each GPU server's NICs are bonded across both planes — half the 400G ports land on Plane 1, half on Plane 2 — so the loss of an entire plane degrades, rather than halts, production training jobs.
The intent is job-level fault tolerance for AI factory operators: an unplanned spine, leaf, TOR, optic, or maintenance window on either plane removes capacity, not availability. NCCL topology hints, GPUDirect RDMA, and PFC/ECN are tuned per-plane so the collective scheduler re-routes survivors without restarting the job. Bisection between planes is provided by the Cross-Plane switches, which the collective uses only when a partition genuinely spans both planes — keeping the steady-state east-west pattern plane-local and 1:1 non-blocking.
Three tiers per plane: TOR → Leaf → Cross-Plane. Within a plane, each block of GPU servers homes to a TOR cluster (8 TORs per cluster, 2 clusters per plane). Plane-internal bisection is delivered by a leaf row that full-meshes to its 16 plane TORs at 400G. The Cross-Plane tier sits above both plane-leaves and full-meshes across Plane 1 and Plane 2 leaves — that is the only tier carrying inter-plane traffic. From the GPU NIC's perspective, the fabric is a flat plane-local L3 RoCE domain; ECMP across leaves and the CP layer is the sole multipath strategy.
Each B300 GPU server carries 8 × CX8 dual-port 400G NICs (16 × 400G physical ports per server, 6.4 Tb/s of scale-out bandwidth). Ports are striped across planes by physical NIC, not by logical link: NIC1..4 port-0 → Plane 1 TORi, NIC1..4{' '} port-1 → Plane 2 TORi; NIC5..8 mirror this on the second TOR cluster. The result: any single TOR, leaf, CDU, or PDU failure removes at most one plane's worth of bandwidth for any affected server — never both. NCCL NCCL_IB_HCA + topology hints surface the plane label so the runtime ranks NICs by plane health before each step.
8 × Blackwell Ultra (B300) SXM GPUs per chassis with NVLink 5 / NVSwitch for intra-node collectives — 1.8 TB/s per GPU NVLink bandwidth. Each GPU is paired 1:1 with a ConnectX-8 (CX8) dual-port 400G NIC on PCIe Gen6 ×16, BDF-affined to the local NUMA. GPUDirect RDMA, per-NIC PFC/ECN, SR-IOV, and per-VF rate-limiting enabled.
Each CX8 exposes 2 × 400G OSFP-RHS ports as a single hardware function. Port-0 homes to a Plane 1 TOR, port-1 to a Plane 2 TOR, never to the same plane. NCCL sees both as independent rails; NCCL_CROSS_NIC=0 within plane, =2 across planes so the runtime prefers plane-local paths and only crosses on bisection-bound collectives. PFC/ECN tuned per port; DCQCN profile pinned per plane.
64 × 800G TH5 class platform (51.2 Tb/s, breakable to 128 × 400G). 64 × 400G ports south to 64 B300 servers (one port per server per plane), 64 × 400G north to plane Leaf. RoCEv2 with PFC + ECN, lossless on the GPU class, ECN-only on the storage class. Plane-local control plane — each plane runs its own BGP-EVPN/L3 RoCE domain.
Same 64 × 800G TH5 platform reused at Leaf, configured as 128 × 400G logical ports. Within one plane: 16 TORs × 64 north-ports = 1,024 uplinks served by 8 Leaves × 128 ports at 1:1. Inter-plane (Cross-Plane) uplinks are reserved on dedicated ports — never mixed with intra-plane bisection on the same port group.
The CP tier is the only fabric element that physically connects Plane 1 and Plane 2. N × 64 × 800G TH5 CP switches full-mesh to all Leaves in both planes. ECMP across CP switches resolves any-source / any-destination bisection across planes. CP is sized for the realistic cross-plane working set — typically 1:2 oversubscribed{' '} relative to plane-local capacity, since correctly-tuned NCCL keeps > 90% of collective traffic plane-local.
400G-OSFP-DR4 NIC-side for in-hall reach (≤ 500 m on single-mode), 400G-OSFP-SR4 on shorter MMF runs. Switch-side 800G-OSFP split via 2 × MPO-12 OM4 for 400G-DR breakouts, or 1 × MPO-16 for native 2 × 400G when leaf/TOR sits in adjacent rows. Plane separation is enforced physically — distinct cable trays, distinct PDU feeds, distinct row groupings.
B300 server envelope ~14–16 kW sustained at 8-GPU class load; per-rack density at 6 servers/rack drives ≥ 90 kW rack power. Liquid-to-chip CDU capacity at the row mandatory; redundant 415V/3-phase feeds, A+B split between PDUs that must not cross planes (so a single PDU loss never blacks out both planes for a server).
Dedicated 1G/10G OOB management plane (BMC, BCM, NetQ) — itself dual-homed but kept off the compute fabric. Storage fabric on a separate DPU-attached plane (BlueField-3 class) with its own switches; storage is never folded into the compute RoCE plane.
The B300 generation puts 2 × 400G per GPU on the table at the NIC. A single-plane fabric would consume that bandwidth on a single failure domain; a dual-plane fabric spends it on simultaneous bandwidth and resilience. Three properties make this the right design for AI factory operators:
Technical article body for {article.title} will appear here. This canvas is scaffolded and ready to receive the architecture write-up — section headings, diagrams, tables, and reference notes.
Technical articles covering the full network design and intent architecture.
{a.desc}