Networking for GPU clusters: InfiniBand vs Ethernet

The GPU fabric is where a cluster either scales or stalls. Collective operations (all-reduce, all-to-all) run at the speed of the slowest link and are sensitive to congestion and tail latency in a way that ordinary datacenter traffic is not. Two technologies serve this role today: InfiniBand and lossless Ethernet with RoCEv2. Both work; the trade-offs are in tooling, ecosystem and cost.

Where the fabric matters

A cluster has three networks. The compute (east-west) fabric connects GPUs across nodes and carries collectives; it needs the most bandwidth and the least latency. The storage network carries datasets and checkpoints and tolerates more jitter. The management/in-band network handles orchestration and can be plain 25/100G Ethernet. This article is about the first one.

The standard ratio for training clusters is one 400G NIC per GPU, so an 8-GPU node carries eight ConnectX-7 (or ConnectX-8 at 800G) adapters, plus a separate NIC or DPU for storage and management. Inference clusters rarely need this: a single 200-400G NIC per node is usually plenty unless you run disaggregated prefill/decode or very large MoE models across nodes.

InfiniBand

NVIDIA Quantum-2 delivers NDR 400 Gb/s per port with 64 ports per 1U switch; Quantum-X800 brings XDR 800 Gb/s. InfiniBand is credit-based and lossless by design, has hardware-offloaded collectives (SHARP) that reduce all-reduce traffic in the switch, and adaptive routing that spreads flows evenly. NCCL, the collectives library beneath PyTorch, is tuned for it first.

• Pros: predictable performance out of the box, SHARP in-network reduction, mature subnet manager (UFM), best-documented reference designs (DGX SuperPOD).
• Cons: single vendor, separate skill set from the Ethernet team, higher switch and cable pricing, no easy integration with existing L3 network fabrics.

Ethernet with RoCEv2

RDMA over Converged Ethernet runs the same verbs interface on Ethernet, but Ethernet is lossy unless configured otherwise. A working RoCE fabric needs PFC (priority flow control) and ECN/DCQCN congestion control, tuned consistently on every switch and NIC. Done right, 400G RoCE delivers all-reduce bandwidth within a few percent of InfiniBand for most job sizes; done wrong, it delivers a fraction and intermittent stalls that are hard to diagnose.

Two ecosystems dominate. NVIDIA Spectrum-X (Spectrum-4/5 switches with BlueField-3 DPUs or ConnectX-7/8 NICs) adds adaptive routing and telemetry-based congestion control that closes most of the gap to InfiniBand. Broadcom Tomahawk 5 (51.2 Tb/s) and Tomahawk 6 (102.4 Tb/s) switches from Arista, Dell, Celestica and others form the open alternative, with the Ultra Ethernet Consortium specification maturing through 2025-2026.

• Pros: multi-vendor, familiar operations, cheaper optics and switches at scale, single fabric for storage and compute if desired.
• Cons: needs careful QoS tuning and validation, tail latency under incast is worse without adaptive routing, fewer turnkey reference designs.

Topology

Both fabrics use non-blocking or lightly oversubscribed fat-tree (Clos) topologies. For GPU clusters the preferred variant is rail-optimized: GPU 0 of every node connects to leaf switch 0, GPU 1 to leaf 1, and so on. Collectives between matching GPUs then stay on one leaf and never cross the spine. A 32-node, 256-GPU pod fits in eight 64-port leaf switches plus a spine layer; up to ~2,000 GPUs is a two-tier design, beyond that three tiers.

Optics and cables

Cabling is often 10-15% of the fabric budget and the most common source of link flaps. Choose by distance.

Source Photonics, along with NVIDIA-branded and other qualified vendors, supplies 400G and 800G transceivers that pass the compatibility checks on Quantum and Spectrum switches. For InfiniBand always buy transceivers on the NVIDIA compatibility list or with confirmed firmware, as unqualified optics are the leading cause of "link up, no traffic" tickets. Linear pluggable optics (LPO) are appearing on 800G Ethernet spines and reduce power per port; ask whether the switch model supports them.

What we recommend in a BOM

Nodeforge supplies Mellanox/NVIDIA Networking switches and adapters, Source Photonics optics and structured cabling for the clusters we design. If you are choosing between fabrics, send us the GPU count and workload mix and we will price both options.