Memory sizing for AI servers: DDR5 RDIMM rules of thumb

System memory is the line item that gets cut first when a GPU server quote is over budget, and the one that most often causes the box to underperform. GPUs are only as fast as the host that feeds them: data loaders, tokenizers, KV-cache offload, checkpoints and model loading all go through DDR5. This guide gives practical sizing rules for AI servers on current EPYC and Xeon platforms.

Know your channels

DDR5 bandwidth comes from parallel channels, and every socket has a fixed number of them. Leaving channels empty is the most common configuration error we see in incoming BOMs.

Peak bandwidth is channels × 8 bytes × transfer rate. A 12-channel Turin socket at DDR5-6000 delivers roughly 576 GB/s; the same socket with only 8 DIMMs installed drops to ~384 GB/s, and with 4 DIMMs to ~192 GB/s. Populate all channels on every socket, even if that means smaller DIMMs.

Rule 1: RAM to VRAM ratio

• Inference nodes: 1× to 1.5× total VRAM. A server with 8× 96 GB cards (768 GB) should carry 768 GB-1.15 TB of DDR5. This covers model loading, CPU-side KV-cache offload and multiple model versions staged in page cache.
• Fine-tuning and training nodes: 2× total VRAM. Optimizer offload (ZeRO-Offload, FSDP CPU offload), dataset shuffling and checkpoint writes need it. HGX H200 (1.1 TB VRAM) ships with 2 TB DDR5 from most OEMs for this reason.
• Data preprocessing and vector-database hosts: size to the working set, not the GPUs. 1-2 TB is typical for embedding pipelines that keep indexes hot.

Rule 2: pick capacity per DIMM from channel count

Once channels are fixed, the target capacity divides cleanly. On a dual-socket Turin board (24 channels), 1 DIMM per channel gives 24 slots: 24 × 32 GB = 768 GB, 24 × 64 GB = 1.5 TB, 24 × 96 GB = 2.25 TB. On dual-socket Xeon 6700 (16 channels): 16 × 64 GB = 1 TB, 16 × 128 GB = 2 TB. Prefer one DIMM per channel with larger modules over two DIMMs per channel: 2 DPC drops the supported speed by one or two bins on most platforms.

Rule 3: speed bins and ranks

DDR5 RDIMMs are sold at 4800, 5600, 6000 and 6400 MT/s. Buy the bin the CPU supports at your population; faster modules simply downclock. On mixed installs, the slowest module sets the speed for the whole channel, so never mix bins in one server.

• Single-rank (1Rx8) 16-32 GB: cheapest, lowest bandwidth per DIMM; fine for 1 DPC on small configs.
• Dual-rank (2Rx4/2Rx8) 32-96 GB: the default for AI servers; rank interleaving improves effective bandwidth.
• 128 GB and 256 GB modules: dual-rank with 3DS stacking; check the memory QVL, as not every board supports every 3DS part.
• MRDIMM (Xeon 6900 only): 8800 MT/s, roughly 35% more bandwidth; premium price, worth it for CPU-bound preprocessing.

ECC and RAS

All DDR5 RDIMMs carry on-die ECC and side-band ECC through the RCD; there is no non-ECC option in the server segment, which is a good thing. What varies is RAS support in the platform: memory page retirement, patrol scrubbing and PPR (post-package repair). Enable patrol scrubbing in BIOS on training nodes; a single uncorrected error mid-run costs hours.

Vendors

Samsung, Micron, SK hynix and Kingston (Server Premier) all ship DDR5 RDIMMs that are qualified on Supermicro, Dell and HPE boards. Differences are mostly in availability, warranty and lead time rather than performance. Two practical notes: OEM-branded modules (Dell, HPE) may be required for the OEM to honour full support on their servers, and Kingston Server Premier is often the fastest to source in volume from Dubai and Hong Kong stock.

What to check on the quote

Nodeforge holds Samsung, Micron and Kingston DDR5 RDIMMs from 32 GB to 128 GB in stock and configures memory as part of every server BOM we quote. If you send us the CPU and GPU choice, we will return a populated memory map, not just a capacity number.