Choosing GPUs for LLM inference vs training

Most GPU purchasing mistakes come from one shortcut: buying the "fastest" card for a workload that is limited by something other than compute. LLM inference and LLM training stress different parts of a GPU, and the right card for one is often a poor fit for the other. This guide walks through the questions that actually decide the choice.

Start with the model, not the GPU

Write down three numbers before looking at any spec sheet: parameter count, the precision you will run at, and the context length you need to serve. Weights alone take roughly 2 bytes per parameter at FP16/BF16, 1 byte at FP8 or INT8, and about 0.5 byte at FP4/NVFP4. A 70B model is therefore ~140 GB at BF16, ~70 GB at FP8 and ~35 GB at 4-bit, before anything else is loaded.

The "anything else" is the KV-cache. Every token in flight keeps its key/value activations resident for every layer. For a 70B-class model at FP16 that is on the order of 300 KB per token; 32 concurrent requests at 8K context add roughly 80 GB on top of the weights. Long-context serving and high concurrency are where memory runs out first, not where compute runs out.

Inference: memory capacity and bandwidth

Decode is bandwidth-bound. Each generated token reads the full weight set (or the active experts for MoE models) from memory. Tokens per second per user scale with memory bandwidth divided by bytes touched per token, which is why HBM cards keep a lead in latency-sensitive serving even when their raw TFLOPS look similar to GDDR parts.

• HBM3e (H200 NVL: 141 GB at 4.8 TB/s; B200: up to 180-192 GB at ~8 TB/s) — highest bandwidth and capacity per card, highest price and power.
• GDDR7 (RTX PRO 6000 Blackwell: 96 GB at ~1.6 TB/s) — the current sweet spot for single-node inference of 30B-120B models at FP8/FP4.
• GDDR6 (L40S: 48 GB at 864 GB/s) — still cost-effective for models up to ~30B at FP8, vision and mixed workloads.

Batching changes the equation. With prefill and large batches the workload becomes compute-bound, and the FP8/FP4 tensor throughput of Blackwell matters. If you serve many short requests through a scheduler such as vLLM, TensorRT-LLM or SGLang, a GDDR card with strong FP8/FP4 rates delivers more tokens per dollar than its bandwidth number suggests.

Training and fine-tuning: bandwidth and interconnect

Training keeps weights, gradients and optimizer state in memory at the same time. With mixed precision and Adam-style optimizers, plan for 12-16 bytes per parameter before activations. Full fine-tuning of a 7B model already wants 100+ GB; a 70B model needs a multi-GPU node with sharding (FSDP, ZeRO-3) regardless of how large a single card is.

That sharding is where the interconnect decides everything. Every step exchanges gradients across all GPUs. Over PCIe 5.0 x16 a card sees ~64 GB/s each direction; over NVLink on an HGX or DGX system each GPU has 900 GB/s (Hopper) or 1.8 TB/s (Blackwell) to its peers. For pre-training or long fine-tuning runs, the NVLink system finishes the same job in a fraction of the wall-clock time and usually costs less per experiment.

• LoRA / QLoRA on models up to ~70B: a single 96 GB or 141 GB PCIe card, or two cards with an NVLink bridge, is enough.
• Full fine-tuning of 7B-13B models: 4-8 PCIe cards work; NVLink bridges on H200 NVL help but are not mandatory.
• Full fine-tuning or pre-training above 30B: HGX H200 / B200 (8× SXM, NVSwitch) or GB200/GB300 NVL systems.

Precision: FP8, FP4 and what they buy you

Hopper and Ada introduced hardware FP8; Blackwell adds FP4 (NVFP4 with block scaling). For inference, FP8 is now routine with negligible quality loss on most open models; FP4 is production-ready for many chat and RAG deployments but should be validated against your own evals before it becomes the default. For training, BF16 with FP8 for selected matmuls is the mainstream recipe; FP4 training remains research territory in 2026.

NVLink vs PCIe in practice

PCIe cards go into ordinary 2U-4U servers, share the system with other workloads, can be bought in ones and twos and are easy to redeploy. SXM/NVLink systems are sold as complete 8-GPU nodes, draw 10-14 kW each, need matching power and cooling, and are allocated by the manufacturers rather than pulled from shelf stock. If you will run inference for the next two years, PCIe is usually right. If you will train, the node is usually right.

Checklist before you order

• Weights + KV-cache at your target concurrency fit in memory with 15-20% headroom.
• Bandwidth per GPU supports your latency target for decode at batch 1 and at typical batch.
• Server has the PCIe lanes (x16 per card), airflow for passive cards and PSU headroom at full TDP.
• Driver, CUDA and inference stack versions support the card generation you are buying.
• Lead time and allocation: PCIe workstation/server cards ship faster than HGX nodes.

Nodeforge stocks H200 NVL, RTX PRO 6000 Blackwell Server Edition, L40S and workstation cards in Dubai and Hong Kong, and sources HGX and GB-series systems through direct manufacturer allocation. Tell us the model and traffic you expect, and we will size the node with you.