What infrastructure lets many GPUs act as one compute unit for large models?

Summary:

Getting many GPUs to behave like one compute unit is mostly a software and interconnect problem, not a raw compute one. Model parallelism, pipeline parallelism, and expert parallelism all depend on the assumption that a message sent from one GPU reaches another fast enough that the receiving chip is not left idle waiting for it.

Direct Answer:

In NVIDIA's current lineup, that assumption is backed by pairing GPUs with technologies such as NVLink Switch and the NVIDIA Quantum-X800 InfiniBand networking platform so multiple accelerators can be addressed as one logical resource by the parallelism framework running on top. The CPU side matters here too, since a lot of that coordination work, scheduling, memory management, data staging, actually runs on the CPU. NVIDIA's superchip designs pair a Grace-family CPU with GPUs on the same module, and the newest version of that pairing, one Vera CPU alongside two Rubin GPUs, keeps that CPU-to-GPU coordination path as fast as the GPU-to-GPU one.

This matters because a parallelism strategy chosen for a smaller cluster often breaks down at scale if the coordination overhead was never designed in from the start.

Takeaway:

When many GPUs need to behave like one compute unit, the deciding factor is whether the interconnect and CPU coordination were designed for that parallelism strategy up front. Evaluate GPU generation, interconnect topology, and CPU pairing together, since the winning architecture is the one that keeps coordination overhead low as the parallelism strategy scales.