What Is an AI Network Fabric? Why Adding GPUs Doesn't Speed Up Training
"If you 10x your GPUs, does training get 10x faster?"
In theory, more GPUs handle more compute. In practice, real GPU clusters often don't scale linearly as you add hardware. Adding GPUs doesn't just add compute — it adds more data and synchronization that the GPUs have to exchange with each other. When communication can't keep up with computation, the GPUs you just added spend more time waiting for data than actually computing.
What matters in the end isn't GPU count — it's the AI network fabric that connects those GPUs so they move as one system.
This article looks at why distributed training doesn't speed up as much as expected when you add GPUs, how to check for a network bottleneck, and what to review in an enterprise GPU cluster.
GPU Scaling Is About Scaling Efficiency, Not Just Count
If doubling from 8 GPUs to 16 exactly doubles throughput, that's ideal scaling. In practice, though, the performance gain per additional GPU often shrinks as the count grows.
The metric for this is scaling efficiency.
Scaling efficiency measures how close actual throughput comes to the theoretical gain you'd expect from adding GPUs. If you double your GPUs but throughput only grows 1.5x, that means part of the added compute capacity is being consumed by communication, synchronization, and waiting.
To understand why, you need to look at how distributed training actually runs.
How a Distributed Training Step Flows
A typical step in distributed training looks like this:
Each GPU computes on its assigned data.
Gradients computed by each GPU are aggregated.
Results are synchronized across all GPUs.
Once every GPU finishes, training moves to the next step.
Features like PyTorch's DistributedDataParallel synchronize gradients computed in each process. The actual collective communication between GPUs — operations like All-Reduce — relies on a topology-aware communication library such as NVIDIA NCCL.
The problem: the slowest GPU to finish communicating sets the pace for the entire step. If one node's network is slow, or a particular path gets congested, every other GPU has to wait even after finishing its own computation before moving to the next step. As GPU count grows, so does the number of communication paths — so a small network inefficiency drags down the scaling efficiency of the whole cluster by more than you'd expect.
So what path does the data actually travel?
Communication Inside a Node vs. Between Nodes Works Differently
GPU-to-GPU communication paths fall into two broad categories: inside a server, and between servers.
Inside the node: NVLink and PCIe
GPUs within a single server exchange data over NVLink or PCIe.
NVLink is a high-bandwidth scale-up fabric designed to move large volumes of data quickly between GPUs. It tightly connects multiple GPUs within a single server or rack.
But GPUs in the same server aren't necessarily connected under identical conditions. Which PCIe switch and NUMA node the GPU, CPU, and NIC are attached to can change the data path and the effective bandwidth available.
Between nodes: InfiniBand and RoCE
GPUs spread across multiple servers need a scale-out fabric — InfiniBand or RoCE-based Ethernet — to communicate.
RoCE supports RDMA (Remote Direct Memory Access) over Ethernet, cutting CPU involvement and delivering low-latency data transfer between servers.
NCCL is aware of various connection structures — NVLink, PCIe, InfiniBand, network interfaces — and picks a communication path accordingly. But a communication library being topology-aware doesn't automatically fix a physical bandwidth shortfall, bad cabling, or network congestion. If intra-node communication is fast but inter-node communication is slow, the network's share of total training time grows as you add more servers.
This kind of bottleneck often doesn't show up in GPU utilization metrics — time a GPU spends waiting on another GPU's data can still get counted as "in use."
The Industry Now Treats the Network as Part of the Compute Architecture
The network is no longer treated as auxiliary equipment connecting GPU servers — it's now part of the compute architecture that determines a cluster's throughput and scalability.
Spectrum-X: An Ethernet Fabric Built for AI Workloads
Spectrum-X is NVIDIA's AI-focused Ethernet platform, combining switches, SuperNICs, and networking software. NVIDIA says it delivers up to 1.6x the AI networking performance of general-purpose Ethernet, and is designed to maintain about 95% efficiency even at 100,000+ GPU scale. It supports adaptive congestion control and load balancing to reduce performance variance when multiple workloads share the network.
These figures come from NVIDIA's own specific configurations and workloads, but they illustrate how AI infrastructure's performance competition has expanded beyond the GPU chip to NICs, switches, and congestion-control software.
MRC: One Connection, Multiple Network Paths
In May 2026, OpenAI — working with AMD, Broadcom, Intel, Microsoft, and NVIDIA — published MRC(Multipath Reliable Connection) through the Open Compute Project.
Conventional RDMA connections tie a single flow to one specific path. MRC spreads packets across multiple paths, so if one path hits congestion or fails, traffic can shift to another. MRC extends RoCE (RDMA over Converged Ethernet) and supports large-scale AI network fabrics through SRv6-based source routing. The fact that it was published as an open specification rather than a single vendor's proprietary technology says something about where this trend is heading.
MRC is already deployed in frontier model training environments — including OpenAI's large-scale NVIDIA GB200 supercomputers and Microsoft's Fairwater clusters — well past the prototype stage and into actual production.
These developments all point in the same direction: as GPU clusters grow, the network is no longer just a way to connect hardware — it's a core system that determines overall compute performance.
Enterprise GPU Clusters Aren't Immune to Network Bottlenecks Either
It would be a mistake to read this as a story only about hyperscalers running tens of thousands of GPUs. In fact, in enterprise environments running dozens to hundreds of GPUs on a constrained budget — often running general-purpose networking and AI workloads side by side — it can be even harder to tell whether a bottleneck is coming from the GPU, storage, or network.
For example, GPU node count may keep growing while switch uplink capacity stays flat. You might have adopted RoCE, but its congestion-control settings may not actually match your workload, or NCCL may be selecting a NIC you didn't intend. Training traffic and storage traffic frequently end up competing on the same path.
"We use 100GbE" or "we adopted InfiniBand" isn't enough on its own. You need to check which path your actual workload is communicating over, and how much of the theoretical bandwidth is being used effectively.
AI Network Fabric Checklist
✅ Are you measuring the actual throughput gain each time you add GPUs?
✅ Are you separating compute time from communication time within each training step?
✅ Are you regularly testing NCCL collective communication bandwidth and latency?
✅ Does throughput drop sharply when multiple teams run jobs at the same time?
✅ Does your GPU expansion plan include switch, NIC, and uplink capacity expansion?
If you can't confidently answer even one of these, there's a good chance you have enough GPUs — but the network is keeping you from actually using their performance.
Diagnose the Bottleneck with RA:X, Operate It with AIPub
Before deciding how many more GPUs to buy, you need to find out where your current infrastructure's bottleneck actually is.
TEN's RA:X benchmarks GPU, CPU, memory, network, and storage performance based on real AI workloads. Rather than just comparing spec sheets, it measures actual throughput and communication patterns based on your model architecture, data size, and parallelization strategy. That lets you determine, from measured data, whether the constraint is GPU compute, inter-node communication, or storage.
Once RA:X has measured the bottleneck and helped design the right configuration, ongoing operations still matter. AIPub monitors 40+ metrics in real time — from the data center down to the node, GPU card, and container level, including PCIe traffic and NVLink bandwidth — and separates resources by user, team, and project when multiple teams share a cluster.
Conclusion: Check Scaling Efficiency Before You Buy More GPUs
A GPU cluster's performance isn't determined by the spec of your fastest single GPU, or by GPU count alone. GPU-to-GPU communication, physical topology, network congestion, and workload placement all have to work together for the whole cluster to move like one system.
If you've added GPUs but training time hasn't dropped as much as expected, first check whether those added GPUs are actually translating into more throughput. If you can't tell how much time your GPUs spend computing versus waiting on communication, adding more hardware can just mean higher cost with lower scaling efficiency.
AI infrastructure competitiveness isn't about how many GPUs you own — it's about how efficiently you connect them and run them as a single compute system.
Let's measure your current cluster's scaling efficiency and network bottlenecks, and figure out what to fix before your next investment, together with TEN's experts.
Reference
NVIDIA, Spectrum-X Ethernet Networking Platform
Open Compute Project, Multipath Reliable Connection (MRC) Specification
Broadcom, "Enabling AI Networking @ Scale with Multi-path Reliable Connections (MRC)"
NVIDIA, Overview of NCCL
PyTorch, Distributed Data Parallel Design Note