InfiniBand vs. Ethernet
If compute is the engine of AI infrastructure, the network is the nervous system. Thousands of GPUs must talk to each other, in lockstep, thousands of times a second, for a training job to make progress at all. Get the network wrong, and expensive GPUs sit idle waiting for data instead of computing. This is why, in this AI infrastructure series, after covering memory (CXL, SOCAMM, HBF, HBM4) and interconnects like UALink and NVLink, it's time to turn to the two standards that carry data between servers and racks: InfiniBand and Ethernet.
A Bit of History
Ethernet is the older of the two by a wide margin. It traces back to 1973 at Xerox PARC, where Robert Metcalfe and colleagues devised a way for machines to share a coaxial cable, and it was standardized as IEEE 802.3 in 1983. It went on to become the default networking technology for practically everything, from campus LANs to hyperscale data centers, scaling from 10 Mbps to 800 Gbps and beyond over five decades. InfiniBand is much younger. It emerged in 1999 when two rival efforts to replace the aging PCI bus, Intel-led Next Generation I/O and the Compaq/IBM/HP-backed Future I/O, merged to form the InfiniBand Trade Association (IBTA). Version 1.0 of the InfiniBand Architecture specification followed in 2000. InfiniBand never did replace PCI in servers, but it found a lasting home in high-performance computing, propelled in large part by Mellanox, the Israeli chipmaker that shipped its first InfiniBand silicon in 2001 and was later acquired by Nvidia in 2020.
Drivers
The agentic AI era has turned networking from a back-office concern into a first-order design decision. A single large language model training run can span tens of thousands of GPUs that must synchronize gradients after every step. Any straggler link stalls the entire job. On the inference side, KV cache sharing, disaggregated prefill-decode architectures, and mixture-of-experts routing all depend on moving large volumes of data between nodes with minimal delay.
The industry has started describing this problem in three layers: scale-up (connecting GPUs within a rack, the domain of NVLink and UALink), scale-out (connecting racks within a data center, historically InfiniBand's stronghold and now the main battleground with Ethernet), and scale-across (connecting entire data centers into one logical cluster). InfiniBand versus Ethernet is fundamentally a scale-out and, increasingly, a scale-across question.
How the Two Standards Work
InfiniBand was built from the ground up as a lossless, switched-fabric network with Remote Direct Memory Access (RDMA) as a native capability. Data moves directly between the memory of one server and another without involving the CPU, and the fabric uses credit-based flow control so that packets are never dropped due to congestion. A centralized subnet manager configures routing across the fabric. This design gives InfiniBand deterministic, very low latency and consistent performance under heavy, synchronized traffic patterns typical of GPU collective operations.
Ethernet, by contrast, grew up as a best-effort, packet-switched network without native lossless guarantees or RDMA. That gap has been closing steadily. RDMA over Converged Ethernet (RoCE), together with Priority Flow Control and Explicit Congestion Notification, brought RDMA-like behavior to Ethernet fabrics. The more recent and more significant step is the Ultra Ethernet Consortium (UEC), formed in 2023 under the Linux Foundation by founding members AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft. In June 2025, the UEC published its 1.0 specification, a stack spanning NICs, switches, optics, and cabling, built around a new transport called Ultra Ethernet Transport (UET) designed for intelligent, multi-path congestion control and native RDMA support, while remaining interoperable across vendors.
Areas of Application
InfiniBand's traditional home is high-performance computing. It was the dominant interconnect on the TOP500 supercomputer list through much of the 2010s and remains the fabric of choice for tightly coupled AI training clusters, where each step of computation depends on rapid, synchronized data exchange between nodes, as well as for HPC scientific simulation. It has also historically been used as a storage-area networking fabric, competing with Fibre Channel for connecting compute to storage arrays.
Ethernet's domain is, by definition, nearly everything else: enterprise data centers, campus and WAN networking, and general-purpose cloud infrastructure. Within AI infrastructure specifically, high-performance Ethernet variants are now displacing InfiniBand in large multi-tenant AI clouds, scale-out training clusters at hyperscalers, and inference-serving infrastructure, where operational familiarity, multi-vendor sourcing, and integration with existing enterprise networks matter as much as raw latency. A number of the largest AI data center buildouts underway, including hyperscaler projects that stitch together multiple sites into a single compute fabric, are being built on Ethernet-based scale-across designs.
USP: What Each Brings to the Table
InfiniBand's unique selling proposition is deterministic performance at the extreme end of the spectrum: the lowest achievable latency, native lossless transport, and hardware-based in-network computing. Nvidia's Quantum-X800 platform, for instance, layers in SHARPv4 (Scalable Hierarchical Aggregation and Reduction Protocol) to perform data reduction operations inside the switches themselves rather than at the endpoints, and it pairs with ConnectX-8 SuperNICs to deliver 800 Gb/s per port, with switches such as the Q3400 offering 144 ports and 115.2 Tb/s of aggregate switching capacity. For workloads where every percentage point of GPU utilization matters and the fabric is purpose-built and single-vendor, InfiniBand tends to be the more reliable choice.
Ethernet's unique selling proposition is universality and openness. It runs on a vast, multi-vendor ecosystem of switch silicon, optics, NICs, and management software, avoiding lock-in to any single supplier. The UEC 1.0 stack is explicitly designed to be hardware-agnostic and royalty-free, aiming to deliver InfiniBand-like performance while preserving the operational familiarity, tooling, and gradual scalability that Ethernet has offered data centers for decades. Nvidia's own Spectrum-X800 platform, built around the Spectrum SN5600 switch and BlueField-3 SuperNICs, is explicitly positioned for multi-tenant generative AI clouds and large enterprises rather than dedicated single-purpose training superclusters.
Ecosystem Players
Nvidia sits at the center of both camps, following its 2020 acquisition of Mellanox. It sells Quantum-series InfiniBand switches alongside Spectrum-X Ethernet switches, and by early 2026 had become the largest single vendor in data center Ethernet switching by revenue as well, growing its Ethernet switch market share from under 4% two years earlier to roughly 21.5% in Q1 2026. Broadcom remains the dominant merchant-silicon supplier for Ethernet, with its Tomahawk series (Tomahawk 6 reaching 102.4 Tbps per chip) and Jericho3-AI chips underpinning many white-box and OEM switches, and is a key contributor to UEC-ready silicon. Arista Networks continues to expand its 800G Ethernet portfolio for AI and cloud workloads and is a vocal advocate for Ethernet's long-term position in AI back-end networking. Other UEC members and contributors include Intel, AMD, Meta, HPE, Cisco, and Microsoft. On the InfiniBand side, the ecosystem is comparatively narrow, with Nvidia as the dominant supplier of switches, adapters, and DPUs since the Mellanox acquisition.
Outlook
The direction of travel is reasonably clear even if the exact numbers vary by source: Ethernet, carried by RoCE today and increasingly by UEC-compliant stacks, is expanding its share of AI back-end networking, particularly in hyperscale, multi-tenant, and inference-heavy environments where openness and operational familiarity carry real weight. InfiniBand is not disappearing; it remains the fabric of choice for organizations building dedicated, tightly coupled training superclusters where deterministic low latency is worth the cost and vendor concentration. The more interesting story going forward may be at the scale-up layer, where NVLink Fusion and UALink are shaping how GPUs talk to each other within a rack, and at the emerging scale-across layer, where both InfiniBand and Ethernet camps are racing to stitch multiple data centers into single logical AI factories.
I might have missed some interesting InfiniBand or Ethernet developments. Please comment and let me know what I should add to the growing list in this article.
