RoCE v2: RDMA over Converged Ethernet for AI-Scale Networking

networksethernet

Oct 16

A GPU cluster with terabytes of pooled memory and blistering local bandwidth is only as good as the fabric that lets thousands of those GPUs exchange gradients, activations, and KV cache fragments with each other in microseconds. That fabric, for a large and growing share of AI training and inference clusters, runs on RDMA over Converged Ethernet version 2, or RoCE v2.

RoCE v2 doesn't get the same attention as HBM or CXL. It's plumbing. But it is also the technology quietly deciding whether an AI training job finishes in an efficient allreduce or stalls waiting on a straggling GPU. In this article, I'll walk through where RoCE v2 came from, how it actually moves data, why it has become the default choice for hyperscale AI networking, and where its rough edges still show.

A Brief History of RDMA and RoCE

Remote Direct Memory Access, or RDMA, is not new. It grew out of the InfiniBand world, where it has been used since the early 2000s to let one machine read or write directly into another machine's memory, bypassing the CPU, the operating system kernel, and the usual TCP/IP socket machinery. This bypass is what gives RDMA its defining characteristics: very low latency, very high throughput, and negligible CPU overhead compared to conventional networking.

For a long time, RDMA and InfiniBand were essentially synonymous, confined mostly to high-performance computing clusters and supercomputers. RoCE, standardized by the InfiniBand Trade Association (IBTA), was the effort to bring RDMA's benefits onto Ethernet, the networking technology that already dominated the rest of the data center. RoCE v1, released in 2010, was the first attempt, keeping InfiniBand's transport and replacing its link layer with Ethernet's. It worked, but it operated only at Layer 2, meaning it was confined to a single Ethernet broadcast domain and couldn't be routed across subnets.

RoCE v2, released in 2014, fixed that limitation by encapsulating the RDMA payload inside UDP and IP headers instead. That one change moved RoCE from Layer 2 to Layer 3, making it routable across a typical data center's Layer 3 Clos network using standard IP infrastructure, ECMP-based load balancing, and existing routers and switches. This is the version that matters today; when people say RoCE in the context of AI networking, they almost always mean RoCE v2.

Why AI Networking Needs RDMA

Training a large language model is not a single computer's job. It is a synchronized, iterative exercise across thousands of GPUs that must repeatedly exchange gradients (allreduce), shuffle tokens between experts in mixture-of-experts models (all-to-all), and, increasingly, move KV cache fragments during long-context inference. These are not casual, bursty traffic patterns; they are large, near-simultaneous flows where every GPU in a group needs to talk to every other GPU with minimal delay.

Conventional TCP/IP networking was never built for this. TCP's kernel-mediated stack, its per-connection retransmission and congestion behavior tuned for the general internet, and the CPU cycles it consumes per packet all add latency and jitter that a synchronized training step cannot tolerate; a single slow GPU or a single delayed collective operation stalls the entire job. RDMA sidesteps most of this by letting the network interface card place data directly into a receiving GPU's or CPU's memory, with the CPU largely out of the data path.

The catch is that RDMA, unlike TCP, was not originally designed to tolerate packet loss gracefully. InfiniBand achieves this through a purpose-built lossless fabric. RoCE v2 has to recreate that same lossless behavior on top of Ethernet, which was historically a best-effort, lossy medium. That requirement shapes almost everything distinctive about how a RoCE v2 network is built and tuned.

Inside RoCE v2: How It Works

At its core, RoCE v2 wraps an InfiniBand-style RDMA transport packet inside a UDP datagram, which is itself wrapped inside a standard IP packet and an Ethernet frame. The UDP destination port is a well-known value (4791) that switches and NICs use to recognize RoCE traffic. The UDP source port, notably, is not used as a real connection identifier; it serves as an opaque flow identifier so that ECMP hashing can spread flows between the same pair of endpoints across multiple network paths.

Queue Pairs and Verbs

Applications don't talk to RoCE v2 through sockets. They use the OFED/InfiniBand verbs API, which exposes queue pairs: a send queue and a receive queue associated with a given connection, plus a completion queue that the application polls (or is interrupted on) to know when work has finished. A NIC, increasingly a "SuperNIC" purpose-built for this role, handles segmentation, reassembly, and delivery of RDMA reads and writes directly against registered memory regions, without the OS or CPU touching the payload.

Keeping It Lossless: PFC, ECN, and DCQCN

Because RDMA transport degrades badly under packet loss, a RoCE v2 fabric has to behave like a lossless network even though it's built from Ethernet switches. Two mechanisms do most of the work. Priority Flow Control (PFC) lets a congested switch port send a pause signal upstream for a specific traffic class, stopping the sender before its buffer overflows and a packet gets dropped. Explicit Congestion Notification (ECN) is the complementary, less blunt instrument: switches mark packets as they start to congest, rather than dropping or pausing them outright, and the marked packets carry that signal back to the sender.

The mechanism that ties these together end to end is Data Center Quantized Congestion Notification (DCQCN), which uses ECN markings arriving at the receiver to trigger a Congestion Notification Packet back to the sender, prompting it to throttle its transmission rate before PFC ever needs to intervene. Getting the ECN thresholds, PFC watermarks, and buffer allocations right for a given cluster's traffic pattern is, by most accounts from operators running RoCE at scale, one of the harder parts of standing up an AI-scale Ethernet fabric. It is workload-sensitive and does not always transfer cleanly from one generation of switch silicon to the next.

RoCE v2's Case for AI Infrastructure

The appeal of RoCE v2 is fundamentally an ecosystem and economics argument as much as a technical one. It runs on Ethernet, which means it can share switches, cabling practices, optics, network operating systems, and, importantly, the networking talent that data center teams already have, rather than requiring a parallel InfiniBand-trained team and a separate supply chain. Ethernet's supplier base is also far broader than InfiniBand's, which for a long time effectively meant NVIDIA/Mellanox.

On raw performance, RoCE v2 does not quite match InfiniBand. Reported end-to-end latencies for well-tuned RoCE v2 deployments are commonly cited at roughly one to five microseconds (some write-ups say two to five), versus roughly half a microsecond to two microseconds for InfiniBand. One 2026 industry write-up, citing unspecified industry benchmarks, puts a properly configured RoCE v2 fabric at around 85 to 95 percent of InfiniBand's training throughput. That gap has been narrowing as switch silicon, SuperNICs, and congestion-control tuning improve, and for many operators the cost, openness, and operational familiarity of Ethernet outweigh a performance difference of that size.

Recent Developments and Ecosystem Players

RoCE v2 has moved well past the pilot stage in the last couple of years. A few developments stand out.

Meta's RoCE Fabric at Hyperscale

Meta has been unusually open about its production experience, publicly describing how it selected RoCE v2 as the transport for the majority of its AI capacity and scaled it from early prototypes to numerous clusters, each accommodating thousands of GPUs. Meta's engineering team designed a two-stage Clos topology it calls an AI Zone, and added an aggregator training switch layer to connect multiple AI Zones as model sizes outgrew a single zone's non-blocking capacity, with cross-zone traffic deliberately oversubscribed and balanced through ECMP and job-scheduling changes that minimize how much cross-zone traffic a given training job generates.

NVIDIA Spectrum-X

NVIDIA's Spectrum-X platform pairs its Spectrum switch silicon with BlueField SuperNICs to bring several InfiniBand-derived techniques, namely lossless operation, adaptive routing, and in-network telemetry, onto an Ethernet fabric. NVIDIA states that Spectrum-X lifts effective bandwidth from a typical 60 percent on standard Ethernet to as high as 95 percent, and improves generative-AI network performance by 1.6x. These are NVIDIA's own figures.

Multipath Reliable Connection (MRC)

In May 2026, OpenAI announced Multipath Reliable Connection (MRC), a new RDMA transport developed with AMD, Broadcom, Intel, Microsoft, and NVIDIA, with the specification published through the Open Compute Project. Where standard RoCE v2 binds a queue pair to a single network path, MRC decouples the RDMA connection from path assignment, spreading the packets of a single connection across many paths at once. It also shifts load balancing and failure handling away from switch control planes and into the endpoints, meaning the NIC and host software stack. According to AMD, MRC uses the NSCC congestion control algorithm that is also defined in the Ultra Ethernet Consortium's congestion control specification.

OpenAI reports that MRC is already deployed across its largest NVIDIA GB200 supercomputers used for frontier model training, including a site with Oracle Cloud Infrastructure in Abilene, Texas, and Microsoft's Fairwater systems. NVIDIA describes MRC as first deployed in production on Spectrum-X, while AMD says its Pensando Pollara 400 NIC supports MRC and that it has run MRC in test clusters with an unnamed cloud provider. Credit for MRC differs by announcer: OpenAI and AMD frame it as a joint specification (AMD says it co-led authorship), while NVIDIA-focused coverage describes it as developed for and in production on Spectrum-X. As of the May 2026 analyst coverage I reviewed, neither AMD nor Broadcom had announced a timeline for MRC-compatible NIC or switch products, and operators with existing RoCE v2 hardware from other vendors cannot adopt MRC without a hardware migration.

Broadcom's Tomahawk 6 and Jericho4

Broadcom has built out a merchant-silicon answer spanning scale-up and scale-out needs. Tomahawk 6 delivers 102.4 Tbps of switching capacity aimed at both scale-up and scale-out AI networks, while the companion Jericho4 fabric router is aimed at extending congestion-free RoCE across data centers, supporting over a million XPUs and lossless RoCE transport beyond 100 kilometers via deep buffering and line-rate MACsec encryption. Broadcom positions both as compliant with Ultra Ethernet Consortium specifications, aiming to keep RoCE v2-based fabrics interoperable across vendors rather than locked into a single company's stack. These are Broadcom's stated specifications.

Cloud Providers: Google Cloud

Google Cloud uses RoCE v2 as the RDMA transport underlying its A3 Ultra and A4 GPU-optimized Compute Engine instances, having previously offered RDMA-like capabilities through its GPUDirect-TCPX and TCPXO networking stack. Within a node, GPUs communicate over NVLink; between nodes, over RoCE.

The Ultra Ethernet Consortium

The Ultra Ethernet Consortium (UEC), launched in 2023 by AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft under the Linux Foundation umbrella, is the industry's multi-vendor effort to standardize an Ethernet transport purpose-built for AI and HPC, rather than leaving each vendor to bolt proprietary extensions onto RoCE independently. UEC published its 1.0 specification in June 2025, followed by revisions 1.0.1 in September 2025, 1.0.2 in January 2026, and 1.0.3 in July 2026 (the latest I could confirm). The specification defines Ultra Ethernet Transport (UET) with three profiles (AI Base, AI Full, and HPC) and folds in packet-level multipathing, selective retransmission, sender- and receiver-side congestion control, and optional packet trimming, among other mechanisms, some of which read as direct answers to RoCE v2's known limitations.

For 2026, UEC's own stated priorities include Programmable Congestion Management, which would let operators implement new congestion-control algorithms via a standard language rather than being locked to whatever a given NIC vendor shipped, plus a companion congestion-signaling standard. UEC compliance is now something vendors like Broadcom explicitly market for their RoCE-capable silicon, and the shared congestion-control lineage between UEC and MRC suggests the efforts are not entirely separate. How MRC and UET will relate over time is not yet settled.

Market Outlook

Sizing this market precisely is difficult, and figures vary widely depending on what's being measured. One market research estimate puts the RoCE-for-AI-workloads market specifically at roughly US$2.6 billion in 2025, growing to about US$3.2 billion in 2026, a reported annual growth rate around 21 percent. A separate, broader estimate of the overall RDMA networking market (InfiniBand and RoCE combined) from the 650 Group puts 2023 demand above US$6 billion, projected to exceed US$22 billion by 2028. These numbers aren't directly comparable (different scopes, different methodologies), but they point in the same direction.

On the Ethernet-versus-InfiniBand picture, Dell'Oro Group reports that Ethernet accounted for about two-thirds of data center switch sales in AI back-end networks in the first quarter of 2026, and for more than two-thirds across full-year 2025, up from a position where InfiniBand held roughly 80 percent in 2023. Two cautions apply. These are revenue shares for Ethernet switches in general, not for RoCE v2 specifically, although RoCE v2 is the dominant RDMA transport on those fabrics. And InfiniBand is not fading quietly: Dell'Oro also reports InfiniBand switch sales more than tripled in that same quarter on the ramp of NVIDIA's Blackwell Ultra platform. SDxCentral's coverage also relays an earlier Dell'Oro projection of around US$80 billion in Ethernet data center switch sales over five years.

Whether Ethernet fully displaces InfiniBand at the very top end of frontier training clusters remains an open question. The more confident claim is that RoCE v2 has become a default choice for a large part of the market, including cloud AI instances, enterprise inference clusters, and much hyperscale training.

Caveats and Open Questions

In keeping with this series' practice of being upfront about the rough edges, a few deserve mention.

First, RoCE v2's lossless behavior is achieved, not native. PFC is a blunt instrument: it operates per priority class across an entire link, and a congested flow can trigger a pause that backs up unrelated traffic sharing the same priority, a known failure mode sometimes called PFC storming or head-of-line blocking, which some published guidance flags as a reason to keep PFC to a single priority class and allocate buffers carefully. Second, DCQCN and PFC/ECN tuning is workload-specific and not trivially portable; several sources describe it as one of the genuinely hard parts of running RoCE v2 at scale, and parameters that work for one traffic pattern (large collective allreduce operations, say) may misbehave under another (bursty incast during checkpoint restores, for instance).

Third, the performance figures in circulation are largely vendor or vendor-adjacent. NVIDIA's 60-to-95-percent utilization and 1.6x claims are NVIDIA's own. The 85-to-95-percent-of-InfiniBand throughput figure traces to a single 2026 write-up that cites unspecified benchmarks. The latency ranges come from vendor-adjacent blogs and disagree slightly with each other (one to five versus two to five microseconds). None of these are independently audited numbers, and they should be read as directional rather than guaranteed.

Fourth, the ecosystem is converging unevenly. MRC is a multi-vendor specification published through OCP with production deployments at OpenAI and its partners, while UEC's UET is a separate, vendor-neutral standards effort that shares some congestion-control lineage with it. Which silicon will ship MRC support beyond the current Spectrum-X deployments, and how MRC and UET coexist, are not settled. Fifth, the sources disagree on who deserves credit for MRC, as noted above, and I have not tried to resolve that.

Finally, most of the adoption specifics in this article come from vendor and hyperscaler announcements and press coverage rather than independently verified operational data. That includes Meta's architecture, Google Cloud's instances, and MRC's production use, which is OpenAI's own account. It is worth keeping in mind when reading claims of near-InfiniBand performance.

Final Words

RoCE v2 is not the newest or the flashiest technology in the AI infrastructure stack, but it is arguably one of the most consequential: it is the transport quietly moving the gradients, activations, and KV cache fragments that keep very large training runs synchronized. Between hyperscaler production deployments, NVIDIA's Spectrum-X, the multi-vendor MRC specification, Broadcom's merchant silicon roadmap, and the Ultra Ethernet Consortium's standardization push, RoCE v2's role looks set to keep growing even as new transports take shape around it.

I might have missed some recent RoCE v2 developments or deployments. Please comment to mention any that I've missed; I'll add on to the growing list in this article.

References

#rocev2#network

Dhupinder Malhotra