NVMe over Fabrics (NVMe-oF): Extending NVMe Across the Network for AI Infrastructure

ai-storageai-infrastructure
Written by Dhupinder Malhotra

NVMe over Fabrics is the mechanism that lets a GPU server reach flash that lives in another box, over a network, while still speaking NVMe. This article covers how it came about, how it works, where AI infrastructure is using it today, and where I think the claims need a harder look.

A quick assumption: I take it that you already know what NVMe is, namely a host-controller interface built around many deep command queues, designed for flash rather than spinning disks. NVMe-oF is what happens when that interface is stretched across a network.

Historical Background

NVMe 1.0 was published in March 2011. Work on extending it over a network began in 2014, and the first NVMe-oF specification followed in June 2016 with an RDMA transport binding. Fibre Channel came next: the T11 committee published FC-NVMe in August 2017. NVMe/TCP was ratified in November 2018 and folded into NVMe-oF 1.1 in 2019, which opened the door to running NVMe-oF on ordinary Ethernet without RDMA-capable hardware.

With the NVMe 2.0 library in June 2021, the standalone NVMe-oF document was absorbed into the NVMe Base Specification, and the RDMA and TCP transports became specifications of their own. They are still actively maintained: NVM Express updated both transport specifications to revision 1.2 with NVMe 2.3 in August 2025, and to revision 1.3 with NVMe 2.4 on 4 August 2026. The 2026 release is mostly not about fabrics. Its primary additions are post-quantum cryptography support, voltage telemetry and migration features, and the only new technical proposal that I could see touching a fabric transport is the post-quantum enhancement (TP4201), which affects the TCP transport specification. So do not read it as an NVMe-oF overhaul.

Drivers for NVMe-oF in AI

Three pressures push AI clusters toward networked flash:

  • Datasets and checkpoints outgrow local capacity. Training sets are often too large for GPU memory, and periodic checkpointing stalls training if storage cannot absorb the burst.

  • Stranded local storage. SSDs inside a server can only be used by that server. Pooling flash behind a fabric lets capacity and bandwidth scale independently of GPU count.

  • Agentic and long-context inference. KV caches grow with context length and with the number of concurrent sessions, and they increasingly need to be shared between nodes rather than rebuilt each time. In the CXL instalment I discussed the DRAM-class end of this problem. NVMe-oF addresses the flash-class end.

The economics reinforce this. GPUs are typically the most expensive part of an AI cluster, so storage is increasingly judged less by raw IOPS and more by how much idle GPU time it avoids.

How NVMe-oF Works

NVMe-oF preserves NVMe's model end to end: queue pairs, commands, namespaces and the identity of each endpoint (an NVMe Qualified Name, or NQN). A host connects to a controller in a remote NVM subsystem, and the remote namespaces appear to the host as block devices. Only the transport binding underneath differs.

Figure 1: One NVMe model, three fabric transports

The Three Transports

  • NVMe/RDMA (RoCE v2, iWARP, InfiniBand): the original binding. RDMA offloads transport work to the NIC, giving the lowest latency and the lowest CPU cost. The price is network engineering: RoCE v2 deployments typically depend on careful Priority Flow Control (PFC) and ECN tuning.

  • NVMe/TCP: carries NVMe capsules over standard TCP/IP, so it needs no special NICs and is routable. Linux has supported it since kernel 5.0, and SPDK since v19.01. It costs more CPU and somewhat more latency than RDMA.

  • NVMe/FC (FC-NVMe): runs NVMe over Fibre Channel using existing FC HBAs and fabrics, and is the natural upgrade path for enterprises with FC SANs.

How big is the TCP-versus-RDMA gap? One Oracle test (40 Gb ConnectX-5 ports, Linux UEK6) measured TCP latency about 30% higher than RDMA for single-threaded, queue-depth-1 8K reads, yet TCP IOPS matched RDMA once the test used 32 threads. That is one vendor's lab, on hardware of that era, so treat it as an illustration of the shape of the trade-off and not as a rule.

GPUDirect Storage: Keeping the CPU Off the Data Path

For GPU servers, the transport is only half the story. NVIDIA's GPUDirect Storage (GDS) lets a DMA engine near the NIC or storage device move data straight into GPU memory, avoiding a bounce buffer in host DRAM. NVIDIA documents GDS as working with local NVMe and with remote storage over NVMe-oF, which is why NVMe-oF and RDMA fabrics show up together in so many AI storage designs.

Figure 2: GPUDirect Storage removes the host-memory bounce buffer

Discovery and Security

At small scale an administrator can configure each host to reach each subsystem by hand. At large scale that breaks down, and the NVMe community defined automated discovery (TP-8009 and TP-8010), including a Centralized Discovery Controller (CDC) that acts as a name server for the fabric. On the security side, NVMe defines DH-HMAC-CHAP for in-band authentication, and TLS 1.3 is part of the NVMe/TCP transport specification.

NVMe-oF in the AI Data Center

The most visible recent development is NVIDIA's Inference Context Memory Storage platform, now branded CMX. NVIDIA introduced it at CES in January 2026, built around the BlueField-4 data processor, and then positioned it inside the BlueField-4 STX reference architecture at GTC on 16 March 2026. CMX adds a pod-level tier that NVIDIA calls G3.5, between node-local SSDs and shared network storage, specifically for KV cache. NVIDIA states that it builds on standard NVMe and NVMe-oF transports, with Spectrum-X Ethernet providing the RDMA fabric and BlueField-4 managing the SSDs on the target side.

Figure 3: KV cache tiers, with the pod-level G3.5 tier that CMX introduces

NVIDIA reports up to 5x higher tokens per second and up to 5x better power efficiency than traditional storage. That is a vendor-reported figure against an unspecified baseline, and I would wait for independent measurements. NVIDIA has said partner platforms are expected in the second half of 2026, which means broad real-world data is still thin.

Outside the NVIDIA reference design, the established pattern remains the same: storage platforms from vendors such as DDN, VAST Data, WEKA, Pure Storage, NetApp, Dell and IBM serve GPU clusters over RDMA-capable fabrics, many with GDS support, and NVMe-oF block arrays feed training and checkpoint workloads directly.

Unique Selling Proposition

  • NVMe semantics preserved across the network: multiple queues, parallel I/O and native NVMe commands, as opposed to the SCSI command model that iSCSI carries.

  • Transport choice: RDMA where latency matters most, TCP where simplicity and reach matter, FC where an FC SAN already exists.

  • Disaggregation: flash can be pooled, shared and scaled independently of compute.

  • Mainstream software support: Linux has mature initiator and target stacks, and Microsoft introduced an inbox NVMe-oF initiator (NVMe/TCP and NVMe/RDMA) in Windows Server Insider builds in March 2026.

Ecosystem Players

  • Standards bodies: NVM Express (base and transport specifications), T11/INCITS (FC-NVMe) and SNIA (education, interoperability and discovery-automation work).

  • NVIDIA: GPUDirect Storage, ConnectX NICs, BlueField DPUs, Spectrum-X Ethernet, and the Dynamo and NIXL software that route KV cache between tiers.

  • Storage vendors: Dell, DDN, HPE, IBM, NetApp, Pure Storage, VAST Data and WEKA. Several are named partners for building BlueField-4-based AI storage platforms.

  • Adapter and fabric vendors: Broadcom and Marvell for FC-NVMe HBAs, plus the RDMA NIC and Ethernet switch makers.

  • Software-defined and NVMe/TCP specialists: Lightbits Labs, which describes itself as the inventor of NVMe/TCP, and Dell, whose SmartFabric Storage Software implements a CDC.

Caveats

NVMe-oF is useful, but several things deserve a sceptical look before anyone designs around it.

  • Transport choice is a real trade-off. RDMA needs a well-tuned lossless-style Ethernet fabric, and PFC misconfiguration can cause head-of-line blocking or even deadlock. TCP avoids that but pays in CPU and latency. Tail latency under congestion matters more than averages, and public benchmarks that report it are scarce.

  • It does not beat local flash. NVM Express itself notes that NVMe/TCP will not improve latency over directly attached NVMe. Remote flash is a capacity and sharing play, not a speed play. For hot KV cache, DRAM or CXL-attached memory remain faster.

  • Vendor-reported performance. The 5x CMX claim and most AI-storage throughput numbers come from the vendors selling the systems. Baselines and workloads are often not disclosed.

  • Lock-in risk sits above the standard. NVMe-oF is open, but CMX combines BlueField-4, DOCA, NIXL and Dynamo into an NVIDIA-specific stack. Also, much of the GDS support list is file systems and parallel file systems, not plain NVMe-oF block targets, so read vendor support claims carefully.

  • Operational and security gaps. Discovery at scale needs a CDC, and SNIA material itself notes that a compromised CDC leaves hosts without storage, so its availability design matters. In-band DH-HMAC-CHAP applies to TCP connections, and an academic analysis from 2022 points out that RDMA transports depend on separate mechanisms such as IPsec.

  • Maturity of the enterprise on-ramp. Windows Server's NVMe-oF initiator is a preview, and Microsoft's own notes list limitations such as CLI-only configuration and no multipathing in the early builds.

  • Cost. Flash is exactly what the AI boom is consuming. Industry voices have warned that NAND supply may stay tight into 2027, which can erode the cost case for flash-heavy tiers.

  • Market numbers disagree. One analyst puts the NVMe-oF market at US$2.85 billion in 2025 rising to US$19.9 billion by 2035, while another estimates a US$5.4 billion NVMe-oF switch market by 2030. Single-publisher forecasts with different scopes should not be compared or quoted as fact.

Outlook

I expect NVMe-oF to show up in AI infrastructure in layers. NVMe/TCP will keep spreading for general block storage, because it runs on infrastructure people already operate. RDMA transports will stay the choice for the latency-critical tiers: GPU data feeding, checkpointing, and the KV cache tier. NVIDIA's CMX and STX designs are the first large-scale test of whether a dedicated, shared flash context tier pays off in production. The next 12 months of partner shipments and independent benchmarks will tell us more than any of today's slides.

NVM Express now ships annual specification updates, which suggests the standard will keep absorbing AI-driven needs at a steady pace.

Final Words

NVMe-oF is the connective tissue between fast GPUs and large, shared flash. It is not new, but the AI era has given it a new job. I may well have missed implementations or deployments. If you have worked with NVMe-oF in an AI cluster, please comment with what you have seen, especially transport choice, tail latency and operational lessons. I will add them to the discussion.

References

Dhupinder Malhotra
Next

NVIDIA’s AI Infrastructure Portfolio: Hardware and Software Overview