Every article in this series so far has examined a single component: a memory form factor, an interconnect standard, a network protocol, a storage framework. Each makes sense on its own, but each also raises the same question: where does it sit, and what is it actually holding up? This article steps back and draws the whole map. If the earlier pieces were individual parts, this one is the assembly drawing.
Defining AI Infrastructure
AI infrastructure is the integrated stack of compute, memory, networking, storage, power and cooling, and the software that binds them, built to train, fine-tune and serve AI models. Definitions in the industry differ at the edges. Some vendors stop at the software below the application layer; others, notably NVIDIA, describe an "AI factory" in which compute, networking, storage, power delivery and cooling are co-designed as one system. I prefer the broader view, for a reason that becomes clear below: at today's scale, the facility is part of the machine.
What separates it from traditional enterprise IT is coupling. A conventional data center runs many loosely related workloads on CPU servers, with racks drawing roughly 5 to 10 kW. An AI cluster behaves like one very large computer. Thousands of accelerators exchange data continuously during training, and a single slow link, a starved GPU or an overheating rack can stall the entire job. Rack power for AI deployments is commonly quoted at 40 to over 100 kW today, so the electrical and thermal design changes along with the silicon.
What Is Driving the Build-Out
Four forces explain why this stack is being reshaped so quickly.
Model and context scale. Large models and long contexts need memory far beyond what one accelerator carries. Inference with very long sequences stores key-value (KV) caches that can reach terabytes. The earlier articles on CXL, SOCAMM and High Bandwidth Flash were all attempts to answer this single pressure.
Utilisation economics. The accelerator is the most expensive element in the cluster. Every cycle it spends waiting for data is money spent on nothing. Seen this way, most of the stack exists to keep the compute busy, and the figure of merit is increasingly tokens delivered per second, per watt and per dollar.
The shift to inference and agents. Agentic workloads keep long contexts alive across many steps, which stresses memory and storage more than raw compute. One industry tracker, citing Bloomberg Intelligence, puts inference at roughly two-thirds of AI compute; I treat that as indicative, since it traces to a single secondary source.
Capital at national-infrastructure scale. Multiple trackers put 2026 capital spending plans of the four largest US hyperscalers above US$700 billion. TrendForce's broader view, covering the top nine cloud providers including Chinese ones, is about US$830 billion. Company guidance moved up through the year; for example, Amazon said in July it expects about US$220 billion and Alphabet raised its range to US$195 to 205 billion. Earlier-in-the-year figures of US$600 to 625 billion are now stale.
Figure 1. The AI infrastructure stack: hardware, software, facility and capital layers.
The Stack, Layer by Layer
1. Compute
Compute is the layer everyone sees first: GPUs, custom accelerators (ASICs), the host CPUs that feed them and the DPUs that offload networking and storage tasks. NVIDIA remains the reference point. Its Vera Rubin generation, due in the second half of 2026, pairs Rubin GPUs, each with 288 GB of HBM4, with Vera Arm CPUs. NVIDIA's stated figures for the rack-scale NVL144 system are 3.6 exaflops of NVFP4 inference and 1.2 exaflops of FP8 training, about 3.3 times the GB300 NVL72. These are vendor-reported and precision-specific; more on that in the caveats.
The second story is custom silicon built by the hyperscalers themselves. Google's seventh-generation TPU, Ironwood, delivers 4,614 FP8 teraflops and 192 GB of HBM3E per chip and scales to pods of 9,216 chips. AWS Trainium3 offers 2.52 FP8 petaflops and 144 GB of HBM3e per chip, with up to 144 chips in an UltraServer. Microsoft (Maia) and Meta (MTIA) have their own programmes. The logic is economic: for stable, high-volume inference, a chip tuned to a known workload can beat a general-purpose one on cost per token. Software maturity is what keeps GPUs dominant for training, and the CUDA ecosystem is the main reason.
2. Memory
Memory sets how much model and context an accelerator can hold, and how fast it can read them. High Bandwidth Memory (HBM) sits on the package next to the GPU and gives the bandwidth; it is also scarce and expensive. TrendForce estimates HBM bit demand grew more than 130% in 2025 and will grow above 70% in 2026, and SK hynix reported its 2026 HBM supply as essentially sold out. The shortage has spilled over: capacity diverted to HBM tightened conventional DRAM, and contract prices were reported to have risen sharply in early 2026. Analysts quoted in trade coverage expect easing from late 2027 and normalisation around 2028.
Because HBM cannot grow fast enough, the industry is building a memory hierarchy around it. CXL adds pooled and shared capacity over PCIe. SOCAMM brings low-power, serviceable LPDDR5X modules to AI servers. High Bandwidth Flash proposes NAND stacked like HBM as a high-capacity, read-heavy tier. Each trades some latency for capacity, and each was covered in its own article earlier in this series.
3. Networking
Networking is best understood as two separate problems. Scale-up connects accelerators inside a rack or pod so they behave as one large GPU. NVIDIA's NVLink dominates here; the open alternative is UALink, whose 2.0 specification was released in April 2026, although the consortium itself says silicon for the 1.0 specification reaches labs in the second half of 2026, with products in 2027. Scale-out connects racks and pods. InfiniBand has been the incumbent; Ethernet is closing the gap through RoCE v2 and the Ultra Ethernet Consortium's 1.0 specification, published in June 2025, which redesigns the transport for multipath delivery and modern congestion control. Co-packaged optics is attacking the power and reach limits of the links themselves. I have covered InfiniBand versus Ethernet, RoCE v2, UALink versus NVLink and CPO in earlier articles.
Figure 2. Scale-up inside the rack versus scale-out across racks, and the storage tier behind both.
4. Storage and Data
Storage has two jobs. The traditional one is feeding training with datasets and writing checkpoints, where throughput and consistency matter more than anything else. The newer one is holding inference context. As the KV cache outgrows HBM it spills into host memory, then into local storage, and latency starts to hurt users. At CES 2026 NVIDIA introduced an Inference Context Memory Storage platform built on its BlueField-4 data processor, effectively declaring KV cache a first-class tier. Storage vendors such as VAST, WEKA and Dell are positioning NVMe-based products for it. NVMe over Fabrics, covered earlier in this series, is the transport that makes shared flash look local to the GPU.
5. Power and Cooling
This is the layer that most surprises newcomers. The IEA estimates that data centers used about 415 TWh in 2024, roughly 1.5% of global electricity, and projects around 945 TWh by 2030, with demand from AI-optimised facilities more than quadrupling. Racks are following the same curve. Schneider Electric, a vendor with an interest in the topic, cites average rack density rising from about 16 kW in 2025 to 27 kW in 2026 and the newest AI systems reaching up to roughly 246 kW per rack. At those densities air cooling is no longer practical, so direct-to-chip liquid cooling is becoming the default; Dell'Oro projects that market at about US$7 billion by 2029.
The harder constraint is the grid. The IEA warns that around 20% of planned data center projects could face delays because of grid strain, and industry surveys describe campuses scaling toward a gigawatt. A cluster with chips on order but no energised connection is idle capital. For many projects in 2026, time-to-power, not chip supply, sets the schedule.
6. Software: From Kernels to Serving
Software turns hardware into something a team can use. At the bottom are runtimes and libraries: CUDA for NVIDIA, ROCm for AMD (covered earlier in this series), and the compilers and collective-communication libraries that move tensors between chips. Above them sit frameworks such as PyTorch and JAX. Above those is the serving and orchestration layer, which has matured quickly. Inference engines (vLLM, SGLang, TensorRT-LLM) run the model; a serving layer handles routing and batching; an orchestration layer schedules and scales. NVIDIA Dynamo, which reached 1.0 in March 2026, sits on top of those engines to coordinate GPU pools, including KV-aware routing and offload to cheaper storage. Kubernetes and Slurm remain the usual schedulers.
7. Where It Runs
The same stack is consumed in several ways: hyperscale clouds, specialised GPU clouds (the "neoclouds"), colocation, and on-premises or sovereign deployments. The choice changes who carries the facility and supply-chain risk, but not the layers themselves.
Summary: The Binding Constraint at Each Layer
How the Layers Interact
The most useful habit for reading AI infrastructure news is to ask which constraint a given announcement relieves. Bottlenecks move. Faster accelerators expose memory bandwidth; more HBM exposes the network; a faster network exposes storage and the KV cache; and a denser rack exposes power and cooling. Roughly, the sequence of the last three years has been compute, then HBM and packaging, then networking, and now power. A weakness at any layer limits everything above and below it, which is why vendors increasingly sell reference architectures and whole racks rather than parts.
Ecosystem Snapshot
Compute: NVIDIA, AMD, Google (TPU), AWS (Trainium), Microsoft (Maia), Meta (MTIA), and Broadcom and MediaTek as custom-silicon design partners.
Memory: SK hynix, Samsung and Micron for HBM and DRAM; Panmnesia, Marvell, Montage, Liqid and Penguin Solutions around CXL; SanDisk and SK hynix on HBF.
Networking: NVIDIA (NVLink, InfiniBand, Spectrum Ethernet); the UALink Consortium; Ultra Ethernet Consortium members including AMD, Arista, Broadcom, Cisco, HPE, Intel, Meta, Microsoft and Oracle.
Storage: VAST Data, WEKA, Dell and the wider NVMe-oF ecosystem.
Facility: utilities and power developers, plus equipment and cooling suppliers such as Schneider Electric.
Software: PyTorch, JAX, CUDA, ROCm, vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, Kubernetes and Slurm.
Caveats Worth Taking Seriously
Vendor numbers are not comparable by default. Peak figures depend on numeric precision (FP4, FP8, FP64) and on whether sparsity is assumed. The Register reported that Trainium3 UltraServer compute can be quoted anywhere between about 363 and 1,452 petaflops depending on sparsity, and criticised Google's "24 times El Capitan" Ironwood comparison for setting FP8 against FP64. NVIDIA's 3.3x Rubin claim, Dynamo's "up to 7x" on Blackwell and storage vendors' time-to-first-token improvements are all vendor-reported, with methodologies that differ from one claim to the next. Treat them as upper bounds measured on the vendor's chosen workload.
Sources disagree, and I have not forced them to agree. Hyperscaler capex totals range from about US$600 billion to US$830 billion depending on date and scope. Vera Rubin appears as NVL144 in most coverage but NVIDIA itself kept the NVL72 name at CES 2026, so the branding is inconsistent across sources. Ironwood HBM bandwidth is quoted as 7.2, 7.37 and 7.4 TB/s across Google material and press coverage. Rack-power projections cluster around 240 to 246 kW. Where figures conflict I have used the more conservative or primary one and listed both in the reference notes.
Several statistics rest on a single secondary source. The two-thirds inference share, NVIDIA's share of accelerators and the memory price swings were all drawn from trade or analyst-summary sites rather than the original reports. Read them as directional.
Lock-in is real at every layer. CUDA and NVLink tie software and system design to one vendor. NVIDIA's context-memory platform is positioned with the Rubin generation, so adopting it is a bet on that roadmap. Custom silicon ties customers to a specific cloud. UALink 2.0 exists as a specification, but products are not expected until 2027, so for now it is a promise rather than an alternative.
The money may not be as safe as the build-out suggests. Trade press has reported that the Bank for International Settlements named circular financing among hyperscalers, neoclouds and lessors as one of its biggest financial-stability concerns, and investors have begun reacting to higher capex guidance. Infrastructure built on forecast demand is exposed if that demand arrives slower or cheaper than planned.
Roadmaps slip. Rubin Ultra, UALink silicon, HBF samples and every capacity forecast in this article are plans, not shipments. I have tried to mark which are which.
Outlook
Over the next 12 to 18 months several threads converge. Vera Rubin ramps through late 2026 into 2027. UALink silicon reaches labs and then products, which will be the first real test of an open scale-up fabric. HBF samples are expected in the second half of 2026, with server-scale solutions the year after. Memory supply is not expected to normalise before late 2027 at the earliest. And power will keep deciding where and how fast capacity is built, which is why rack-level power architecture and on-site generation are now part of AI infrastructure planning.
The larger change is one of mindset. A few years ago, AI infrastructure meant buying accelerators. Today it means designing a system in which memory, network, storage, power and software are balanced against each other, and in which the cheapest fix is rarely at the layer where the symptom appears.
Final Words
This map will date quickly, and I have surely left out layers and players that deserve a place on it. If you work in one of them, please add what I have missed or corrected in the comments, and tell me which layer you would like examined next.
