The AI Rack Became a Data Center Design Problem

The rack crossed a boundary.

With A100 and H100, NVIDIA delivered an integrated eight-GPU server. Operators could configure how many servers to place in a rack, how densely to power them, and how to connect them through InfiniBand or Ethernet.

GB200 NVL72 changes that contract. Its 72 GPUs, 36 Grace CPUs, 18 compute trays, and nine NVLink switch trays form one rack-wide scale-up system. The rack is no longer packaging around the computer. It is part of the computer.

That change pulls the building into the architecture. Electrical distribution, direct liquid cooling, network rails, floor loading, firmware qualification, and commissioning now determine whether the accelerator system can operate as specified.

TL;DR

• A100 and H100 scale up inside an eight-GPU server; GB200 and GB300 NVL72 extend the scale-up domain across 72 GPUs in one rack.
• Published power figures describe different boundaries: GPU design power, server maximum, rack IT load, provisioned capacity, and facility load are not interchangeable.
• NVIDIA describes GB200 at about 120 kW of rack consumption and GB300 at up to 142 kW; the labels are related but not identical measurements.
• Liquid cooling does not automatically remove air cooling. Published hybrid designs still leave roughly 13% to 30% of rack heat in the air.
• Rubin’s proposed next step makes power allocation visible to workload software, but public production measurements remain behind the architecture claims.

The deployment unit moved from server to rack

The useful distinction is scale-up versus scale-out.

Scale-up joins processors into a tightly coupled domain with enough bandwidth and low enough latency for frequent model communication. Scale-out connects those systems into a larger cluster through a network.

In DGX A100 and DGX H100, eight GPUs share a local NVSwitch fabric. Communication beyond those eight GPUs crosses ConnectX adapters and the scale-out network.

In NVL72, NVLink becomes rack infrastructure. Eighteen compute trays connect through nine NVLink switch trays, making all 72 GPUs part of one rack-wide scale-up domain. InfiniBand or Spectrum-X still connects multiple racks, but more model communication can remain inside the rack.

NVIDIA’s published Vera Rubin Pod expands the physical system further: 40 racks, 1,152 Rubin GPUs, and five purpose-built rack-scale system types spanning compute, CPU capacity, inference, storage, and networking.

The rack does not eliminate the distributed system. It moves one of its most performance-sensitive boundaries.

Blackwell still exists in smaller server configurations. The claim is narrower: NVL72 introduced a top-end configuration whose scale-up and service boundary is the rack.

Five different numbers are called power

The fastest way to misunderstand AI infrastructure is to place every published watt figure on one line.

An A100 GPU’s design power, a DGX H100 server maximum, a GB300 rack envelope, and a facility’s utility demand measure different things. Provisioned electrical capacity adds redundancy and derating. Cooling equipment has its own capacity and power consumption. None of those numbers automatically describes energy per completed model request.

The U.S. Department of Energy’s data-center metering guide separates power from energy and stresses measurement at explicit boundaries. Product specifications help size infrastructure. Meters show what the deployed workload consumed.

The generation table therefore preserves the original label and system boundary:

The two charts use different boundaries and labels. They are not an efficiency comparison. GB200 and GB300 contain nine times as many GPUs as a DGX server, plus CPUs, NVSwitch trays, networking, and rack-level power equipment.

Four DGX H100 systems can create a rack above 40 kW, but NVIDIA’s H100 electrical design allows operators to reduce density when local power or cooling cannot support that configuration. NVL72 is less divisible: the rack-scale fabric is part of the product.

A 140 kW rack connects directly to the building

A conventional server rack receives a manageable set of branch circuits and rejects most heat into room air. A 120–142 kW NVL72 system changes both paths.

Schneider’s GB200 facility design traces power through medium-voltage equipment, transformers, UPS modules, distribution panels, overhead busway, rack feeds, power shelves, busbars, and point-of-load conversion. Dell’s IR7000 rack specification supports as many as eight power feeds and two internal busbars. Installed conversion capacity can exceed expected rack consumption because the design must preserve operation during a failed feed or power shelf.

That distinction matters. Eight 33 kW GB200 power shelves create 264 kW of installed conversion capacity, but NVIDIA documents them as two redundant power zones. The rack is not a 264 kW load.

The power path becomes a mechanical problem as well as an electrical one. Raising voltage reduces current for the same delivered power. One megawatt at 50 volts represents 20,000 amps; at 800 volts it represents 1,250 amps before losses and engineering margin.

That is why NVIDIA, Eaton, and the Open Compute Project are developing 800 VDC rack-distribution architectures. These are roadmap systems associated with future 600 kW to 1 MW rack classes, not proof that megawatt racks are broadly shipping.

Liquid cooling is a chain of systems

Direct liquid cooling does not mean that facility cooling disappears.

Cold plates collect heat from GPUs and CPUs into a technology cooling loop. A coolant distribution unit, or CDU, controls that clean loop and transfers its heat through a heat exchanger. A facility-water loop then moves the heat toward a dry cooler, cooling tower, or chiller.

The CDU is a hydraulic boundary, a control system, and a failure domain. A cooling-loop failure must therefore participate in the rack’s protection and shutdown logic.

Published designs also show why the word liquid-cooled needs a qualifier:

These are different product and facility assumptions, not one industry progression. Memory, networking, storage, and power-conversion equipment can leave a material residual air load after the accelerators move to water.

Warm-water designs change the heat-rejection path. NVIDIA, Lenovo, and Supermicro describe newer systems accepting inlet water near 45°C. Higher water temperatures can expand the hours when a facility rejects heat without compressor-based chilling. They do not remove the flow calculation: heat load, temperature rise, pipe size, pressure drop, and pump energy remain linked.

Redundancy crosses the electrical and thermal systems. Schneider’s published design specifies N+1 UPS modules, N+1 PDUs, and N+1 CDUs, but each protects a different boundary. Pumps, controls, and CDUs need backed-up power if the compute equipment must survive a utility or distribution event.

Rack density becomes a utility and siting constraint

The facility problem does not stop at the busway and CDU.

NVIDIA’s GB200 SuperPOD reference architecture groups eight NVL72 racks into a 576-GPU scalable unit. At the published rack design points, the compute racks alone move the electrical requirement into the megawatt class before adding scale-out networking, storage, pumps, residual air cooling, and conversion losses. A campus built from many scalable units therefore depends on upstream transformers, substations, utility interconnection, generator strategy, and the rate at which power capacity can be added.

Heat rejection has the same aggregation problem. The technology loop can remain closed and clean while transferring heat through CDUs, but the facility still needs somewhere to reject that heat. The Schneider reference design shows how dry coolers, chillers, or evaporative systems change the mechanical boundary. Climate, available land, noise limits, water availability, and local operating rules determine which path is practical.

“Liquid cooled” does not by itself describe water consumption. A closed secondary loop moves heat without continually consuming its full flow volume. Water use depends on the facility-side rejection design, blowdown and treatment, local weather, and whether the site uses dry or evaporative cooling. The useful planning questions are therefore separate:

The rack is the deployable computer, but the utility connection and heat-rejection plant set the maximum size and pace of the fleet.

Networking becomes part of rack qualification

A100 and H100 racks could be assembled from independently serviceable eight-GPU systems. Their NVLink fabric ended at the server. NVL72 spreads NVLink, NVSwitch, power, and coolant across the rack.

That creates at least three separate traffic classes:

1. GPU collectives over ConnectX and the compute fabric.
2. Models, training data, checkpoints, and storage traffic over infrastructure paths that can include BlueField.
3. Management and control over out-of-band networks connecting BMCs, switches, telemetry, power, and cooling controllers.

NVIDIA’s GB200 fabric documentation keeps those paths distinct. Its DGX SuperPOD design uses Quantum InfiniBand, while the GB300 enterprise design demonstrates Spectrum-X Ethernet. InfiniBand versus Ethernet is a deployment choice, not a clean generation boundary.

Topology must agree across GPU identity, NIC affinity, switch plane, scheduler placement, and NCCL’s collective topology. A network can pass link tests and still underperform when ranks, rails, and adapters do not align.

The larger scale-up domain creates a larger potential failure domain. Losing an A100 or H100 server removes eight tightly coupled GPUs. Losing or degrading an NVL72 rack can affect a 72-GPU domain. Redundant network planes protect against selected link and switch failures; they do not automatically cover correlated firmware, rack power, coolant, or NVLink control failures.

The rack must be commissioned as one system

The installation is not complete when the compute trays boot.

The Open Compute Project’s liquid-cooling commissioning guidance calls for flushing, filtration, material compatibility, hydraulic balancing, pressure and leak testing, controls validation, and extended observation. Electrical commissioning must test feeds, redundancy, protection, and load behavior. Network commissioning must validate rails, congestion behavior, storage paths, and management isolation.

The physical route into the data hall is part of the plan. Dell publishes a 1,900 kg maximum payload for its IR7000 rack before site-specific sidecars and external infrastructure. Structural review must cover the final static load, concentrated loads at casters or leveling feet, and the rolling path from the loading dock.

This is why OEMs, ODMs, electrical vendors, and cooling vendors now participate in the platform architecture. The operational conclusion is that a qualified rack-plus-facility design is safer than assembling components whose specifications appear individually compatible.

Rubin makes power visible to the scheduler

Blackwell makes the rack a scale-up computer. Rubin’s published architecture begins treating a fixed site-power envelope as a scheduling constraint.

NVIDIA describes 45°C liquid cooling, rack-level power telemetry, dynamic power allocation, and MaxLPS—software intended to schedule workloads within a shared power envelope. NVIDIA reports that the mechanism can place more GPUs under the same power budget, but the published documentation labels the example illustrative and the testing incomplete.

Those are vendor tests and evolving designs. NVIDIA’s MaxLPS documentation says testing remains in progress. Power steering can reduce short-duration peaks or allocate a fixed envelope more productively; it does not automatically reduce total energy.

The architectural proposal still matters. It connects workload placement to a configured rack or site power budget rather than assuming unconstrained electrical capacity.

What production evidence proves

Oracle’s account of GB200 deployment confirms the physical problem: racks above 120 kW at peak, high-density three-phase delivery, direct-to-chip liquid loops, CDUs, and rack-level monitoring. CoreWeave’s NVL72 documentation treats the full 18-node rack as the deployment unit.

Those deployments establish NVL72 as a production rack system. They do not disclose calibrated workload traces, full cooling energy, water use, availability, or equal-work comparisons against other platforms.

A useful independent comparison would hold constant:

• model and output-quality target;
• prompt, context, output length, concurrency, and latency targets;
• completed work under power caps and component failures;
• rack IT energy, facility energy, cooling energy, and water use;
• operator effort, repair time, and degraded operation.

MLPerf’s power policy is stronger than quoting TDP because it measures a defined system under test at the wall. It still does not answer the facility question unless the submitted boundary includes the rack, network, pumps, and heat-rejection systems being compared.

What’s missing

Public specifications now describe rack-wide compute systems in detail. Public operating evidence remains thinner.

The missing dataset includes sustained rack input under representative workloads, idle and capped behavior, sub-second power transients, pump and CDU consumption, residual room-cooling energy, water use, component failure rates, repair time, and workload goodput at equal facility energy.

Rubin widens that gap. Its pod composition, 45°C cooling, power steering, and future 800 VDC distribution create a coherent systems argument. Independent Rubin measurements and complete public facility envelopes were not available at this cutoff.

So what

The accelerator choice now reaches further into the building.

An NVL72 purchase commits more than GPU capacity. It shapes the scale-up domain, network rails, electrical feeds, coolant loops, rack weight, commissioning process, failure model, and the software that coordinates all of them.

The evidence suggests that NVIDIA gains a platform advantage by co-designing these layers and publishing one validated system boundary. A multi-vendor alternative must assign someone to qualify the boundaries between suppliers.

The unresolved question is operational rather than numerical: if workload software begins reallocating power across racks in real time, who owns the safety contract between the scheduler and the electrical plant—and what independent test proves that contract fails safely when a workload surge and a power-path failure happen together?

Part of the AI Compute Landscape and the NVIDIA chapter. Start with The GPU Stopped Being the Product, then continue with The AI Buildout Is Constrained by Its Slowest Layer.