---
title: "The AI Rack Became a Data Center Design Problem"
description: "Blackwell moved NVIDIA's top-end scale-up domain from an eight-GPU server into a liquid-cooled rack, making power, cooling, networking, and commissioning part of the computer."
canonical_url: "https://artificialcuriositylabs.ai/posts/2026-08-30-ai-rack-data-center-design-problem/"
md_url: "https://artificialcuriositylabs.ai/posts/2026-08-30-ai-rack-data-center-design-problem.md"
published_at: "2026-08-30T07:00:00.000Z"
tags:
- "ai-infrastructure"
- "nvidia"
- "data-centers"
- "liquid-cooling"
- "networking"
---
The rack crossed a boundary.
With A100 and H100, NVIDIA delivered an integrated eight-GPU server. Operators could configure how many servers to place in a rack, how densely to power them, and how to connect them through InfiniBand or Ethernet.
[GB200 NVL72](https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/system.html) changes that contract. Its 72 GPUs, 36 Grace CPUs, 18 compute trays, and nine NVLink switch trays form one rack-wide scale-up system. The rack is no longer packaging around the computer. It is part of the computer.
That change pulls the building into the architecture. Electrical distribution, direct liquid cooling, network rails, floor loading, firmware qualification, and commissioning now determine whether the accelerator system can operate as specified.
## TL;DR
- A100 and H100 scale up inside an eight-GPU server; GB200 and GB300 NVL72 extend the scale-up domain across 72 GPUs in one rack.
- Published power figures describe different boundaries: GPU design power, server maximum, rack IT load, provisioned capacity, and facility load are not interchangeable.
- NVIDIA describes GB200 at about 120 kW of rack consumption and GB300 at up to 142 kW; the labels are related but not identical measurements.
- Liquid cooling does not automatically remove air cooling. Published hybrid designs still leave roughly 13% to 30% of rack heat in the air.
- Rubin's proposed next step makes power allocation visible to workload software, but public production measurements remain behind the architecture claims.
## The deployment unit moved from server to rack
The useful distinction is scale-up versus scale-out.
**Scale-up** joins processors into a tightly coupled domain with enough bandwidth and low enough latency for frequent model communication. **Scale-out** connects those systems into a larger cluster through a network.
In [DGX A100](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/dgx-a100/dgxa100-system-architecture-white-paper.pdf) and [DGX H100](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-dgx-h100-datasheet.pdf), eight GPUs share a local NVSwitch fabric. Communication beyond those eight GPUs crosses ConnectX adapters and the scale-out network.
In NVL72, NVLink becomes rack infrastructure. Eighteen compute trays connect through nine NVLink switch trays, making all 72 GPUs part of one rack-wide scale-up domain. InfiniBand or Spectrum-X still connects multiple racks, but more model communication can remain inside the rack.
NVIDIA's published [Vera Rubin Pod](https://developer.nvidia.com/blog/nvidia-vera-rubin-pod-seven-chips-five-rack-scale-systems-one-ai-supercomputer/) expands the physical system further: 40 racks, 1,152 Rubin GPUs, and five purpose-built rack-scale system types spanning compute, CPU capacity, inference, storage, and networking.
The rack does not eliminate the distributed system. It moves one of its most performance-sensitive boundaries.
| Generation | Tightly coupled scale-up unit | Scale-out boundary | Operator's primary integration unit |
| ---------------- | ----------------------------- | ---------------------------------------- | ----------------------------------- |
| A100 | Eight-GPU server | Between servers | Server |
| H100 | Eight-GPU server | Between servers | Server and configurable rack |
| GB200 NVL72 | 72-GPU rack | Between racks | Complete liquid-cooled rack |
| GB300 NVL72 | 72-GPU rack | Between racks | Complete liquid-cooled rack |
| Vera Rubin NVL72 | 72-GPU rack | Between racks and specialized rack roles | Rack and pod |
Blackwell still exists in smaller server configurations. The claim is narrower: NVL72 introduced a top-end configuration whose scale-up and service boundary is the rack.
## Five different numbers are called power
The fastest way to misunderstand AI infrastructure is to place every published watt figure on one line.
An A100 GPU's design power, a DGX H100 server maximum, a GB300 rack envelope, and a facility's utility demand measure different things. Provisioned electrical capacity adds redundancy and derating. Cooling equipment has its own capacity and power consumption. None of those numbers automatically describes energy per completed model request.
| Term | What it measures | What it does not establish |
| ------------------------- | ----------------------------------------------------------------------------- | ------------------------------------ |
| GPU TDP or TGP | Thermal or board-power envelope for one accelerator | Actual workload draw or server input |
| Server maximum power | Highest documented input for the full server | Typical or sustained production draw |
| Rack IT load | Power entering the IT equipment inside the rack boundary | Utility-side facility demand |
| Provisioned rack capacity | Breaker, busway, PDU, or power-shelf capacity including margin and redundancy | Expected consumption |
| Facility load | IT plus distribution losses, pumps, fans, chillers, and heat rejection | A fixed multiple of the rack label |
The [U.S. Department of Energy's data-center metering guide](https://betterbuildingssolutioncenter.energy.gov/sites/default/files/attachments/Data_Center_Metering_and_Resource_Guide.pdf) separates power from energy and stresses measurement at explicit boundaries. Product specifications help size infrastructure. Meters show what the deployed workload consumed.
The generation table therefore preserves the original label and system boundary:
| System | Published figure | Boundary | Cooling | Evidence meaning |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------: | ------------------------------------- | ------------------------------- | --------------------------------------------- |
| [DGX A100](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/dgx-a100/dgxa100-system-architecture-white-paper.pdf) | 6.5 kW | Eight-GPU server | Air | Maximum system specification |
| [DGX H100](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-dgx-h100-datasheet.pdf) | About 10.2 kW | Eight-GPU server | Air | Maximum system power |
| [GB200 NVL72](https://www.nvidia.com/en-us/data-center/gb200-nvl72/) | About 120 kW | 72-GPU rack | Direct liquid plus residual air | Approximate rack consumption |
| [GB300 NVL72](https://www.nvidia.com/en-us/data-center/gb300-nvl72/) | Up to 142 kW | 72-GPU rack | Direct liquid plus residual air | Product maximum |
| [Schneider GB200 facility design](https://download.se.com/files?p_Doc_Ref=RD109DSR0_EN&p_enDocType=Application%20Notes) | 132 kW | Facility sizing point for one IT rack | Hybrid liquid and air | Published design maximum with allowance |
| [Future 800 VDC systems](https://www.opencompute.org/blog/new-ocp-800-vdc-power-collaboration-to-create-an-open-architecture-for-ai-data-centers) | Toward 1 MW | Roadmap rack class | Future liquid-cooling designs | Architecture target, not current product draw |
```mermaid
xychart-beta
title "Eight-GPU server maximum power"
x-axis ["DGX A100", "DGX H100"]
y-axis "kW" 0 --> 12
bar [6.5, 10.2]
```
```mermaid
xychart-beta
title "Published NVL72 rack power figures"
x-axis ["GB200 consumption", "GB300 up to"]
y-axis "kW" 0 --> 150
bar [120, 142]
```
The two charts use different boundaries and labels. They are not an efficiency comparison. GB200 and GB300 contain nine times as many GPUs as a DGX server, plus CPUs, NVSwitch trays, networking, and rack-level power equipment.
Four DGX H100 systems can create a rack above 40 kW, but [NVIDIA's H100 electrical design](https://docs.nvidia.com/dgx-superpod/design-guides/dgx-superpod-data-center-design-h100/latest/electrical.html) allows operators to reduce density when local power or cooling cannot support that configuration. NVL72 is less divisible: the rack-scale fabric is part of the product.
## A 140 kW rack connects directly to the building
A conventional server rack receives a manageable set of branch circuits and rejects most heat into room air. A 120–142 kW NVL72 system changes both paths.
[Schneider's GB200 facility design](https://download.se.com/files?p_Doc_Ref=RD109DSR0_EN&p_enDocType=Application%20Notes) traces power through medium-voltage equipment, transformers, UPS modules, distribution panels, overhead busway, rack feeds, power shelves, busbars, and point-of-load conversion. [Dell's IR7000 rack specification](https://www.dell.com/support/manuals/en-us/poweredge-rack-ir7000/ir7000_ism_pub/technical-specifications) supports as many as eight power feeds and two internal busbars. Installed conversion capacity can exceed expected rack consumption because the design must preserve operation during a failed feed or power shelf.
That distinction matters. Eight 33 kW GB200 power shelves create 264 kW of installed conversion capacity, but [NVIDIA documents them as two redundant power zones](https://docs.nvidia.com/dgx/dgxgb200-user-guide/hardware.html). The rack is not a 264 kW load.
The power path becomes a mechanical problem as well as an electrical one. Raising voltage reduces current for the same delivered power. One megawatt at 50 volts represents 20,000 amps; at 800 volts it represents 1,250 amps before losses and engineering margin.
That is why [NVIDIA](https://blogs.nvidia.com/blog/800-vdc-power-architecture-ai-factory/), [Eaton](https://www.eaton.com/us/en-us/catalog/data-center-power-distribution/eaton-800-vdc-sidecar.html), and the [Open Compute Project](https://www.opencompute.org/blog/new-ocp-800-vdc-power-collaboration-to-create-an-open-architecture-for-ai-data-centers) are developing 800 VDC rack-distribution architectures. These are roadmap systems associated with future 600 kW to 1 MW rack classes, not proof that megawatt racks are broadly shipping.
## Liquid cooling is a chain of systems
Direct liquid cooling does not mean that facility cooling disappears.
Cold plates collect heat from GPUs and CPUs into a technology cooling loop. A coolant distribution unit, or CDU, controls that clean loop and transfers its heat through a heat exchanger. A facility-water loop then moves the heat toward a dry cooler, cooling tower, or chiller.
```mermaid
flowchart LR
CHIP["GPU and CPU cold plates"] --> TCS["Technology cooling loop"]
TCS --> HX1["CDU heat exchanger"]
HX2["CDU heat exchanger"] --> FWS["Facility water loop"]
FWS --> HR["Heat rejection
dry cooler, tower, or chiller"]
HX1 -. "heat transfer
fluids remain isolated" .-> HX2
AIR["Residual rack air heat"] --> ROOM["Room cooling and humidity control"]
```
The CDU is a hydraulic boundary, a control system, and a failure domain. A cooling-loop failure must therefore participate in the rack's protection and shutdown logic.
Published designs also show why the word _liquid-cooled_ needs a qualifier:
| Published design | Heat moved through liquid | Residual air load |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------: | ----------------: |
| [Schneider GB200 design](https://download.se.com/files?p_Doc_Ref=RD109DSR0_EN&p_enDocType=Application%20Notes) | 87% | 13% |
| [Vertiv high-density AI design](https://vertiv.com/49e8a2/globalassets/documents/ai-hub-reference-design/vertiv-360ai-reference-design-007.pdf) | About 80% | About 20% |
| [Typical HPE hybrid description](https://www.hpe.com/us/en/newsroom/press-release/2024/10/hpe-unveils-industrys-first-100-percent-fanless-direct-liquid-cooling-systems-architecture.html) | About 70% | About 30% |
| [HPE fanless architecture](https://www.hpe.com/us/en/newsroom/press-release/2024/10/hpe-unveils-industrys-first-100-percent-fanless-direct-liquid-cooling-systems-architecture.html) | Up to 100% | Up to 0% |
These are different product and facility assumptions, not one industry progression. Memory, networking, storage, and power-conversion equipment can leave a material residual air load after the accelerators move to water.
Warm-water designs change the heat-rejection path. [NVIDIA](https://developer.nvidia.com/blog/nvidia-vera-rubin-pod-seven-chips-five-rack-scale-systems-one-ai-supercomputer/), [Lenovo](https://lenovopress.lenovo.com/lp2357-lenovo-nvidia-gb300-nvl72-rack-scale-ai), and [Supermicro](https://www.supermicro.com/en/solutions/liquid-cooling) describe newer systems accepting inlet water near 45°C. Higher water temperatures can expand the hours when a facility rejects heat without compressor-based chilling. They do not remove the flow calculation: heat load, temperature rise, pipe size, pressure drop, and pump energy remain linked.
Redundancy crosses the electrical and thermal systems. [Schneider's published design](https://download.se.com/files?p_Doc_Ref=RD109DSR0_EN&p_enDocType=Application%20Notes) specifies N+1 UPS modules, N+1 PDUs, and N+1 CDUs, but each protects a different boundary. Pumps, controls, and CDUs need backed-up power if the compute equipment must survive a utility or distribution event.
## Rack density becomes a utility and siting constraint
The facility problem does not stop at the busway and CDU.
[NVIDIA's GB200 SuperPOD reference architecture](https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/dgx-superpod-architecture.html) groups eight NVL72 racks into a 576-GPU scalable unit. At the published rack design points, the compute racks alone move the electrical requirement into the megawatt class before adding scale-out networking, storage, pumps, residual air cooling, and conversion losses. A campus built from many scalable units therefore depends on upstream transformers, substations, utility interconnection, generator strategy, and the rate at which power capacity can be added.
Heat rejection has the same aggregation problem. The technology loop can remain closed and clean while transferring heat through CDUs, but the facility still needs somewhere to reject that heat. The [Schneider reference design](https://download.se.com/files?p_Doc_Ref=RD109DSR0_EN&p_enDocType=Application%20Notes) shows how dry coolers, chillers, or evaporative systems change the mechanical boundary. Climate, available land, noise limits, water availability, and local operating rules determine which path is practical.
“Liquid cooled” does not by itself describe water consumption. A closed secondary loop moves heat without continually consuming its full flow volume. Water use depends on the facility-side rejection design, blowdown and treatment, local weather, and whether the site uses dry or evaporative cooling. The useful planning questions are therefore separate:
| Site constraint | Question the rack specification cannot answer |
| --------------------------- | -------------------------------------------------------------------------------------- |
| Utility capacity | When can the required megawatts actually be energized? |
| Substation and distribution | Which equipment, redundancy, and fault boundaries carry the aggregate load? |
| Heat-rejection site | Where can the rejected heat leave the facility across seasonal conditions? |
| Water | Does the selected rejection system consume water, and is that supply durable? |
| Expansion sequence | Can electrical and mechanical capacity arrive in the same increments as compute racks? |
The rack is the deployable computer, but the utility connection and heat-rejection plant set the maximum size and pace of the fleet.
## Networking becomes part of rack qualification
A100 and H100 racks could be assembled from independently serviceable eight-GPU systems. Their NVLink fabric ended at the server. NVL72 spreads NVLink, NVSwitch, power, and coolant across the rack.
That creates at least three separate traffic classes:
1. **GPU collectives** over ConnectX and the compute fabric.
2. **Models, training data, checkpoints, and storage traffic** over infrastructure paths that can include BlueField.
3. **Management and control** over out-of-band networks connecting BMCs, switches, telemetry, power, and cooling controllers.
[NVIDIA's GB200 fabric documentation](https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/network-fabrics.html) keeps those paths distinct. Its DGX SuperPOD design uses Quantum InfiniBand, while the [GB300 enterprise design](https://docs.nvidia.com/enterprise-reference-architectures/nvl72-ai-factory/latest/overview.html) demonstrates Spectrum-X Ethernet. InfiniBand versus Ethernet is a deployment choice, not a clean generation boundary.
Topology must agree across GPU identity, NIC affinity, switch plane, scheduler placement, and [NCCL's collective topology](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/). A network can pass link tests and still underperform when ranks, rails, and adapters do not align.
The larger scale-up domain creates a larger potential failure domain. Losing an A100 or H100 server removes eight tightly coupled GPUs. Losing or degrading an NVL72 rack can affect a 72-GPU domain. Redundant network planes protect against selected link and switch failures; they do not automatically cover correlated firmware, rack power, coolant, or NVLink control failures.
## The rack must be commissioned as one system
The installation is not complete when the compute trays boot.
The [Open Compute Project's liquid-cooling commissioning guidance](https://www.opencompute.org/documents/ocp-acs-commissioning-requirements-rev0-5-pdf) calls for flushing, filtration, material compatibility, hydraulic balancing, pressure and leak testing, controls validation, and extended observation. Electrical commissioning must test feeds, redundancy, protection, and load behavior. Network commissioning must validate rails, congestion behavior, storage paths, and management isolation.
The physical route into the data hall is part of the plan. [Dell publishes a 1,900 kg maximum payload](https://www.dell.com/support/manuals/en-us/poweredge-rack-ir7000/ir7000_ism_pub/technical-specifications) for its IR7000 rack before site-specific sidecars and external infrastructure. Structural review must cover the final static load, concentrated loads at casters or leveling feet, and the rolling path from the loading dock.
```mermaid
flowchart TB
SITE["Site readiness
power + water + structure + access"]
INSTALL["Rack placement
feeds + manifolds + fabrics"]
FLUSH["Cooling preparation
flush + filter + balance + leak test"]
POWER["Electrical tests
protection + redundancy + load steps"]
NET["Network tests
rails + storage + management isolation"]
BURN["Integrated burn-in
compute + cooling + power + telemetry"]
ACCEPT["Operational acceptance
failure tests + baselines + runbooks"]
SITE --> INSTALL
INSTALL --> FLUSH
INSTALL --> POWER
INSTALL --> NET
FLUSH --> BURN
POWER --> BURN
NET --> BURN
BURN --> ACCEPT
```
This is why OEMs, ODMs, electrical vendors, and cooling vendors now participate in the platform architecture. The operational conclusion is that a qualified rack-plus-facility design is safer than assembling components whose specifications appear individually compatible.
## Rubin makes power visible to the scheduler
Blackwell makes the rack a scale-up computer. Rubin's published architecture begins treating a fixed site-power envelope as a scheduling constraint.
NVIDIA describes 45°C liquid cooling, rack-level power telemetry, dynamic power allocation, and [MaxLPS](https://docs.nvidia.com/dsx/maxlps/overview)—software intended to schedule workloads within a shared power envelope. NVIDIA reports that the mechanism can place more GPUs under the same power budget, but the published documentation labels the example illustrative and the testing incomplete.
Those are vendor tests and evolving designs. NVIDIA's MaxLPS documentation says testing remains in progress. Power steering can reduce short-duration peaks or allocate a fixed envelope more productively; it does not automatically reduce total energy.
The architectural proposal still matters. It connects workload placement to a configured rack or site power budget rather than assuming unconstrained electrical capacity.
```mermaid
flowchart LR
SITE["Configured site power envelope"] --> SCHED["MaxLPS scheduling policy"]
JOBS["Training and inference jobs"] --> SCHED
SCHED --> R1["Rack 1 power cap"]
SCHED --> R2["Rack 2 power cap"]
SCHED --> R3["Rack 3 power cap"]
```
## What production evidence proves
[Oracle's account of GB200 deployment](https://blogs.oracle.com/cloud-infrastructure/behind-the-scenes-scale-nvidia-gb200-nvl72-deployments) confirms the physical problem: racks above 120 kW at peak, high-density three-phase delivery, direct-to-chip liquid loops, CDUs, and rack-level monitoring. [CoreWeave's NVL72 documentation](https://docs.coreweave.com/platform/instances/nvl72) treats the full 18-node rack as the deployment unit.
Those deployments establish NVL72 as a production rack system. They do not disclose calibrated workload traces, full cooling energy, water use, availability, or equal-work comparisons against other platforms.
A useful independent comparison would hold constant:
- model and output-quality target;
- prompt, context, output length, concurrency, and latency targets;
- completed work under power caps and component failures;
- rack IT energy, facility energy, cooling energy, and water use;
- operator effort, repair time, and degraded operation.
[MLPerf's power policy](https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc) is stronger than quoting TDP because it measures a defined system under test at the wall. It still does not answer the facility question unless the submitted boundary includes the rack, network, pumps, and heat-rejection systems being compared.
## What's missing
Public specifications now describe rack-wide compute systems in detail. Public operating evidence remains thinner.
The missing dataset includes sustained rack input under representative workloads, idle and capped behavior, sub-second power transients, pump and CDU consumption, residual room-cooling energy, water use, component failure rates, repair time, and workload goodput at equal facility energy.
Rubin widens that gap. Its pod composition, 45°C cooling, power steering, and future 800 VDC distribution create a coherent systems argument. Independent Rubin measurements and complete public facility envelopes were not available at this cutoff.
## So what
The accelerator choice now reaches further into the building.
An NVL72 purchase commits more than GPU capacity. It shapes the scale-up domain, network rails, electrical feeds, coolant loops, rack weight, commissioning process, failure model, and the software that coordinates all of them.
The evidence suggests that NVIDIA gains a platform advantage by co-designing these layers and publishing one validated system boundary. A multi-vendor alternative must assign someone to qualify the boundaries between suppliers.
The unresolved question is operational rather than numerical: if workload software begins reallocating power across racks in real time, who owns the safety contract between the scheduler and the electrical plant—and what independent test proves that contract fails safely when a workload surge and a power-path failure happen together?
---
_Part of the [AI Compute Landscape](https://artificialcuriositylabs.ai/series/ai-compute/) and the NVIDIA chapter. [Start with The GPU Stopped Being the Product](https://artificialcuriositylabs.ai/posts/2026-08-30-nvidia-gpu-stack-evolution/), then continue with [The AI Buildout Is Constrained by Its Slowest Layer](https://artificialcuriositylabs.ai/posts/2026-08-30-ai-buildout-constrained-slowest-layer/)._