The GPU Stopped Being the Product

NVIDIA’s most important architectural change is not a faster GPU. It is the expansion of what counts as the computer.

The Tesla P100 arrived as an accelerator inside a server. The A100 made the eight-GPU system a repeatable unit. Hopper joined the CPU and GPU more tightly. Blackwell added a top-end configuration whose scale-up boundary is a liquid-cooled rack. Vera Rubin extends the design across compute racks, networking, power, cooling, and inference orchestration.

At the high end, the product is no longer only a chip. It is a large part of the AI data center assembled as one machine: Tensor Cores change the math, HBM feeds it, NVLink expands the scale-up domain, networking connects racks, and the software stack makes each generation usable.

TL;DR

• P100 established the modern foundation with HBM2, NVLink, and the first DGX system; V100 added Tensor Cores.
• A100 turned the GPU into elastic infrastructure through TF32, structured sparsity, Multi-Instance GPU, faster NVLink, and a standardized eight-GPU NVSwitch system.
• Hopper optimized transformers directly through FP8 and Transformer Engine, while Grace Hopper introduced coherent CPU-GPU coupling.
• Blackwell introduced a top-end 72-GPU liquid-cooled rack with 130 TB/s of summed bidirectional NVLink endpoint bandwidth, while B200 also remained available in server configurations.
• Vera Rubin doubles NVLink bandwidth again and raises HBM bandwidth to 22 TB/s per GPU. Groq-derived LPX racks add specialized inference paths, while Rubin remains in production ramp rather than broad availability.

Start before A100

The useful starting point is the Tesla P100 and Pascal GP100 architecture in 2016. P100 introduced three pieces that still define NVIDIA’s systems:

1. HBM on the GPU package for high-bandwidth access to model data.
2. NVLink for GPU-to-GPU communication beyond PCIe.
3. DGX-1, an integrated eight-GPU server with a tested software stack.

P100 did not have Tensor Cores. Its AI acceleration came from running FP16 arithmetic at twice the rate of FP32 on ordinary CUDA cores. The Tesla V100 Volta architecture made the next decisive move in 2017: 640 first-generation Tensor Cores dedicated to matrix multiplication.

Pascal made the GPU a better parallel processor. Volta began shaping its numerical machinery around neural networks.

The names hide overlapping product lines. H200 is a Hopper memory upgrade, while Blackwell Ultra expands Blackwell’s memory and low-precision throughput. Grace Hopper, GB200, and Vera Rubin are platforms spanning CPUs, GPUs, memory, interconnects, and systems.

The generation map

The table uses representative high-end data-center configurations. SXM and rack-scale modules run at different power and bandwidth levels from PCIe cards with the same architecture. Peak Tensor Core figures also change numerical format across generations, so they are not one continuous performance benchmark.

The status boundary matters. On August 27, AWS received its first Vera CPU server and Rubin GPU for deployment work. NVIDIA still describes production shipments as beginning in fall 2026. Early component delivery is not general availability of complete NVL72 systems.

Three caveats matter:

• NVIDIA’s NVLink numbers add transmit and receive bandwidth. They are not one-way payload rates.
• Sparse throughput assumes the required structured sparsity pattern. It is not dense performance.
• FP16, TF32, FP8, FP4, and NVFP4 trade precision, range, storage, and compute differently. A larger headline FLOPS number at a narrower format does not mean every workload runs proportionally faster.

Component 1: Compute changed from general arithmetic to model-aware arithmetic

P100 accelerated FP16 on CUDA cores. V100 added dedicated Tensor Cores that multiplied FP16 matrices and accumulated in FP32, giving neural-network matrix operations their own hardware.

A100’s third-generation Tensor Cores added TF32, BF16, IEEE FP64, INT8, INT4, and 2:4 structured sparsity. One matrix engine now covered training, inference, and scientific computing.

The H100 architecture added FP8 and the first Transformer Engine, coordinating lower- and higher-precision execution by transformer layer.

Blackwell extended that direction to FP6 and FP4 with a second-generation Transformer Engine. Rubin’s third-generation Transformer Engine adds sixth-generation Tensor Cores and up to 50 petaflops of NVFP4 inference compute according to NVIDIA. That is a vendor peak, not a cross-generation application benchmark.

Component 2: Memory became as important as compute

Neural networks repeatedly move weights, activations, and cache state. More arithmetic units do not help when those units wait for data.

P100’s HBM2 placed stacked memory beside the GPU on a silicon interposer. It delivered 16 GB at 720 GB/s. V100 raised bandwidth to 900 GB/s. The original A100 reached 1.555 TB/s with 40 GB, and the later A100 80 GB exceeded 2 TB/s.

H100 increased bandwidth to 3.35 TB/s, but H200 demonstrates why capacity and bandwidth must be separated. H200 retained Hopper compute while moving to 141 GB of HBM3e at 4.8 TB/s. On memory-intensive large-model inference, that change can matter without a new Tensor Core generation.

Blackwell increased the representative B200 package to 180 GB at up to 8 TB/s. Blackwell Ultra reached 288 GB. Rubin keeps 288 GB but replaces HBM3e with HBM4 and raises the vendor-published peak to 22 TB/s.

The chart keeps one physical quantity on the axis, but it combines product introductions, later memory variants, and Rubin’s production-ramp specification. Every value is a published peak rather than achieved application bandwidth.

Rubin’s published HBM bandwidth is more than 30 times P100’s. Power and cooling rose with it: P100 SXM was rated at 300 watts, A100 SXM at 400 watts, H100 SXM at up to 700 watts, and B200 at up to 1,000 watts per GPU.

Component 3: Packaging and CPU coupling expanded the processor boundary

P100, V100, A100, and H100 were large monolithic GPU dies. Each generation pushed close to the practical manufacturing limit for one piece of silicon.

Blackwell changed the package. Two reticle-limited dies connect through a 10 TB/s chip-to-chip interface and present themselves as one CUDA GPU. NVIDIA could increase transistor count to 208 billion without requiring one manufacturable die of that size.

Rubin continues the dual-reticle approach with 336 billion transistors. NVIDIA calls the inter-die connection NV-HBI. From the programming layer, the goal remains one GPU abstraction. Underneath it, the package has become a tightly integrated multi-die system.

Once a “GPU” contains multiple compute dies, HBM stacks, high-speed die links, NVLink interfaces, and control engines, the package is already a system.

Grace Hopper tightened that relationship. A Grace CPU and Hopper GPU communicate over coherent NVLink-C2C, giving the CPU and GPU a higher-bandwidth path and a coherent memory relationship inside one superchip.

Blackwell’s GB200 platform connects one Grace CPU to two Blackwell GPUs. The GB200 NVL72 rack contains 36 Grace CPUs and 72 Blackwell GPUs. Vera Rubin replaces Grace with the Vera CPU while retaining the 36-to-72 relationship.

This creates two different fabrics:

• NVLink-C2C connects the CPU and GPU inside a superchip.
• NVLink and NVSwitch connect GPUs across the scale-up domain.

PCIe remains the host and peripheral fabric, while NVLink-C2C and NVLink carry the tightly coupled processor paths.

Component 4: NVLink turned multiple GPUs into one scale-up computer

PCIe connects devices to a host. NVLink exists because tightly coupled GPU workloads need a different path.

P100 supported four first-generation NVLinks for 160 GB/s of aggregate bidirectional bandwidth per GPU. DGX-1 arranged eight GPUs in a hybrid cube mesh, so not every pair had a dedicated direct connection.

V100 increased per-GPU NVLink bandwidth to 300 GB/s. DGX-2 then introduced NVSwitch, replacing the fixed mesh with a switched 16-GPU fabric.

A100 doubled per-GPU bandwidth to 600 GB/s and standardized an eight-GPU fully connected NVSwitch baseboard. H100 reached 900 GB/s, while external NVLink Switch systems stretched the scale-up fabric beyond one server.

Blackwell moved the fabric across an entire rack. GB200 and GB300 NVL72 connect 72 GPUs through fifth-generation NVLink at 1.8 TB/s per GPU, producing NVIDIA’s rounded 130 TB/s aggregate endpoint figure.

Rubin doubles the per-GPU figure to 3.6 TB/s and the 72-GPU aggregate to 260 TB/s.

The directionality deserves emphasis. A100’s 600 GB/s means 300 GB/s in each direction. Blackwell’s 1.8 TB/s and Rubin’s 3.6 TB/s are also bidirectional endpoint totals. The 130 TB/s and 260 TB/s rack numbers multiply those endpoint figures by 72 GPUs. They do not state the fabric’s one-way bisection bandwidth.

The scale-up domain supports frequent, low-latency communication inside one model execution. Scale-out networking joins systems and racks into a larger cluster.

Component 5: Networking and cooling expanded the deployment boundary

NVIDIA’s 2020 acquisition of Mellanox brought the network into the platform.

Mellanox added several distinct components:

• ConnectX network adapters move data between GPU systems.
• Quantum InfiniBand provides a purpose-built scale-out fabric for large compute clusters.
• Spectrum-X Ethernet combines Ethernet switches, adapters, congestion control, and software for AI traffic.
• BlueField DPUs handle infrastructure work including networking, storage, isolation, and security.

Successive NVIDIA systems used 100 Gb/s EDR, 200 Gb/s HDR, 400 Gb/s NDR or Spectrum-X, and then 800 Gb/s Quantum-X800 and Spectrum-X800. Rubin adds ConnectX-9 while retaining InfiniBand and Ethernet scale-out paths.

NVLink and InfiniBand are not competing names for the same connection:

The deployment unit grew with the fabric. The first DGX-1 was a 3U, air-cooled server rated at 3.2 kW. DGX A100 grew to 6U and 6.5 kW, while DGX H100 reached roughly 10.2 kW.

GB200 NVL72 changes the boundary: 72 GPUs, 36 CPUs, nine NVSwitch trays, power shelves, networking, and direct-to-chip liquid cooling form one rack-scale system. NVIDIA documents approximately 120 kW for that rack, while GB300 NVL72 raises the published envelope to about 142 kW.

Those figures do not compare efficiency because the measured boundary grows from one eight-GPU server to an entire 72-GPU rack. They show the architectural transition: electrical delivery, cooling, switching, and compute are now specified together.

Component 6: CUDA expanded into an operating stack for AI infrastructure

CUDA is the longest-running source of continuity. The CUDA programming model, drivers, compilers, and libraries let applications survive multiple hardware generations without starting over.

The stack around CUDA kept expanding:

Volta exposed Tensor Cores through CUDA 9 and accelerated libraries. Ampere added TF32, sparsity, CUDA Graphs, and MIG. Hopper paired FP8 hardware with Transformer Engine. Blackwell paired FP4 with TensorRT-LLM, NeMo, and rack-scale NVLink. Dynamo moved optimization above the model server into routing, KV-cache placement, prefill/decode separation, and worker-to-worker data movement.

CUDA compatibility is valuable, but it is not magic portability. The CUDA compatibility model has minimum driver requirements and version boundaries. Native cubins target specific compute capabilities. TensorRT engines can be tied to library versions and GPU assumptions. MIG profiles differ by architecture. An optimized FP8 or NVFP4 execution recipe may need rebuilding for a different generation.

The application logic may travel. The highest-performance execution artifact often does not.

Groq adds a specialized inference path

The Groq relationship adds one important direction to NVIDIA’s architecture: the GPU does not need to execute every phase of inference.

This was not an acquisition of Groq. In December 2025, Groq announced a non-exclusive license of its inference technology to NVIDIA. Several Groq leaders and team members joined NVIDIA, while Groq remained an independent company and continued operating GroqCloud.

NVIDIA used the licensed technology in the Groq 3 LPX rack, which it positions beside Vera Rubin NVL72. One representative pattern separates prefill from decode: Rubin processes the prompt and creates the KV cache, LPX generates output tokens, and NVIDIA Dynamo coordinates routing and data movement.

The pattern matters more than the product pairing. NVIDIA is extending its platform around heterogeneous inference—general-purpose GPUs for broad model execution, specialized processors for latency-sensitive phases, and software that decides where each phase runs.

What measured performance can actually tell us

Peak specifications describe hardware ceilings. Vendor platform claims describe selected workloads and configurations. Neither is a substitute for a controlled benchmark.

The methodologically cleanest comparison in the available results is MLPerf Training v0.7. NVIDIA submitted an eight-GPU V100 DGX-1 and an eight-GPU DGX A100 to the same BERT benchmark under the same release. Applying MLPerf’s rule of discarding the fastest and slowest runs and averaging the remaining eight gives roughly 166.2 minutes for V100 and 49.0 minutes for A100—about 3.4 times faster.

The result combines hardware and generation-appropriate software rather than isolating the chip—which matches how users experience the system.

Later comparisons are harder:

• NVIDIA reported up to 6.7 times higher H100 performance than its first A100 submission in MLPerf Training v2.1, but “up to” selects the strongest workload and spans different submission vintages.
• In MLPerf Inference v4.0, NVIDIA reported about 28% greater H200 performance than H100 at the same 700-watt envelope and up to 45% at 1,000 watts. The equal-power result is the cleaner evidence for HBM3e’s contribution.
• Blackwell and Blackwell Ultra have measured MLPerf submissions, but accelerator count, numerical format, model, power, and software must match before calculating a ratio.
• Rubin had no comparable public MLPerf submission by August 30, 2026. Its performance and cost claims remain NVIDIA measurements until reproducible benchmark systems appear.

There is no honest single line for “GPU performance over time.” Workload, precision, sparsity, GPU count, power, and software all move.

What’s missing

The stack still lacks a neutral system benchmark that matches how current AI infrastructure is purchased and operated.

MLPerf is the strongest public mechanism available, but the product boundary keeps expanding faster than the benchmark boundary. A useful rack-scale comparison would hold all of these constant:

• model and quality target;
• prompt and output-length distribution, including concurrency;
• time to first token and inter-token latency;
• sustained throughput under concurrency;
• training or inference precision;
• dense versus sparse execution;
• rack, network, cooling, and facility power;
• software engineering required to reach the result;
• failure recovery and degraded-operation behavior;
• acquisition price and cost per completed unit of work.

Without that boundary, a vendor can compare one Blackwell rack with many Hopper servers, or FP4 with FP8, or projected utilization with measured throughput. The numbers may all be correct while the comparison remains unhelpful.

Rubin makes the gap larger. NVIDIA now describes benefits from GPU architecture, HBM4, NVLink 6, Vera CPUs, Spectrum-X, power smoothing, liquid cooling, Dynamo, and workload scheduling in one platform claim. That is a reasonable systems argument. It also makes independent reproduction much harder.

So what

NVIDIA’s evolution from P100 to Rubin is the story of bottlenecks moving outward.

P100 attacked memory and PCIe. V100 attacked matrix math. A100 attacked precision, utilization, and multi-tenancy. Hopper attacked transformer execution. Blackwell attacked the server boundary. Rubin attacks the rack and pod as operating systems for continuous inference.

That changes how to evaluate the company and its competitors. Comparing accelerator FLOPS is now the narrowest possible cut. The real comparison is the amount of useful model work a complete system produces within its power, latency, reliability, and software constraints.

The GPU did not disappear. It became the center of a much larger machine.

The open question is whether this integration remains one durable platform advantage or creates enough cost, power, and operational complexity for the stack to split again—between general-purpose GPUs, specialized inference processors, open interconnects, and software that can schedule work across all of them.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.