Groq Makes the Compiler Part of the Processor

Groq’s architectural bet is that an inference processor should execute a plan, not discover one while the model is running.

The Language Processing Unit removes much of the reactive machinery found in conventional processors: hardware caches, dynamic arbitration, speculative execution, and out-of-order scheduling. Its compiler decides where operations run, where data lives, and when each value moves. The processor then follows that schedule across a large on-chip SRAM system.

That changes the unit worth evaluating. One LPU does not hold a modern large language model. The useful machine is the connected system of LPUs, its compiler-generated schedule, and the cloud service around it.

TL;DR

• Groq moves instruction scheduling, tensor placement, and much data movement into the compiler, leaving simpler hardware to execute a predetermined program.
• SRAM supplies high bandwidth and predictable access time, but limited capacity forces model weights and execution across many connected LPUs.
• Deterministic chip execution reduces one source of latency variation; it does not remove API networking, admission control, queueing, or shared-service contention.
• Groq is strongest where one user’s token rate matters: voice, coding loops, agents, and other interactive inference. That is different from maximizing total batched throughput.
• NVIDIA did not acquire Groq. It licensed Groq’s inference technology non-exclusively and hired Groq leaders and staff; Groq remains independent and continues to operate GroqCloud.

The processor begins in the compiler

Modern GPUs are also compiler-driven machines. Framework graphs are lowered into kernels, kernels are compiled, and serving engines plan memory and batch work. The distinction is what remains unresolved at runtime.

A GPU retains hardware mechanisms that react to events while a program executes: caches answer memory requests, arbiters resolve contention, and schedulers choose from ready work. Groq’s original Tensor Streaming Processor paper describes the opposite cut. It eliminates reactive elements such as caches and arbiters so the compiler can reason about execution timing precisely.

The hardware is functionally sliced. Separate regions perform matrix operations, vector arithmetic, memory access, data movement, and instruction control. Values stream between those regions through a software-controlled interconnect. The compiler maps the model graph onto that physical machine and emits a cycle-level schedule.

Groq describes this as a programmable assembly line: each functional unit receives an instruction, takes data from a named stream, performs its operation, and places the result onto another stream. Hardware does not wait to discover which path wins arbitration because the compiler has already assigned the path and time.

This is not “software instead of hardware.” The compiler works because the hardware was built to expose timing and data movement directly. Groq makes the compiler part of the processor’s effective control plane.

SRAM makes speed possible—and scale necessary

Groq places the primary working memory on the processor in SRAM. SRAM is faster and more predictable than fetching weights from off-chip DRAM or HBM, but it occupies much more silicon area per stored bit.

The first-generation chip described in the architecture literature contains approximately 220 MB of SRAM. Groq’s current eight-accelerator GroqNode lists 1.76 GB of aggregate on-die memory, consistent with eight chips at that capacity. A 70-billion-parameter model requires roughly 35 GB merely to store weights at four bits per parameter, before accounting for scales, activations, code, and key-value cache.

One chip is therefore the wrong capacity boundary. Groq partitions model weights and operations across multiple LPUs, then streams activations through the connected machine. GroqNode exposes 88 chip-to-chip connectors and is positioned for multi-server and multi-rack scaling without a separate external switch in the LPU fabric.

The result resembles a pipeline more than a pool of interchangeable accelerators. Each stage owns part of the model and passes intermediate values to the next stage according to the compiled schedule. Adding LPUs expands both SRAM capacity and execution resources, but it also makes partitioning, communication, and pipeline utilization part of the performance result.

Groq’s latest rack specification makes that system boundary explicit. Its inference platform page lists 256 LPUs, 128 GB of on-chip SRAM, and 40 PB/s of aggregate SRAM bandwidth per rack for the Groq 3 generation. Those are vendor specifications, not independently measured application results.

The architectural trade is clear: spend silicon and system complexity on low-latency SRAM and predictable movement, then recover model capacity by joining many processors into one scheduled machine.

Deterministic execution is not deterministic service

Groq uses “deterministic” precisely at the accelerator layer. Given the same compiled program, the LPU schedule specifies when instructions execute and when values move. The hardware does not introduce cache misses or arbitration delays that change the cycle count.

A cloud request crosses a larger system:

client network
  → API gateway
  → admission control and queue
  → model replica selection
  → prompt processing
  → scheduled LPU execution
  → streamed response

Only one part of that chain is the deterministic machine. Request bursts can still create queues. Prompts have different lengths. Tool calls introduce external dependencies. Network paths vary. A shared service can also reserve, batch, or route work differently as load changes.

This distinction matters because latency has several meanings:

Groq’s strongest public evidence is output speed. Its GroqCloud model catalog publishes model-specific rates ranging into hundreds of tokens per second, with smaller models reaching about 1,000 tokens per second. These are service specifications and vary by model.

Artificial Analysis independently measures Groq’s public endpoints. At the August 30, 2026 cutoff, it reports the fastest tracked Groq configuration near 980 output tokens per second. For the same Llama 3.3 70B model, its provider comparison measures Groq at roughly 298 output tokens per second—more than 20 times one of the slower tracked providers.

That is useful independent evidence for interactive speed. It is not a rack-throughput benchmark, a power measurement, or a total-cost comparison. Artificial Analysis measures hosted APIs; it does not expose how many chips, replicas, queued users, or reserved capacity sit behind each endpoint.

The distinction also explains Groq’s workload fit. Fast sequential generation is valuable for conversational voice, coding agents waiting inside a loop, interactive search, and workflows that call a model repeatedly. Offline jobs with large batches care more about total tokens per dollar and total tokens per rack than the token rate of one stream.

SemiAnalysis argues that SRAM-heavy systems trade capacity available for weights and key-value cache against high single-user token rates, while GPUs can win aggregate throughput and cost through batching. That is the right comparison boundary: latency-first and throughput-first inference are different products even when both report “tokens per second.”

GroqCloud is the complete product

The LPU does not arrive as a general accelerator card with a CUDA-like programming surface. GroqCloud presents supported models through an API that is largely compatible with the OpenAI client pattern. The compiler, model artifacts, partitioning, and machine topology remain behind the service boundary.

That lowers adoption work for a supported model: change the endpoint and test behavior. It also moves control to the provider. Model availability, context limits, rate limits, pricing, quantization, and rollout schedules follow the GroqCloud catalog rather than the customer’s own accelerator fleet.

The fit is strongest when:

• the model and operators are supported by Groq’s compiler and service;
• low per-user generation latency changes the product experience;
• the workload is stable enough to justify a compiled deployment;
• API consumption is preferable to owning the hardware and compiler lifecycle.

The fit weakens when:

• the workload requires custom or rapidly changing operators;
• the model is absent from the supported catalog;
• large dynamic key-value caches dominate memory demand;
• maximum batched throughput matters more than individual stream speed;
• infrastructure ownership or artifact control is mandatory.

GroqCloud therefore sells more than processor time. It packages a compiler-controlled distributed machine behind a familiar inference contract.

NVIDIA licensed the technology; it did not acquire Groq

The NVIDIA relationship is easy to overstate because technology and senior personnel moved together.

On December 24, 2025, Groq announced a non-exclusive inference technology licensing agreement with NVIDIA. Founder Jonathan Ross, president Sunny Madra, and other Groq staff joined NVIDIA. The same announcement says Groq would remain an independent company under CEO Simon Edwards and that GroqCloud would continue operating.

Reuters described the structure as a technology license plus executive hiring that stopped short of a formal acquisition. Reported deal values remain reporting, not disclosed transaction terms from either company.

The distinction now appears in two product paths:

1. GroqCloud remains Groq’s independently operated inference service.
2. NVIDIA Groq 3 LPX uses licensed Groq technology inside NVIDIA’s Vera Rubin platform.

NVIDIA’s Groq 3 LPX architecture description specifies a 256-LPU rack with 128 GB of SRAM, 40 PB/s of SRAM bandwidth, and 640 TB/s of rack-scale communication. NVIDIA announced the product in full production in August 2026, with initial cloud availability planned later in the year.

LPX does not establish that every model should run entirely on an LPU rack. NVIDIA presents a heterogeneous design coordinated by Dynamo:

• Vera Rubin GPUs can process prefill, attention, and key-value cache.
• LPX can execute latency-sensitive feed-forward or mixture-of-experts decode stages.
• The two systems exchange intermediate activations according to the selected serving pattern.

NVIDIA calls these patterns prefill-decode disaggregation, attention-FFN disaggregation, and external-drafter speculative decoding. The published numbers—including 3,431 output tokens per second on one 100K-context test—come from NVIDIA-operated LPX systems and vendor-selected configurations, even when Artificial Analysis supplied the test harness.

The evidence floor is narrower than “Groq replaces GPU inference.” LPX shows that a latency-specialized SRAM machine can become one stage inside a larger GPU inference system. GPUs retain the memory-heavy and general-purpose work; LPUs handle selected decode paths where deterministic scheduling and SRAM bandwidth matter most.

What’s missing

Groq has credible evidence for high output speed and a coherent explanation for why the architecture produces it. The public record remains thinner on aggregate system economics.

A complete comparison needs the same model, quality target, prompt distribution, context length, concurrency, and service-level objective across systems. It also needs measured rack power, total output throughput, tail latency, compiler coverage, failed-compilation rate, capacity reservation, and cost under sustained load.

No current public MLPerf result establishes a standardized Groq-to-GPU comparison across those boundaries. GroqCloud API benchmarks prove that users can receive tokens quickly. They do not reveal whether the underlying rack delivers the best aggregate tokens per watt or tokens per dollar.

So what

Groq’s contribution is larger than a fast inference endpoint. It redraws the processor boundary around the compiler and the connected machine.

SRAM makes each operation fast and predictable. The compiler turns that memory and compute into a timed execution plan. The interconnect lets many small memory domains behave like one model-serving pipeline. GroqCloud turns that system into an API product, while NVIDIA’s licensed LPX path places the same architectural idea inside a heterogeneous GPU platform.

The open question is operational: as inference splits into prefill, attention, feed-forward, speculative decoding, and cache management, does the winning system use one broad accelerator—or does the compiler become the place where several specialized processors are assembled into one service?

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.