--- title: "d-Matrix Moves the Math Into Memory" description: "d-Matrix treats inference as a memory-placement problem, combining digital in-memory compute, an SRAM performance tier, LPDDR capacity, and standard data-center fabrics." canonical_url: "https://artificialcuriositylabs.ai/posts/2026-08-30-d-matrix-moves-math-into-memory/" md_url: "https://artificialcuriositylabs.ai/posts/2026-08-30-d-matrix-moves-math-into-memory.md" published_at: "2026-08-30T07:00:00.000Z" tags: - "ai-infrastructure" - "inference" - "semiconductors" - "memory" - "d-matrix" --- Inference is becoming a memory-placement problem. A modern accelerator can perform arithmetic faster than it can repeatedly move model weights into the compute engines. Autoregressive decoding makes the mismatch visible: every generated token requires another pass through the model, often at a batch size small enough that raw matrix throughput is not the limiting resource. [d-Matrix Corsair](https://www.d-matrix.ai/product/) attacks that boundary by placing digital arithmetic beside a large SRAM tier. The system still needs off-chip capacity memory, PCIe, Ethernet, compilation, quantization, and distributed execution. The architectural bet is narrower and more interesting than “replace the GPU”: keep the latency-critical working set near the arithmetic and use the rest of the memory hierarchy deliberately. ## TL;DR - Corsair combines digital in-memory compute with 2 GB of high-bandwidth SRAM per card and up to 256 GB of off-chip capacity memory. - The useful deployment boundary expands from a paired PCIe card to an eight-card server and then a 64-card rack because larger models must be partitioned across multiple SRAM pools. - d-Matrix reduces physical adoption friction by using standard PCIe and Ethernet, but model compilation, compression, partitioning, and numerical validation remain accelerator-specific. - The clearest near-term pattern may be heterogeneous inference: use a specialist for a memory-bound phase while GPUs retain compute-heavy phases. ## The bottleneck is the trip to memory The arithmetic in transformer inference is not uniform. Prefill processes many prompt tokens in parallel and can keep a large compute engine busy. Decode generates tokens sequentially. At small batch sizes, the accelerator repeatedly reads model weights to produce relatively little new work. The service becomes sensitive to memory bandwidth, placement, and communication latency. Corsair puts a digital in-memory compute engine beside what d-Matrix calls **Performance Memory**. One card provides 2 GB of that SRAM at a published 150 TB/s, plus as much as 256 GB of **Capacity Memory** off chip. A two-card pair doubles those figures and connects through the company’s DMX Bridge. Those two tiers serve different jobs: | Memory tier | Published card capacity | Architectural job | Main constraint | | ----------------------- | -----------------------------------------------------------------------------: | -------------------------------------------------------------------------- | ------------------------------------------------------- | | Performance Memory | 2 GB SRAM | Keep latency-sensitive weights and activations next to the compute engines | Capacity | | Capacity Memory | Up to 256 GB | Hold larger models and support throughput-oriented execution | Lower bandwidth and greater movement cost | | Distributed rack memory | 128 GB aggregate SRAM and up to 16.4 TB capacity in the 64-card reference rack | Partition larger models and requests across cards and servers | Fabric latency, partitioning, and software coordination | This is not one flat pool. The compiler and runtime must decide what fits in the fast tier, how the model is divided, and when data moves between cards or servers. ```mermaid flowchart LR MODEL["Model checkpoint"] COMP["Aviator compiler
compression and partitioning"] CAP["Capacity Memory
larger model storage"] PERF["Performance Memory
SRAM beside compute"] DIMC["Digital in-memory
compute engines"] FABRIC["DMX Bridge, PCIe,
or Ethernet"] NEXT["Next card or
inference stage"] MODEL --> COMP COMP --> CAP COMP --> PERF CAP --> PERF PERF <--> DIMC DIMC --> FABRIC FABRIC --> NEXT ``` The diagram captures the trade: less distance between arithmetic and the fast memory tier, followed by more responsibility for placement and distributed execution. ## A card is not the complete product Corsair uses an industry-standard full-height, full-length PCIe Gen5 form factor. That makes it physically familiar, but the deployment unit changes with the model. The company describes three reference boundaries: | Boundary | Configuration | Published memory boundary | Intended use | | --------- | --------------------------- | -------------------------------------------------------- | --------------------------------------- | | Card pair | Two connected Corsair cards | 4 GB Performance Memory; up to 512 GB Capacity Memory | Smaller models and low-latency stages | | Server | Eight cards | 16 GB Performance Memory; up to 2 TB Capacity Memory | Multi-card inference within one host | | Rack | Eight servers, 64 cards | 128 GB Performance Memory; up to 16.4 TB Capacity Memory | Larger models and higher request volume | The aggregate SRAM determines which model weights can remain in the fastest tier. d-Matrix describes smaller models as an appliance-level fit and larger models as a rack-level problem. That is why the system architecture matters more than the card specification. Within a node, PCIe and DMX Bridge connect the accelerators. Across nodes, the design uses Ethernet. d-Matrix’s [JetStream](https://www.d-matrix.ai/why-we-decoupled-execution-to-accelerate-i-o/) NIC is intended to let accelerators coordinate transfers without routing every step through the host control path. The company says each JetStream connects four Corsair cards to a leaf switch and extends accelerator-to-accelerator execution across servers. That is still a distributed system. The architecture reduces some host-mediated handshakes; it does not remove switches, congestion, topology, failure handling, or the need to keep model partitions synchronized. ## The software stack carries the memory policy In-memory compute does not make a PyTorch model automatically executable. The [Aviator software stack](https://www.d-matrix.ai/product/) contains model templates, compression tools, an MLIR-based compiler, a distributed inference engine, host runtime, device firmware, and operational tooling. It integrates with PyTorch and Triton DSL at the upper boundary, then produces artifacts for Corsair’s spatial architecture and numerical formats. | Software layer | Responsibility | | ----------------- | ------------------------------------------------------------ | | Model Factory | Provides distributed model templates | | Compressor | Converts weights into supported block floating-point formats | | Compiler | Places and schedules operations for the accelerator | | Inference Engine | Coordinates execution across multiple cards | | Host Runtime | Manages interaction between server CPUs and cards | | Chip Runtime | Launches device work and manages on-chip execution | | Operational tools | Monitor cards and analyze workload performance | This is the portability boundary. A framework checkpoint may be shared with another accelerator, but the compiled artifact, quantization choices, partitioning plan, and performance profile are hardware-specific. The numerical choice matters. Corsair supports microscaling block floating-point formats, including MXINT8 and MXINT4 modes. Lower precision reduces storage and movement, but every model still needs accuracy evaluation. Peak low-precision throughput is not proof that a particular model preserves quality or meets a production latency target. ## Standard racks reduce one kind of risk d-Matrix does not require a proprietary wafer-scale chassis or a new liquid-cooled megawatt rack. Corsair is a PCIe card, and the company positions air-cooled server and rack designs that use Ethernet for scale-out. That can reduce facility integration work: - standard server mechanics; - familiar PCIe enumeration; - Ethernet-based rack networking; - incremental deployment beside existing accelerator pools; - independent scaling of specialist and GPU capacity. It does not remove system integration. A 64-card model partition still needs validated servers, network topology, firmware, cooling, power, observability, and replacement procedures. The operator also needs a serving layer that knows which model artifact can run on which pool. The deployment pattern is therefore less “one new universal accelerator” and more “one specialized pool behind a stable inference contract.” ## Heterogeneous inference is the stronger first use The most useful evidence for Corsair is not a vendor comparison claiming that one rack replaces a GPU fleet. It is a narrower experiment that gives the specialist one phase of the request. A [partner engineering study from Gimlet Labs](https://gimletlabs.ai/blog/low-latency-spec-decode-corsair) evaluated Corsair for speculative decoding while leaving prefill and verification on GPUs. The target model was `gpt-oss-120b`; a 1.6-billion-parameter draft model ran on two Corsair cards. Gimlet reported a 2–10× improvement in interactivity over its GPU-only speculative-decoding configuration at matched energy-efficiency points. The boundary matters: - the analysis combined measured and modeled data; - it modeled an 8K-input, 1K-output coding workload; - token-acceptance assumptions materially affected the result; - the exact comparison GPU was not disclosed; - the test evaluated one inference phase, not complete replacement of the target-model hardware. That caveat does not weaken the result. It makes the result useful. A memory-bound draft phase fits SRAM-centric hardware differently from compute-heavy prefill and batched verification. The experiment supports a heterogeneous architecture where each phase runs on the hardware that matches its bottleneck. ```mermaid flowchart LR REQ["Request"] PREFILL["Prefill
GPU pool"] DRAFT["Draft tokens
Corsair pool"] VERIFY["Verify tokens
GPU pool"] ACCEPT{"Accepted?"} OUTPUT["Stream output"] REQ --> PREFILL PREFILL --> DRAFT DRAFT --> VERIFY VERIFY --> ACCEPT ACCEPT -->|yes| OUTPUT ACCEPT -->|continue| DRAFT ``` This is disaggregated inference in concrete form. It adds network hops and orchestration, so the specialist must save more time than the handoff consumes. ## The evidence is still uneven d-Matrix publishes large claims for Corsair, including tens of thousands of tokens per second and substantial improvements in latency, power efficiency, and total cost. Its [launch announcement](https://www.d-matrix.ai/announcements/d-matrix-unveils-corsair-the-worlds-most-efficient-ai-computing-platform-for-inference-in-datacenters/) labels the performance and cost estimates preliminary and subject to change. The product page also mixes several boundaries: - card-level peak arithmetic; - server and rack aggregate memory; - model-specific projected performance; - preliminary comparisons against H100; - capacity-mode and performance-mode configurations. Those numbers are not interchangeable. A useful independent comparison needs the same model, precision, quality threshold, prompt and output lengths, batching policy, service-level objective, power boundary, and complete system cost. No Corsair result appeared in the [MLCommons Inference results available during this August 30, 2026 review](https://mlcommons.org/benchmarks/inference-datacenter/). This is a bounded research finding, not proof that a submission cannot exist elsewhere. Public partner and integration announcements establish ecosystem activity, but they do not provide the same evidence as measured production utilization, availability, or cost per accepted token. The roadmap adds another evidence boundary. d-Matrix says its [Pavehawk 3DIMC test silicon](https://www.d-matrix.ai/scaling-ai-inference-with-3dimc/) is operating in the lab and that a future Raptor architecture will use stacked memory. Those are development milestones and targets, not shipping-product measurements. ## What d-Matrix changes in the landscape d-Matrix is not trying to recreate the entire NVIDIA stack. It attacks one expensive path: inference workloads where moving weights dominates useful arithmetic. The physical interface remains PCIe and Ethernet. The differentiated layer is the memory-compute complex, followed by the compiler and runtime required to use it. That makes d-Matrix easier to combine with other accelerators than a closed, facility-scale replacement architecture. It also means the surrounding control plane becomes critical. The model registry needs multiple compiled artifacts. The router needs to understand workload phases. Observability needs to compare latency, throughput, quality, energy, and handoff overhead across dissimilar pools. The strategic question is not whether every token should run on in-memory compute. It is whether enough stable, memory-bound work can be isolated to keep the specialist busy without making the serving system harder to operate than the savings justify. If heterogeneous inference becomes the normal design, the unresolved boundary moves upward: who owns the optimizer that decides when a request should cross from a GPU into a specialist—and what evidence proves that decision still wins after network, compilation, capacity, and operational costs are included? --- _Part of the [AI Compute Landscape](https://artificialcuriositylabs.ai/series/ai-compute/) — an ongoing exploration of accelerator architectures, software stacks, and data-center systems._