There Is No Universal AI Runtime—So Build a Portable Control Plane

AI accelerators are not interchangeable computers. A CUDA kernel does not become a Trainium executable because both systems run PyTorch, and an inference server supporting two GPU families does not erase their different compilers, libraries, memory models, collectives, or performance envelopes.

The portable layer sits higher.

Applications can call one stable service contract. A model registry can govern several hardware-specific artifacts. A router can place each request on a qualified pool. Observability can compare the resulting service, and policy can decide when moving a workload is economically justified.

That is a portable control plane—not a universal runtime.

TL;DR

• Kernels, compiled artifacts, collectives, quantization paths, and performance tuning remain tied to an accelerator stack.
• Serving frameworks can present common APIs across several backends, but backend support does not guarantee identical model coverage, behavior, or economics.
• The durable architecture keeps NVIDIA, AMD, TPU, Trainium, and specialist accelerators in separate pools behind one service contract.
• Model governance, workload qualification, routing policy, service-level telemetry, and cost policy are more portable than execution.
• Portability is an operating model: maintain a broad compatibility pool and move stable workloads only after another pool passes correctness, performance, and cost gates.

The runtime boundary is real

CUDA defines a GPU programming model, runtime, compiler toolchain, and libraries for NVIDIA hardware. CUDA C++ device code is compiled by nvcc, which can produce architecture-specific binary images and PTX for later compilation by the CUDA runtime. That mechanism provides compatibility within the NVIDIA platform; it does not produce an executable for an unrelated accelerator.

HIP deliberately resembles CUDA and includes tools that translate many CUDA API calls. The same documentation also states that HIP code must be compiled for a specific AMD GPU architecture and that the resulting binaries contain code for that target. Source translation reduces migration work. It does not turn a CUDA binary into an AMD binary.

The boundary becomes clearer with cloud accelerators. The Neuron compiler transforms framework graphs into a Neuron Executable File Format artifact that the Neuron runtime loads on Neuron devices. PyTorch/XLA lets applications retain familiar PyTorch APIs, but it traces operations into an intermediate graph and sends HLO to XLA for compilation. Changes in graph shape can cause additional compilations, and hardware-specific optimization still happens below the framework.

OpenXLA is a useful portability layer precisely because it separates a versioned operation set from target-specific backends. StableHLO can carry a model graph between framework and compiler, while each backend still performs its own optimization and code generation. The graph can travel farther than the executable.

This produces a simple hierarchy:

The mistake is not using abstraction. The mistake is placing the abstraction below the level where the evidence supports it.

One contract, separate hardware pools

The stable unit should be a service, not a device. Each accelerator family operates as a separately qualified pool with its own software image, driver, runtime, serving engine, model artifacts, deployment process, and failure domain.

This design does not schedule one container indiscriminately across every device. It selects a pool that already has a tested artifact and a known operating envelope for the requested model, precision, context length, batch profile, and service objective.

The model catalog therefore needs more than a model name. A useful entry binds a logical model version to a set of qualified implementations:

logical model: reasoning-model-v3
evaluation policy: eval-suite-2026-08

qualified implementations:
  - pool: nvidia-hopper
    artifact: fp8-tensor-parallel-8
    status: production
  - pool: amd-mi300
    artifact: fp8-tensor-parallel-8-rocm
    status: canary
  - pool: trainium
    artifact: neuron-compiled-seq4096
    status: batch-only

The logical model gives applications a stable name. Each implementation retains its own artifact identity, software bill of materials, evaluation results, supported limits, and rollback history. Promotion happens per implementation, not once for the abstract model.

Serving frameworks widen the interface, not the binary

Modern serving projects make this architecture easier, but none removes the qualification work.

vLLM supports hardware plugins outside its main repository. SGLang documents support for multiple accelerator families and offers an API compatible with common language-model request patterns. These are valuable common surfaces. Their hardware paths can still differ in kernel availability, quantization support, distributed execution, model coverage, and release maturity.

Triton Inference Server makes the separation visible: every model is associated with a backend, and the backend is the implementation that executes it. Triton can wrap TensorRT, PyTorch, ONNX Runtime, vLLM, or custom logic, but its own documentation warns that not every backend is supported on every platform.

ONNX Runtime follows a similar pattern through Execution Providers. The application uses a consistent API while a provider claims supported nodes or subgraphs and hands them to a hardware-specific library. That is portable application integration with specialized execution underneath. It is not proof that every model will partition, compile, or perform equally on every provider.

KServe can standardize how a model service is declared and operated on Kubernetes. Its serving runtimes still select concrete backends, images, resource requests, and model formats. The operational object becomes portable; the implementation remains qualified for a particular pool.

The practical contract should cover:

• logical model and version;
• task type and request schema;
• streaming and cancellation behavior;
• context, input, and output limits;
• safety and data-handling policy;
• latency and availability objectives;
• error categories and retry rules;
• response metadata identifying the implementation used.

It should not promise identical tokenization, numerical output, latency, or cost unless those properties were explicitly tested.

The scheduler places workloads; it does not port them

Cluster schedulers already understand heterogeneous resources. Kubernetes device plugins let vendors advertise specialized devices to the kubelet. A cluster can expose resources such as nvidia.com/gpu and amd.com/gpu, then use labels and affinity to place a workload on the intended node type.

Slurm GRES similarly models GPU types as generic resources and lets a job request a specific type and count. Ray defines accelerator resources for NVIDIA GPUs, AMD GPUs, Neuron cores, TPUs, and other devices, then assigns tasks or actors to nodes with the requested resource.

These systems solve allocation. They do not determine whether a model is correct on that accelerator, whether the serving engine has the required kernel, or whether the workload meets its latency target. A portable control plane must make placement eligibility a product of qualification:

eligible pool =
  supported model
  + approved artifact
  + compatible request limits
  + passing evaluation
  + healthy capacity
  + acceptable economics

Only after those gates pass should the router compare eligible pools.

Route on workload shape and evidence

A lowest-cost accelerator rule is too crude. Inference cost changes with context length, prefill-to-decode ratio, output length, concurrency, batch formation, cache reuse, quantization, and the amount of idle capacity held to protect latency.

The routing decision needs a qualification matrix:

Routing also occurs at more than one level. Ray Serve’s routing documentation distinguishes model-level ingress routing from replica selection. A cross-hardware control plane needs both: first select the qualified implementation and pool, then let the local serving system select a replica using queue, cache, and locality signals.

The broad compatibility pool plays an important role. New models, unusual operators, rapidly changing context requirements, and workloads without enough evaluation evidence stay there. Specialist pools earn traffic when the workload becomes stable enough to compile, measure, and govern.

Normalize outcomes, preserve native telemetry

Observability is portable only at the service layer. Time to first token, inter-token latency, total response time, requests per second, tokens per second, error rate, evaluation score, queue delay, and cost per accepted output can be defined consistently.

The lower-level evidence remains native. GPU occupancy, NeuronCore utilization, compiler-cache behavior, HBM counters, collective timing, and device power are not one universal metric family. Flattening them into a single “accelerator utilization” number destroys the details needed to diagnose a pool.

The clean design keeps two views:

1. Normalized service telemetry compares the outcome delivered to the application.
2. Backend-native telemetry explains why one implementation produced that outcome.

Economic policy should use both. A pool that has a lower device-hour price but compiles frequently, rejects more requests, requires excess standby capacity, or produces lower-quality outputs can have the higher service cost.

Compare service cost, not accelerator price

A portable control plane needs one economic denominator across every pool:

service cost per accepted unit =
  reserved accelerator and host capacity
  + network, storage, and energy
  + amortized model-porting and software work
  + operator and support cost
  + failed, retried, and quality-rejected work
  ------------------------------------------------
  accepted outputs that met the service objective

The denominator can be accepted tokens, images, training steps, or completed jobs, but it must include the quality and latency objective. Utilization belongs inside the calculation because reserved capacity accrues cost while idle. Availability belongs inside it because failed work and standby capacity reduce the accepted output produced by the same reservation.

This illustrative monthly comparison shows why hardware price alone is misleading:

The numbers are hypothetical; the mechanism is not. The specialist has lower compute and energy cost but loses the comparison because it carries porting work, idle capacity, and failed output across a smaller qualified workload. If model coverage grows, utilization rises, and software work is amortized across more accepted output, the same pool can later become the cheaper path.

The control plane should therefore store both the current service cost and the evidence behind it: measurement interval, reservation model, utilization, failure rate, quality threshold, energy boundary, labor allocation, and software version. Routing on a number without that provenance recreates the same false portability problem at the economic layer.

What is still missing

The software projects above provide pieces of this architecture, not one finished control plane. Schedulers allocate devices. Serving engines execute models. Model registries track artifacts. Gateways route requests. Telemetry systems collect measurements.

The missing layer is a durable qualification record connecting all of them:

• which logical model maps to which hardware-specific artifact;
• which evaluation approved that artifact;
• which request shapes it supports;
• which software and driver versions produced the result;
• which service and economic thresholds it currently meets;
• which policy moved traffic and how to reverse that decision.

Without that record, “multi-accelerator” becomes a collection of independent deployments sharing an endpoint. With it, hardware diversity becomes governable.

So what

Do not wait for a universal AI runtime. The compilers, kernels, collectives, and memory systems are moving in the opposite direction—toward deeper specialization.

Build portability where it holds: the application contract, model identity, evaluation policy, workload router, service telemetry, and economic decision. Keep execution pools separate, qualify every implementation, and route only across proven choices.

The open thread is how much control-plane standardization is possible before the policy itself becomes coupled to the dominant serving engine. If every backend exposes different cache state, queue semantics, compilation behavior, and power evidence, the router may remain portable in name while its best decisions depend on vendor-specific signals.

Part of the AI Compute Landscape — an ongoing exploration of accelerator architectures, software stacks, and data-center systems.