--- title: "Harness vs. Runtime: When to Graduate" description: "Harness gives you a working agent in 20 seconds with 25 lines of configuration. Runtime takes 398 lines of code. Both run on the same compute. The graduation trigger that matters is not hooks or custom loops — it's per-user outbound identity." canonical_url: "https://artificialcuriositylabs.ai/posts/harness-vs-runtime/" md_url: "https://artificialcuriositylabs.ai/posts/harness-vs-runtime.md" published_at: "2026-07-13T07:00:00.000Z" tags: - "agents" - "agentcore" - "bedrock" - "harness" --- ## The short version I ran the same support-triage task on both AgentCore run modes. Harness stood up a working agent in 20.4 seconds against 398 lines of Runtime Python, and it passed the IAM-gated and no-tool scenarios cleanly. It could not enforce per-user Cedar policies on the refund tool. Both modes end up on the same compute. CloudTrail records harness operations under `AWS::BedrockAgentCore::Runtime`. The difference is who owns orchestration — you write the code (Runtime) or AgentCore handles it (Harness). The documented graduation triggers are hooks, custom loops, and bidirectional streaming. The barrier I hit first was none of those — it was per-user outbound identity. What follows is what worked, what broke, and where the hard boundary is. --- ## What Harness gives you without writing agent code For nearly every AgentCore capability, Harness delivers it in configuration: Memory (all four strategies), Gateway tool wiring, Browser and Code Interpreter as built-in tools, outbound Identity, automatic Observability, model switching mid-session, versioning, and VPC networking. All available without customer code. Runtime offers the same capabilities — with "you write the code required" for each. The session manager, the memory client, the tool orchestration, the span instrumentation — all available, all require code. The three things the documented feature grid says Harness cannot do: ``` ✗ Hooks (AfterToolCallEvent, BeforeToolCallEvent) ✗ Non-agent-loop patterns (graph, workflow, tree-of-thought) ✗ Bidirectional streaming ``` Those are the published graduation triggers. A fourth one — per-user outbound identity — isn't documented; I found it by running the config. --- ## H1: Two API calls, no entrypoint code The Harness create-and-invoke path is exactly as small as advertised. The entire orchestration for a triage agent — model, system prompt, one Gateway tool, memory — is about 25 lines passed to `create_harness()`. The equivalent Runtime agent is **398 lines**. Harness is roughly 6% of the Runtime file. Two measurements stand out: - **Time to READY: 20.4 seconds.** From `create_harness()` to a harness that answers, a little over 20 seconds. The equivalent Runtime path builds a container — around 180 seconds before the first invoke. Harness is ~9× faster to first response because there's no image to build. - **The invoke is not SSE.** `boto3`'s `invoke_harness` returns a proper event stream. A JWT-protected harness requires raw HTTPS `POST` to `/harnesses/invoke?harnessArn=...` and parsing an `application/vnd.amazon.eventstream` body with `botocore.eventstream.EventStreamBuffer`. Two scenarios passed cleanly: - **The no-tool password reset:** correct steps, clean end of turn, no tool call. Passed. - **The web-search region question:** the harness called the WebSearch gateway tool twice and cited live results. Passed. Both authorize outbound with the harness's own IAM role (SigV4). There is no per-user identity involved. This is Harness at its best: a governed tool, wired by ARN, no code. Then I pointed it at the Cedar-gated refund tool. It stopped there. --- ## H2: The wall is identity, not hooks The support agent has a refund tool behind a `CUSTOM_JWT` gateway with a Cedar policy engine in enforce mode. The policies are per-user: ``` permit alice process_refund when amount < 500 // standard tier forbid bob process_refund // blocked, any amount permit carol process_refund when amount < 2000 // enterprise tier ``` Cedar evaluates on the **principal** — explicitly the invoking user's identity from the JWT `sub` claim. The Runtime path is trivial: the agent forwards the caller's own inbound JWT straight through to the gateway. alice's refund is permitted, bob's is forbidden, on the first call, for any user, with no extra machinery. A Harness has no code, so it can't forward anything. It authorizes outbound calls through its configured `outboundAuth`, which offers exactly three modes: `awsIam`, `none`, and `oauth`. I tried the two that could carry identity. **`awsIam` outbound.** The harness signs the gateway call with its own execution role. Cedar sees the role, not a user — there is no `OAuthUser` principal, so no per-user policy matches. The harness simply is not alice. Correct behavior, wrong outcome. **`oauth` outbound.** This is the intended path: the harness mints the invoking user's token from AgentCore's Identity token vault. It's also where the real work was. Making it go took five distinct fixes, each a genuine requirement: 1. **The response is a binary event stream**, not SSE — parse with `EventStreamBuffer`, and headers are plain strings. 2. **The execution role needs vault access** — `bedrock-agentcore:GetResourceOauth2Token` plus `secretsmanager:GetSecretValue`. Without it the tool fails with `AccessDeniedException`. 3. **The grant type defaults to the wrong flow.** `oauth` outbound defaults to `CLIENT_CREDENTIALS` (machine-to-machine). The provider is user-federation (3LO) — it's either M2M *or* 3LO, never both. The request fails with `Error parsing ClientCredentials response`. Fix: set `grantType=AUTHORIZATION_CODE`. 4. **Authorization-code outbound needs a return URL** — `defaultReturnUrl`, registered on the workload identity. 5. **The token is vaulted under the wrong identity.** This is the blocker. The vault is keyed on `(workload_identity, user_id)`. I had pre-consented users and stored their tokens under a standalone workload identity. But a Harness calls the vault under **its own** workload identity — minted when the harness is created. It looks up alice under *that* identity, finds nothing, and falls back to starting a fresh browser-consent flow — which a headless server invocation cannot complete. Every case failed with `You must provide a ResourceOauth2ReturnUrl to proceed with this flow`. --- ## Why re-vaulting doesn't fix it You could fix issue 5 mechanically: create the harness, read back its generated workload identity, register return URLs on it, run the 3LO consent loop keyed to *that* identity, then invoke. It would work. It also carries three operational costs: - The harness's workload identity **doesn't exist until the harness does**, so you can't pre-consent users ahead of deploying. - It **changes on every rebuild** — a new suffix each time — so re-consenting all users becomes part of your deploy. - 3LO requires **per-user browser consent** the first time regardless, which is inherent to the flow. The one clean escape would be token passthrough — let the harness forward the inbound user JWT unchanged, the way Runtime code does. The gateway target model has a passthrough concept, but it is not exposed on the harness `outboundAuth` surface. Token exchange (OBO, RFC 8693) is the other theoretical route, and it's dead because Cognito doesn't implement it. **The honest finding: per-user Cedar through a Harness is architecturally possible and operationally impractical.** Not because policy enforcement fails — Cedar at the gateway is identical for both paths — but because the code-less path binds outbound identity to an ephemeral, per-deploy workload identity and a consent step it can't drive headless. --- ## The decision rule Use Harness until you hit one of these. The first three are documented; the fourth is the one this run added. **Trigger 1 — Hooks.** You need to intercept what the model sees after a tool call. The `AfterToolCallEvent` pattern — sanitizing an authorization failure, normalizing output, grounding the model on a deterministic fact — is Runtime-only. If your tool returns something the model must *interpret* rather than a clean fact, you need the hook. **Trigger 2 — Custom loop logic.** Tree-of-thought, graph routing, supervisor/worker, A2A handoffs — any pattern where the orchestration loop itself carries business logic. Harness runs one managed loop; you can't fork it. **Trigger 3 — Model-level controls.** Prompt-cache keys, extended-thinking budgets, per-token routing between a cheap model and an expensive one. Harness `bedrockModelConfig` exposes `modelId`, `temperature`, `maxTokens`, `topP`. Anything finer belongs in code. **Trigger 4 — Per-user outbound identity.** If a tool enforces authorization on the end user's identity — per-user Cedar, an OAuth API scoped to the user, anything that needs the caller's own token downstream — the Harness path forces you through vault tokens keyed to an ephemeral workload identity plus a consent flow it can't run headless. Runtime forwards the caller's JWT in one line. This is the trigger the feature grid doesn't name, and for identity-aware agents it's the one you hit first. --- ## What this means for the support agent The support agent needs Runtime for two independent reasons, each tied to specific evidence. The first is **Trigger 1**. The `AfterToolCallEvent` hook intercepts the policy engine's raw output and replaces it with a clean `REFUND APPROVED / BLOCKED:` fact before the model writes the customer reply. Without it, the model infers from raw authorization data — sometimes correct, sometimes fabrication. Cedar at the gateway works in Harness; the deterministic post-processing of Cedar's decision does not. The second is **Trigger 4**. The refund tool is per-user Cedar. Runtime forwards each caller's JWT straight to the gateway, so the standard-tier user is permitted and the forbidden user is blocked on the first call with no vault, no pre-consent, no workload-identity juggling. The Harness equivalent is the five-layer, per-deploy, consent-bound path above. Both are boundaries, not defects. Harness handled the IAM-gated web search and the no-tool tickets with 25 lines of config and a 20-second cold start — less work than the Runtime equivalent. The identity-bound, hook-dependent refund path is what needs code. That is the graduation the two-model system is designed for: start on Harness, cross to Runtime when a trigger fires, and pay the code cost only for the parts that require it. --- ## Results summary | Metric | Harness | Runtime | |---|---|---| | Config lines to working agent | ~25 lines | 398 lines | | Time from create to first invoke | 20.4 s | ~180 s (container build) | | Password reset (no tool) | pass | pass | | EU region web search (IAM gateway) | pass | pass | | Per-user Cedar refund (JWT gateway) | blocked — outbound identity | pass (JWT forwarded) | | Memory wiring | one config block | ~40 lines of code | | Observability | automatic | ADOT + instrumentation | The pattern is consistent: for governed tools that authorize on the agent's own identity (IAM) or on nothing, Harness is less work and faster. For tools that authorize on the *user's* identity, or that need deterministic post-processing of the tool result, Runtime is not optional. --- ## So what Start on Harness. For governed tools that authorize on the agent's identity or on nothing, 25 lines of config and a 20-second cold start beat the Runtime equivalent. Watch for the identity boundary. The first time a tool needs to act as the user rather than as the agent, move to Runtime. That is where the config-only model ends and code begins. I still haven't fully worked through what happens with token passthrough if it *were* exposed on the Harness `outboundAuth` surface — whether the per-user vault keying would become irrelevant if the harness could forward the inbound token verbatim. That's the operational escape hatch nobody's tried yet.