TL;DR
- A 186-request matrix produced 84.4% normalized input savings with one write and 20 reads, but changing the cache key every call cost 25% more than uncached input.
- Reordering tools or changing a description or schema forced a full rewrite; changing tool results after a stable breakpoint preserved the cached prefix.
- TTL controls how long reusable work survives. The harness controls whether two requests contain reusable work at all.
- Treat rendered-prefix stability, cache reads, cache writes, and post-compaction recovery as production metrics, not provider defaults.
A cache flag can be enabled while an agent pays to rebuild the cache on every call.
That is the central mistake in how prompt caching gets discussed. The API exposes the capability. The harness determines whether the capability produces a hit, a miss, or a stream of expensive writes.
I tested that distinction across 186 successful requests in 40 staged agent sessions, then ran a separate 30-track retention suite across OpenAI and Claude models on Amazon Bedrock. The result was consistent: prompt caching is not primarily a TTL decision. It is a request-construction discipline.
The provider owns the cache. The harness owns the prefix.
The same API produced opposite economics
The test matrix covered implicit and explicit caching, append-only conversations, prompt sizes from below the cache threshold to 240,000 tokens, tool mutations, cache-key changes, concurrent calls, and delayed retention probes.
The strongest cost result came from the simplest pattern: write one stable prefix, then reuse it.
| Request pattern | Normalized input savings |
|---|---|
| One cache write, one cache read | 32.5% |
| One cache write, three cache reads | 61.2% |
| One cache write, five concurrent reads | 70.8% |
| One cache write, twenty concurrent reads | 84.4% |
| New cache key on every request | 25% more expensive |
The calculation uses the current GPT-5.6 pricing shape documented by OpenAI and Amazon Bedrock: writes cost 1.25 times normal input, while reads receive a 90% discount.
The feature did not create the savings. Reuse did. With no reuse, the write premium made caching worse than ordinary input.
That distinction becomes more important inside an agent loop. A coding harness rebuilds a request from system instructions, repository guidance, tool definitions, conversation history, tool calls, tool results, environment state, and the newest user turn. Every one of those components can preserve or destroy the common prefix.
Tool catalogs are part of the cache key
The tool tests made the harness boundary visible.
An unchanged tool catalog produced an 8,556-token cache read. Reordering the same tools caused a new write. Changing one tool description caused a new write. Changing one JSON schema caused a new write.
The tools remained semantically similar. The rendered prefix did not.
This matches the provider contracts. OpenAI requires tools and images to remain identical for an exact prefix match. Amazon Bedrock processes cacheable content in tools → system → messages order; changing tools invalidates the system and message caches that follow. Anthropic documents the same hierarchy.
The practical consequence is larger than “sort your JSON.” The tool registry is now part of the cost architecture.
A harness that discovers tools dynamically, changes descriptions at runtime, emits schemas in nondeterministic order, or injects session-specific metadata before its stable instructions can have caching enabled and still miss continuously. Tool discovery, serialization, and prompt construction are no longer separate implementation details. They meet in the rendered prefix.
Dynamic work belongs after the boundary
Dynamic tool results behaved differently in the same matrix.
I kept the instructions and tool catalog fixed, placed an explicit breakpoint after them, and changed only the tool result that followed. Both requests retained the same 8,556-token cache hit with no new write.
That is the architecture:
[stable tools]
[stable system instructions]
[stable repository or task context]
[cache breakpoint]
[conversation turns]
[tool calls and results]
[latest user input]The exact fields differ by provider. The invariant does not: stable content first, volatile content later.
OpenAI’s Prompt Caching 201 guide describes the same pattern in Codex: system instructions, tool definitions, sandbox configuration, and environment context stay consistently ordered, while new messages append to the conversation. The guide also reports one coding workload improving its hit rate from 60% to 87% after adding a stable prompt_cache_key.
The cache key matters because reuse has two conditions: the prefix must match, and requests must reach compatible cache capacity. A key that changes every call fragments reuse. A key shared across too much traffic can overflow one routing bucket. The harness owns that partitioning decision.
TTL is retention, not reuse
The original version of my prompt-caching explainer overemphasized the five-minute TTL. The live tests showed why that is the wrong organizing principle.
Provider and model contracts differ:
| Tested model family | Requested retention | Observed result |
|---|---|---|
| GPT-5.6 Luna and Terra | 30-minute minimum | Hit at 10 minutes; rewrite at 31 minutes |
| Claude Sonnet 5 and Opus 4.8 | 5 minutes | Rewrite at 6 minutes |
| Claude Sonnet 5 and Opus 4.8 | 1 hour | Hit at 31 minutes; rewrite at 61 minutes |
Each delayed probe used an independent cache entry, so an earlier hit could not refresh the entry tested later. For GPT-5.6, 30m is the only supported value and sets a minimum lifetime, not an exact expiration. OpenAI may retain an entry longer. These observations establish the cache state at each probe time, not the exact millisecond of eviction.
TTL answers, “How long can matching work remain reusable?” Prefix identity answers, “Is this still the same work?” The second question belongs to the harness.
The six harness responsibilities
Prompt-cache performance emerges from six parts of the agent harness.
| Harness component | Cache responsibility |
|---|---|
| Instruction manager | Keep durable instructions byte-stable; move timestamps and request IDs out of the prefix |
| Tool registry | Use deterministic tool order, descriptions, and schemas |
| Context builder | Put shared repository context before session-specific state |
| Conversation manager | Append turns instead of rewriting earlier history |
| Compactor | Treat compaction as a planned cache reset, then stabilize the new summary |
| Router and telemetry | Use stable cache keys and measure reads, writes, misses, cost, and time to first token |
Compaction exposes the hardest tradeoff. Removing or summarizing old turns reduces context size, but rewriting history also breaks the existing prefix. OpenAI’s caching guide calls this tension out directly: context engineering rewards selective change, while prompt caching rewards stability.
The right response is not to avoid compaction. It is to make the reset observable. Record the prompt and tool bundle fingerprint, expect one cold write after compaction, then verify that later turns read the new prefix. A cache miss caused by an intentional state transition is healthy. A cache miss caused by randomized tool order is a regression.
The research reaches the same conclusion
The strongest external confirmation comes from a study of more than 500 long-horizon agent sessions across OpenAI, Anthropic, and Google models.
The study measured 45–80% API cost reductions and 13–31% time-to-first-token improvements. Its more important result was architectural: system-prompt-only caching and strategies that excluded dynamic tool results produced more consistent benefits than naive full-context caching. In one condition, full-context caching made latency worse because the system paid write overhead for dynamic content that was not reused.
That is the same failure mode the cost table exposes. A cache write is an investment. The harness needs to decide whether the content before a breakpoint has a realistic path to later reads.
Anthropic describes the production consequence even more directly in its Claude Code prompt-caching engineering notes: the team builds the harness around prompt caching, alerts on hit-rate regressions, preserves the main conversation prefix during compaction, and avoids changing tool definitions mid-session. That is not an API configuration checklist. It is an agent architecture.
Latency also needs separate treatment. In my matrix, cache behavior was functionally reliable, but short-prompt latency did not improve consistently. One 128,000-token Terra request improved from 3.935 seconds cold to 1.868 seconds warm; the comparable Luna calls were effectively unchanged. Cost and token accounting were stronger signals than a handful of end-to-end timing samples.
What’s missing: cache observability at the harness layer
Provider metrics report cached tokens and cache writes. They do not explain which harness change fragmented the prefix.
A production harness needs four additional signals:
- A fingerprint for the rendered stable prefix.
- A separate fingerprint for the ordered tool bundle.
- Cache-read and cache-write tokens by prompt version.
- An event marking intentional resets such as compaction, tool-version changes, or model switches.
Without those signals, a declining hit rate appears as a larger bill. With them, it becomes a deploy regression tied to a specific prompt or tool bundle.
Anthropic has started exposing cache diagnostics that identify whether two requests diverged in tools, system instructions, or message history. The broader opportunity is provider-neutral diagnostics inside the harness, before requests leave the application.
So what
Enabling prompt caching is the first line of implementation, not the end of the work.
The durable pattern is simple: serialize stable instructions and tools deterministically, place dynamic state after a deliberate boundary, append conversation turns, partition cache keys around real reuse, and make writes as visible as reads.
The unresolved part is dynamic tool discovery. Loading every tool preserves one stable catalog but spends context and can reduce tool-selection quality. Loading tools on demand improves relevance but changes the prefix. Deferred tool search and request-level tool restrictions offer a middle path, but the right balance still needs workload-level measurement.
That is the next cache problem for coding agents: not whether the API supports caching, but whether a changing tool surface can remain both selective and reusable.