TL;DR
- A model asked to rank 500 models by intelligence-per-dollar returns a ranking that looks right and was never computed. The fix is not a better prompt. It is running the code.
- AgentCore Code Interpreter is a session-scoped managed sandbox:
StartCodeInterpreterSession→InvokeCodeInterpreter(writeFiles/executeCode/readFiles) →StopCodeInterpreterSession. Python 3.12 with pandas and numpy already in the image. - I pushed a 528-model dataset in, computed the intelligence-price Pareto frontier in the sandbox, and checked it against a known-good result. Exact match. Then I corrupted the input and the frontier changed — proof the code ran on the data rather than echoing an answer.
- Standalone, the tool emits CloudWatch metrics but no trace span. Run inside AgentCore Runtime, the same call nests into one end-to-end trace with the agent’s reasoning and the memory write. Metrics are a property of the tool; the distributed trace is a property of Runtime.
- The scalable shape is not “compute on every question.” It is ingest-on-a-schedule, serve-precomputed-results. The sandbox belongs in the scheduled step.
The problem: a fluent code writer with no runtime
Agents generate Python fluently. By default they cannot run it.
The failure mode is quiet. Ask an agent to rank 500 models by intelligence-per-dollar and it produces a table that looks exactly right, cited with confidence, and was never computed. The model approximated the arithmetic — mentally sorted, estimated, pattern-matched against training data — and handed you a plausible result. For a single well-known number that is often fine. For a Pareto frontier over a 500-row dataset with two continuous dimensions, it is not. The ranking changes every week as new models land and prices shift. The model’s training data is stale. The output looks authoritative and is wrong.
This is not a prompt problem. You cannot instruct your way to a correctly-computed frontier. A language model that cannot execute code can only narrate computation. Narrated computation is not computation.
The task I used to test this: the intelligence-price Pareto frontier of the LLM landscape. Pareto-efficient means no cheaper model has a higher intelligence score. Over 528 models with 351 priced, the frontier is 13 models — a small set, and one the model cannot derive by reading a table, because it requires scanning all pairs and eliminating dominated ones. The right answer is a computed property of the data, not a recalled one.
What AgentCore Code Interpreter is
A session-scoped managed sandbox. No resource to create — there is a built-in identifier aws.codeinterpreter.v1. The data plane is three calls:
StartCodeInterpreterSession → sessionId (microVM starts)
invoke_code_interpreter(...) → results (write files, execute code, read files)
StopCodeInterpreterSession → done (microVM tears down, no standing cost)The managed image is Python 3.12.13 with pandas 2.3.1 and numpy 1.26.4 pre-installed. Standard data work needs no pip install. The sandbox has a real isolated filesystem — writeFiles lands a file at a path, listFiles shows it, readFiles returns it, and a file written in session A is gone when session B starts.
The IAM surface is minimal: bedrock-agentcore:{Start,Invoke,Stop}CodeInterpreterSession on arn:aws:bedrock-agentcore:<region>:<account>:code-interpreter/*.
The run: computing a frontier the honest way
The data source matters here and is itself a “judge the layer” moment. Artificial Analysis publishes benchmark intelligence and pricing for LLMs. Their Terms of Service explicitly forbid scraping. They also publish a free Data API. So this task, which looked like a Browser job, was actually an API job — the right layer was the sanctioned endpoint, not a scraper. I stored the dataset in S3 and read it into the sandbox at compute time.
The compute logic is the same pareto_efficient() function that runs in the model-analysis repo:
import pandas as pd, numpy as np, json
df = pd.read_csv("models.csv")
price = df["price"].to_numpy(dtype=float)
idx = df["intelligence"].to_numpy(dtype=float)
def pareto_efficient(price, index):
n = len(price)
finite = np.isfinite(price) & np.isfinite(index) & (price > 0)
best = -np.inf
on = np.zeros(n, dtype=bool)
for p in np.sort(np.unique(price[finite])):
g = finite & (price == p)
gb = np.max(index[g])
if gb > best:
on[g & (index == gb)] = True
best = gb
return on
df["on_frontier"] = pareto_efficient(price, idx)
front = df[df["on_frontier"]].sort_values("price")
print(json.dumps({
"frontier_size": len(front),
"cheapest": front.iloc[0]["name"],
"cheapest_price": float(front.iloc[0]["price"]),
"smartest": front.sort_values("intelligence").iloc[-1]["name"],
}))This runs inside the sandbox via executeCode. The result came back in 0.95 seconds of in-sandbox execution time (1.1 seconds round-trip on a cold session). The frontier: 13 models, Qwen3.5 0.8B at $0.02/1M tokens on the cheap end, Claude Opus 4.8 on the smart end.
That matched the ground-truth frontier in the model-analysis repo exactly — same 13 models, same set, no differences.
Proving it computed
The worry with any sandbox is whether the code actually ran or whether the response is a canned answer. The test is simple: corrupt the input and see if the output changes.
I re-ran with every price forced equal. If the code runs, the frontier must collapse to one model — only the single highest-intelligence point survives when price can no longer discriminate. It did: 13 models → 1 model. The code ran on the data I pushed. It did not echo anything.
This is the difference between a computed result and a simulated one, made observable in two lines.
Where the computation shows up: metrics vs. trace
Standalone — a direct SDK call with no Runtime involved — the Code Interpreter call emits CloudWatch metrics but no trace span. Namespace AWS/Bedrock-AgentCore, dimensions Operation + Resource=aws.codeinterpreter.v1 + ToolName. The metrics are there: Invocations, Latency, Duration, Errors. The executeCode average latency was 559 ms, max 1,145 ms across the session. But there is no aws/spans entry. No CloudWatch Logs group for the sandbox. The sandbox’s stdout returns only in the API response.
Run inside AgentCore Runtime, the same calls nest into a single span tree rooted at the Runtime invocation:
| Span | Duration (ms) |
|---|---|
| POST /invocations (Runtime root) | 3,170 |
| StartCodeInterpreterSession | 1,271 |
| InvokeCodeInterpreter (writeFiles) | 77 |
| InvokeCodeInterpreter (executeCode) | 1,414 |
| StopCodeInterpreterSession | 146 |
| CreateEvent (Memory persist) | 3 |
The invoke_agent Strands span (~1,370 ms) sits in the same tree, so the agent’s reasoning, the sandbox compute, and the memory write are all visible in one view.
The precise conclusion: metrics are a property of the tool; the distributed trace is a property of Runtime. Nesting the Code Interpreter call inside a Runtime invocation is what unifies them. A standalone call is metered but not traced. This matters for debugging — if you want to correlate a computation with the reasoning that triggered it, the tool needs to run inside the Runtime, not alongside it.
The architecture that matters: schedule, don’t compute per question
The naive design is to run the sandbox on every question. That is wrong, and the rate limit forces the right shape anyway.
Artificial Analysis’s free Data API allows 10 requests per 24 hours. You cannot fetch per question. So the architecture splits into two clocks:
Ingest (slow, scheduled): one job pulls the latest AA data, writes a dated snapshot to S3 — s3://model-analysis-snapshots-<account>/snapshots/<YYYY-MM-DD>/llms.csv — and updates a latest/ pointer. One AA API call per ingest run. The API key lives in the AgentCore Identity vault, fetched at ingest time via the Runtime’s workload identity. The LLM never sees the key.
Compute (fast, on-demand, zero AA calls): the agent reads a dated snapshot from S3 into the sandbox and computes the frontier. “Frontier at timestamp T” is a compute over the snapshot from T. “How has the frontier changed?” is a compute across the snapshot series. Both are pure arithmetic on already-fetched data.
The agent’s data-path precedence: S3 snapshot (default) → live AA fetch (opt-in, ingest path only) → bundled fallback snapshot. This means an interactive question about the frontier never touches the AA API rate limit and never goes to the network.
The frontier is a deterministic function of a fixed snapshot. It does not need to be recomputed per question — it needs to be computed once per snapshot and served from the result. The sandbox belongs in the scheduled step, not in the interactive path.
When Code Interpreter is the wrong tool
For fixed, known computations run on a schedule, you may not need Code Interpreter at all. Pandas in a plain Lambda does the same work. The sandbox earns its place when the agent writes arbitrary analysis code at question time — novel, unplanned cuts over data where you do not know in advance what the code will look like.
The decision is the same “judge the layer” question as everything else in this series: reach for the managed primitive when the capability gap is real. Code Interpreter closes the gap between “the agent writes Python” and “the Python runs.” If the computation is fixed and you own the code, a Lambda is simpler and cheaper. If the code is agent-generated and the result needs to be trustworthy, the sandbox is the right layer.
What’s missing
The sandbox’s default network posture has documented hardening guidance — VPC mode and DNS firewall — that I have not tested. The experiment was run in the default network mode, which is sufficient for read-only analyst tasks that push data in and pull results out. For tasks that need to reach external services from inside the sandbox, the network posture matters and needs to be verified before trusting it with anything sensitive.
There is also no CloudWatch Logs group for the sandbox. Stdout from a long-running or streaming computation returns only in the API response. If you need to observe compute progress in real time, you have to instrument the code to emit structured output at each step and read it from the response stream rather than from a log group.
So what
A language model asked to compute will narrate instead. The Code Interpreter is the difference between a narrated answer and a computed one — and the tamper check is how you tell which you have.
Put the sandbox where the computation is expensive and the data changes: in the scheduled ingest step that materializes a dated result. Put the interactive agent on the fast path: read the precomputed answer, cite the snapshot date, serve it in milliseconds. The agent that answers your question should be reading a number that was computed earlier, not guessing one live.
Part of a series on Amazon Bedrock AgentCore, each post grounded in a live experiment. Related posts: The AgentCore Map, AgentCore Runtime: Where an Agent Actually Runs, AgentCore Memory: What Survives the Session, Two Ways to Authorize an Agent Tool, Who May Call What.