Where AI Agent Tokens Go: Cache, Context, Idle Time
For a month we read a large chat of developers and DevOps engineers who run coding agents every day. The most common complaint was not quality. It was cost: "41% of the weekly limit in three days", "the plan runs out in an hour", "the same work cost five times more this week". Most people could not say where the tokens went.
The answer is almost always the same. An agent does not pay mostly for thinking. It pays for reading. Every turn of the agent loop sends the whole conversation to the model again: the system prompt, the tool definitions, every tool result so far. Whether that re-read costs full price or a tenth of it depends on the prompt cache. This post shows the math with real prices, the traps that silently turn cheap turns into expensive ones, and a checklist for long-running agents such as an AI SRE agent working on an incident.
Prices below are Anthropic's published API prices on 2026-11-08, from the pricing page and the prompt caching docs. Other providers use the same ideas with different numbers.
The Agent Loop Re-Reads Everything
Take one investigation: 40 tool calls (queries, log reads, kubectl commands). The context starts at 20K tokens and grows to 150K as tool results pile up. Each turn, the model reads the whole context again. The sum over 40 turns is about 3.4 million input tokens. Output is small, about 1,500 tokens per turn, so 60K tokens in total.
On Claude Opus 5.5 ($4 per million input tokens, $20 per million output tokens):
| Without cache | With cache | |
|---|---|---|
| Context re-read (input) | 3.4M × $4 = $13.60 | 3.25M cache reads × $0.20 = $0.65 |
| New tokens written to cache | none | 150K × $5 = $0.75 |
| Output | 60K × $20 = $1.20 | 60K × $20 = $1.20 |
| Total | $14.80 | $2.60 |
The same investigation costs almost six times more without cache hits. Note also that without the cache, the output, the part where the model actually works, is less than 10% of the bill. The rest is reading.
So the real question about agent cost is: how often does a turn miss the cache?
Trap 1: The Cache Expires While You Think
The default cache lives for 5 minutes. Each cache hit refreshes it for free. The lifetime is counted from the start of the request, so a long generation uses part of the 5 minutes.
An agent that works without pause keeps its cache warm. A human in the loop does not. During an incident, an engineer reads the agent's findings, checks a dashboard, talks to a colleague, and comes back 10 minutes later with a follow-up question. That one turn now writes the full 150K context again at the write price:
- Cache hit: 150K × $0.20 = $0.03
- Cache miss after idle: 150K × $5 = $0.75
One follow-up costs 25 times more because of a coffee break. People in the chat noticed exactly this: resuming a large session after a pause "eats several percent at once". One user added a marker to their status line that turns orange 15 minutes before the cache expires and red at 5 minutes.
What helps:
- 1-hour cache. Writes cost 2x the input price instead of 1.25x, but the entry survives pauses up to an hour. It pays off when the gaps between turns are longer than 5 minutes but shorter than an hour.
- Keep-alive. Re-send the last request with
max_tokens: 0shortly before the cache expires. It bills only a cache read and resets the timer. Anthropic documents this pattern for pre-warming. - Compact before you pause. A smaller context is cheaper to rewrite.
Trap 2: Silent Cache Invalidation
The cache is a prefix match. Tools come first, then the system prompt, then messages. Any change in the prefix invalidates everything after it, and the API returns no error. You see it only as cache writes where you expected reads. Common causes in agent harnesses:
- Tool definitions change. Connecting or disconnecting an MCP server mid-session changes the tool list, which invalidates the whole cache.
- A timestamp or request ID in the system prompt. It makes every request a new prefix.
- Thinking or effort settings change. These are rendered into the prompt, so a change invalidates the message history.
- Model switch. A cache belongs to one model. Routing turns between two models gives you two cold caches.
To check, log cache_read_input_tokens and cache_creation_input_tokens for every request. If writes grow on every turn while the prefix should be stable, something in the prefix changes.
Trap 3: Fixed Overhead on Every Turn
Everything in the prefix is paid on every turn, as a cache read at best. Tool schemas, tool descriptions, skill descriptions, and long system prompts add up. One user in the chat counted 166 installed skills that cost about 10,000 tokens per turn before the agent did anything. Over a 40-turn investigation that is 400K tokens of overhead.
Keep the per-turn part small. One way is to move tool details out of the prompt into a file the agent reads only when it needs them. Akmatori does this: the agent calls connectors through a gateway, and their names, parameter schemas and examples are in a generated connected-tools/SKILL.md file that the skill points to. Disable the tool instances an installation does not use, so they are not in that list.
Myth: Long Context Costs More per Token
A common belief in the chat was that tokens get two or three times more expensive above 250K context. Check this for your provider. On Anthropic's API, Claude Opus 5.5 and Sonnet 5.5 bill the full 1M context window at the standard rate: a 900K-token request pays the same per-token price as a 9K-token one. The exception is Claude Haiku 5.5: a request with a prompt over 100,000 tokens pays $0.50 instead of $0.10 per million input tokens, five times more, and cache reads count toward that threshold.
Long context still costs more in total, because every turn re-reads it. And quality can drop in very long contexts. But the price per token is not the reason to compact, at least on these models.
Parallel Agents Multiply the Context
Several users burned a weekly plan in a day by running five or six agent sessions at once. Each session has its own context and its own cache. Each sub-agent starts with its own prompt and pays its own cache writes. Six parallel agents re-read six contexts. The work may finish faster, but the token bill grows with the number of agents, not with the amount of work.
Use parallel agents when the speed matters, and give each one a small, focused context. A sub-agent that reads 200 log lines and returns a 20-line summary saves the main agent from carrying those 200 lines on every later turn.
Count Cost per Solved Task
The cheapest model per token is not always the cheapest per result. A model that misses the root cause and needs a second investigation, or a human, costs more than a stronger model that gets it right once. Our Claude Haiku 5.5 SRE Bench results show this: a fast, cheap model solved the hardest incidents but missed the easiest one by skipping one command. Measure cost and success together on your own incidents, as SRE Bench does.
Akmatori records input, output, cache read and cache write tokens and an estimated cost for every agent run. The Runs tab of an incident shows the model, time and tokens of each run, so you can see which alerts are expensive to investigate.
Checklist for Long-Running Agents
| Check | Why |
|---|---|
| Log cache reads and writes per request | Misses are invisible otherwise |
| Keep tools, system prompt and model stable within a session | Any change invalidates the cache |
| No timestamps or IDs in the system prompt | They break the prefix on every request |
| Use the 1-hour cache or a keep-alive when humans pause | A miss after idle rewrites the whole context |
| Load only the tools and skills the task needs | They are paid on every turn |
| Delegate log reading to sub-agents with small contexts | The main context stays small |
| Compare models by cost per solved task | Retries and wrong answers cost more than tokens |
Conclusion
An agent's token bill is mostly the price of reading its own context again. Cache hits make that cheap, and idle gaps, changed prefixes and oversized tool lists make it expensive. Measure cache reads and writes, keep the prefix stable, and judge models by cost per solved task. To run an agent on your own models where the token price is your GPU cost, see Self-Hosted AI SRE Agent With Local Models.
If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.