Agent Fleet Observability: Agent Output Is Now Logs

One agent in a terminal is a chat. Twenty agents across repos, hosts and incidents are a fleet, and their output is no longer something you read. It is something you collect, index and alert on, the same way you treat service logs.

Terminal tools already show part of the picture. Our post on Herdr covers agent state in panes. But "which pane is blocked" is only the first question. An operator running a fleet also asks: what is each agent trying to do, how long has it been at it, what has it cost, is its prompt cache about to go cold, and which manager spawned which worker?

Below: the signals, where Claude Code and the OpenTelemetry GenAI conventions expose them, and the traps.

Six Signals a Fleet View Must Show

State

Claude Code's hooks reference lists the lifecycle events you need: SessionStart "when a session begins or resumes", Notification "when Claude Code sends a notification", SubagentStart "when a subagent is spawned", SubagentStop "when a subagent finishes", Stop "when Claude finishes responding", and SessionEnd "when a session terminates". The Notification matcher separates permission_prompt from idle_prompt, which is the difference between "needs approval" and "waiting for the next task". Every hook gets session_id, so a short script can write one state line per session to a log your collector already ships.

Intent

State says the agent is working. Intent says on what. Claude Code exports this through its OpenTelemetry monitoring, but not by default: "User prompt content is not collected by default. Only prompt length is recorded." You enable it with OTEL_LOG_USER_PROMPTS=1. Likewise, "tool input arguments and parameters are not logged by default"; OTEL_LOG_TOOL_DETAILS adds them, including Bash commands. The OpenTelemetry GenAI conventions take the same line: message content and tool arguments are opt-in attributes.

That default is right for privacy and wrong for a fleet view. For an agent with production shell access, tool details are your audit trail: turn them on and keep the logs where your secrets policy allows.

Duration

One number hides the important split. The status line docs give two: cost.total_duration_ms, "total wall-clock time the session has been running", and cost.total_api_duration_ms, "total time spent waiting for API responses". A session with 40 minutes of wall time and 3 minutes of API time is not thinking hard. It is waiting on a tool or a person. With tracing on, the span claude_code.tool.blocked_on_user shows the human wait directly, next to claude_code.tool.execution.

One more trap: a slow agent may be retrying. The monitoring docs say a single claude_code.api_error is emitted only after retries are exhausted, and CLAUDE_CODE_MAX_RETRIES defaults to 10. Count retries, or the fleet view shows a healthy agent that is quietly stuck.

Cost

Each claude_code.api_request event carries model, cost_usd, duration_ms, input_tokens, output_tokens, cache_read_tokens and cache_creation_tokens. That is enough to attribute spend per session and per model. Two caveats from the docs themselves: "cost metrics are approximations", and the status line cost is "computed client-side at list price" and "may differ from your actual bill". Use these numbers for budgets and outliers, and the provider's billing data for the invoice. Our token cost guide covers where the tokens go.

Cache Expiry

This is the signal most dashboards miss. Anthropic's prompt caching docs say: "By default, the cache has a 5-minute lifetime. The cache is refreshed for no additional cost each time the cached content is used." The clock starts at the start of the request, and "time spent generating a response counts against the lifetime." A 1-hour TTL is available with "ttl": "1h" in cache_control.

The price gap is large. Cache writes cost 1.25 times the base input price for the 5-minute TTL and 2 times for the 1-hour TTL. Cache reads cost 0.1 times base on most models, and less on some newer ones. Claude Code splits the writes for you: api_request also reports cache_creation_5m_tokens and cache_creation_1h_tokens.

So "cache cold in 90 seconds" belongs next to each idle agent, computed as the start of its last request plus its TTL.

Tree

A manager that fans out to five workers is one job, not six sessions. Claude Code's beta tracing (CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1) models this: "When the Agent tool, or legacy Task tool, spawns a subagent, the subagent's API and tool spans nest under the parent's claude_code.tool span." Spans carry agent_id and parent_agent_id, and Bash and PowerShell subprocesses inherit a TRACEPARENT environment variable, so work an agent starts outside Claude Code can join the same trace.

For agents you build yourself, the GenAI conventions define invoke_agent and execute_tool spans with gen_ai.agent.name, which "MUST have low cardinality". Put the run id in the trace, not in the agent name. Note that those convention pages are marked Development, so pin your instrumentation version.

The tree shows what a flat list cannot: a manager that says done while a worker still runs, a worker blocked on approval that holds up the whole job, and costs rolled up to the job that caused them.

Worked Example: The Cache Clock

Take a worker holding a large cached prefix on a model with 0.1x cache reads. Call the price of that prefix at base input rate P.

• A warm turn reads the prefix: 0.1 P.
• A turn after the cache expired writes it again: 1.25 P. That is 12.5 times the warm turn for the same work.

Now the worker waits for a manager or a human, and the gap is longer than 5 minutes but under an hour, twice in an hour of work:

The rule falls out of the multipliers: if an agent is likely to sit idle between 5 and 60 minutes even once within the hour, the 1-hour TTL pays for itself. If it runs turn after turn with no gaps, the 5-minute TTL is cheaper. A fleet view that shows idle time per agent tells you which kind each agent is. Without it you are guessing.

Wiring It Up

A minimal Claude Code setup that exports metrics, events and traces:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
export OTEL_LOG_TOOL_DETAILS=1     # intent: tool names and arguments

Set the protocol explicitly: the docs say Claude Code "has no default protocol". Then join the signals with the keys the docs give you: session.id on everything, request_id from llm_request spans to api_request events, and tool_use_id from tool spans to tool_result and tool_decision events.

Failure Modes

• Transcripts lag. The hooks docs say the transcript file "is written asynchronously and may lag the in-memory conversation". Do not compute state from the tail of a transcript. Use the events.
• Intent is blank by default. Prompts and tool arguments are off until you opt in.
• Cost is an estimate. Reconcile with billing before you charge back.
• Retries look like thinking. An error event appears only after retries run out.
• "Done" is not "correct". Stop means the model finished responding, not that the task succeeded.
• Conventions move. The GenAI conventions are in Development. Attribute names may change between releases.

Checklist

• One state line per session from Notification, Stop and SubagentStop hooks.
• Tool details on, with retention matched to your secrets policy.
• Wall time and API time side by side, with human wait as its own number.
• Cost per session, rolled up per tree, marked as an estimate.
• Cache expiry time for each idle agent, and a TTL chosen from measured idle gaps.
• Parent and child ids on every span, and the run id in the trace, not in the agent name.

Where Akmatori Fits

Akmatori runs SRE agents against production systems, so the audit trail is part of the product. As the connectors docs say, "every tool call is recorded in the run log of its incident or request", and the API docs say an incident is returned "including execution state and logs". Tools run through the MCP Gateway, which builds each run's allowlist from its run kind and refuses every other tool, as described in the architecture docs. Akmatori also records which memories each run searched, read and cited, so you can see what the agent relied on, not only what it did.

Conclusion

Once agents work in parallel, their output is operational data. Treat it like logs: structured events, consistent ids, a parent for every child, and alerts on the six signals above. Most of it can be collected today; the work is turning it on and joining it.

If your team wants AI-assisted incident workflows with every tool call on the record, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.