Observability is the practice of capturing enough structured signal from a running agent system to answer, after the fact, questions like why did it do that, how long did that step take, which tool failed, and what did the model actually see.

Without observability, an agent system is a black box that emits free-text replies and either works or doesn't. With observability, the same system becomes legible — failures have causes, costs have breakdowns, regressions have signatures.

Three signal types

A useful observability setup captures three kinds of signal:

  • Traces. A step-by-step record of one run: each model call, each tool call, each decision point, in order. Traces are what you reach for when something went wrong and you need to reconstruct the path.
  • Metrics. Aggregated counts and distributions: turn counts, tool-call latency, token spend per session, error rates per tool, compaction frequency. Metrics are what you reach for when you want to know whether things are healthy at a glance.
  • Logs. Free-form, line-oriented events. Less structured than traces, more searchable than metrics. Useful for context that doesn't fit either schema cleanly: a retry reason, a human note, an external signal.

The three types are not interchangeable. Traces answer what happened, in what order. Metrics answer how often, how fast, how much. Logs answer what was the operator thinking when this ran.

What to capture

A useful rule: capture what an operator would need to reconstruct a session six months from now, and nothing more.

Concretely:

  • Model calls. Prompt, response, token counts, latency, model version. The minimum is the call itself.
  • Tool calls. Tool name, arguments (sanitized), result (truncated if large), success or failure, latency.
  • Decisions. Where the agent loop branched: which tool was chosen over which alternative, which sub-agent was spawned, which memory was retrieved.
  • Errors. Stack traces, retry counts, the recovery path that succeeded or failed.
  • State transitions. When a state graph moved from node A to node B, and what condition triggered the move.

The cost of over-capturing

Observability is not free. Three costs to weigh:

  • Volume. Every trace is a write to disk or a network round-trip. At scale, traces can dwarf the work they describe.
  • Latency. A synchronous trace call on the hot path slows the agent. Asynchronous, batched capture is usually the right move.
  • Privacy. Traces contain user input, retrieved private data, and sometimes credentials. Captured traces become a privacy surface that needs its own access controls.

The discipline is to capture the smallest signal that supports the debugging you actually do. Operators who capture everything end up querying nothing, because the noise drowns the signal.

See also

  • Agent Loop — observability is layered on top of the loop, one record per step.
  • State Graph — state transitions are first-class signal in graph-routed systems.
  • Tool — tool calls are the most important signal to capture, because they are where the agent meets the world.

The observability stack

The observability stack typically has three layers:

1. Logs. The structured records of events. The logs answer "what happened?" The logs are the source of truth for the agent's behavior.

2. Metrics. The aggregated counts of events. The metrics answer "how often?" The metrics are the source of truth for the agent's performance.

3. Traces. The connected records of events across services. The traces answer "where did the time go?" The traces are the source of truth for the agent's latency.

The three layers together produce the observability stack. The stack is what the operator uses to debug the agent, the stack is what the operator uses to evaluate the agent, and the stack is what the operator uses to improve the agent.