Why does basic LLM telemetry stop working once the model becomes an agent?

The tempting answer is to track prompts, outputs, latency, tokens, and errors. That answer is not useless, but it is too vague to operate. LLM observability tracks prompts, outputs, model choices, latency, tokens, errors, and quality signals for model calls. Agent workflows need that foundation, but they also need traces for tools, context, guardrails, approvals, retries, and side effects.

Generated hand-drawn illustration of an agent session state ledger with tool calls and trace checkpoints.

Direct answer

LLM observability tracks prompts, outputs, model choices, latency, tokens, errors, and quality signals for model calls. Agent workflows need that foundation, but they also need traces for tools, context, guardrails, approvals, retries, and side effects.

The common mistake

The sharper operating question is:

Where this gate sits

LLM observability is the lower layer. It explains generation behavior; agent observability explains operational behavior across the workflow.

Signals to capture

SignalWhat to inspectGate action
Model callPrompt, output, model, latency, tokensDebug generation
QualityGrounding, hallucination, format, refusalTune or eval
ContextRetrieval source, trust, recencyExplain evidence
ToolCall, result, side effectExplain action
ControlGuardrail, approval, evalExplain permission

Running example

A support model gives a wrong policy answer. LLM observability shows the prompt and output. Agent observability shows the retrieval source was stale and the policy eval was skipped. The fix is not only a better prompt; it is a context and gate fix.

Implementation checklist

  • Separate model-call metrics from workflow metrics.
  • Track cached, input, and output tokens by run type.
  • Record model choice and fallback decisions.
  • Join LLM output quality to retrieval and tool context.
  • Escalate to agent tracing when side effects are involved.

What changes in production

In a demo, LLM observability can look like a reviewer preference. In production, it has to become a branch in the agent runtime.

The branch is simple: if the system sees “Prompt, output, model, latency, tokens”, it should debug generation. If it sees “Grounding, hallucination, format, refusal”, it should tune or eval. If it hits “The output was fine but the tool action was wrong”, the run should not continue as if nothing happened.

For LLM observability, that is the difference between a content checklist and a control gate. The gate changes the next action while the run is still alive.

What to log in the trace

  • model_name
  • prompt_ref
  • output_ref
  • input_tokens
  • output_tokens
  • cached_input_tokens
  • linked_trace_id

Review packet

A reviewer, on-call owner, or future incident review should be able to answer three LLM observability questions from the packet:

  1. What evidence triggered this LLM observability gate?
  2. What action did this LLM observability gate allow, deny, retry, or escalate?
  3. What would have happened if the LLM observability gate had been absent?

For LLM observability, the packet should point directly at the trace fields above and the specific signal row that caused the decision. If the packet only says “agent requested approval” or “policy failed,” it is not yet operational evidence.

When to escalate

  • The output was fine but the tool action was wrong.
  • Latency or cost shifted because orchestration changed.
  • A retrieval chunk altered behavior.
  • A model-call dashboard cannot explain a customer-visible side effect.

Frequently Asked Questions

What is LLM observability?

LLM observability is visibility into model calls: prompts, outputs, model selection, latency, token use, errors, and quality signals. It is necessary for agents, but not sufficient for tool-using workflows.

Why is agent observability broader?

Agents call tools, retrieve context, retry, route to approvals, and create side effects. Those operations can fail even when the underlying model call looks normal.

What should teams add first?

Add trace IDs that link model calls to retrieval, tools, guardrails, evals, approvals, and final actions. That gives LLM metrics the workflow context they need.

The Takeaway

LLM observability explains the model call. Agent observability explains the decision system around it.

Sources