Why did the agent get cheaper and worse at the same time?

The tempting answer is to cache as much prompt context as possible. That answer is not useless, but it is too vague to operate. Prompt caching reduces cost and latency when stable prompt prefixes stay stable across requests. In agent systems, it works best when durable instructions, schemas, and tool manuals are separated from volatile user data, retrieval, short-lived state, and approval context.

Generated hand-drawn illustration of stable prompt blocks separated from volatile task context.

Direct answer

Prompt caching reduces cost and latency when stable prompt prefixes stay stable across requests. In agent systems, it works best when durable instructions, schemas, and tool manuals are separated from volatile user data, retrieval, short-lived state, and approval context.

The common mistake

The sharper operating question is:

Where this gate sits

Prompt caching is a cost gate that belongs next to tracing and evaluation. It should save tokens only where quality gates prove the reused context is still valid.

Signals to capture

SignalWhat to inspectGate action
Stable contractRole, style, invariant policyPlace early
Tool manualSchemas, signatures, examplesKeep stable
Semi-stable policyVersioned rules and docsCache by version
Volatile contextUser data, retrieval, current statePlace late
Quality guardEval and trace signalBlock stale reuse

Running example

A coding agent keeps its style guide, tool schema, and repo workflow stable. It keeps the user request, current file diff, test output, and deployment status late in the prompt. The trace shows which runs benefit from cache and whether evals detect stale behavior.

Implementation checklist

  • Put stable text before volatile text.
  • Version policy blocks so cache behavior changes when rules change.
  • Keep user-specific facts and retrieval late unless they are intentionally reused.
  • Track cached-token share by workflow, not only globally.
  • Run evals when prompt layout changes, because cheaper can still be worse.

What changes in production

In a demo, Prompt caching can look like a reviewer preference. In production, it has to become a branch in the agent runtime.

The branch is simple: if the system sees “Role, style, invariant policy”, it should place early. If it sees “Schemas, signatures, examples”, it should keep stable. If it hits “Cached context includes user-specific or time-sensitive facts”, the run should not continue as if nothing happened.

For Prompt caching, that is the difference between a content checklist and a control gate. The gate changes the next action while the run is still alive.

What to log in the trace

  • prompt_layout_version
  • cacheable_prefix_hash
  • cached_input_tokens
  • volatile_context_bytes
  • eval_result_after_cache
  • cost_saved_estimate

Review packet

A reviewer, on-call owner, or future incident review should be able to answer three Prompt caching questions from the packet:

  1. What evidence triggered this Prompt caching gate?
  2. What action did this Prompt caching gate allow, deny, retry, or escalate?
  3. What would have happened if the Prompt caching gate had been absent?

For Prompt caching, the packet should point directly at the trace fields above and the specific signal row that caused the decision. If the packet only says “agent requested approval” or “policy failed,” it is not yet operational evidence.

When to escalate

  • Cached context includes user-specific or time-sensitive facts.
  • Cost improves but eval pass rate drops.
  • Prompt layout changes without regression tests.
  • The team cannot tell which prefix was reused for a bad run.

Frequently Asked Questions

What is prompt caching?

Prompt caching reuses stable prompt prefixes so repeated requests can be cheaper and faster. It is most useful when invariant instructions, schemas, and policies stay early and volatile context stays late.

What should not be cached?

Avoid caching user-specific facts, fresh retrieval, current state, approval packets, short-lived secrets, and anything whose staleness would change the correct action.

How do you know caching is safe?

Track cached-token behavior in traces and run evals around prompt-layout changes. If cost drops while task success or grounding falls, the cache layout is wrong.

The Takeaway

Prompt caching is not just a billing feature. It is prompt architecture: stable meaning early, volatile truth late, quality gates around both.

Sources