inferwire
/
AI·5 min read

TokenPilot Cuts Agent Costs by Keeping Prompt Caches Intact

Researchers show that trimming an agent's context can break prompt caching, and cut costs by 56 to 87 percent on two benchmarks with a system built around a stable prompt prefix.

TL;DR

  • TokenPilot, from Zhejiang University and partners, cuts agent inference costs by keeping prompt prefixes identical, so provider caches stay valid while old context is trimmed [^1].
  • On PinchBench and Claw-Eval it cut costs by 61 and 56 percent in isolated runs and by 61 and 87 percent in continuous sessions [^1].

Background

Agents that run for many turns resend a growing history with every model call. Prompt caching softens the cost: content the provider has seen before is billed at a discount, and Anthropic charges 10 percent of the base input price for cached reads [^2]. But the cache only helps if the prompt begins with exactly the same content as before. Pruning or summarizing old context shrinks the prompt, yet it can change that prefix and lose the cache [^1].

What happened

The paper, posted to arXiv on June 15 by researchers from Zhejiang University, HomologyAI and other institutions, argues that existing context managers trade one cost for another. Truncating or shifting context lowers the token count per turn but "shatters prompt prefix continuity", and the resulting cache misses can wipe out the savings [^1]. TokenPilot works at two levels. Where messages enter the context, it replaces volatile fields in the system prompt, such as working directory paths, timestamps and session identifiers, with static placeholders and moves tool definitions to the end of the system prompt, so the prefix stays byte-identical from the first turn [^1]. It also shrinks bulky tool output before it enters the history by deduplicating repeated results, truncating long outputs and slimming HTML. A recovery tool lets the agent fetch the full payload when the short version lacks something [^1].

Inside the running context, TokenPilot tracks each segment as active, completed or evictable. A completed segment keeps its place as long as it may still be useful to later tasks, and eviction runs on a batch schedule, set to three turns, rather than every turn [^1]. A separate model, Qwen3.5-35B-A3B, judges which segments are done; across the continuous PinchBench run it cost less than $0.03 [^1].

All methods used GPT-5.4-mini as the agent model. The team tested on PinchBench, with 123 tasks in 11 categories, and on 161 tasks from Claw-Eval, against nine context managers including LLMLingua-2, Keep-Last-N, summarization and MemOS [^1]. Cache hits and misses came from the provider's API metadata and were priced at $0.075 per million cached tokens and $0.75 per million uncached ones [^1]. In isolated mode on PinchBench, TokenPilot cost $3.22 against $8.31 for an agent without context management and scored 81.0 against 80.5 [^1]. In continuous mode on Claw-Eval, the unmanaged agent's bill rose to $81.52, while TokenPilot's came to $10.58 [^1]. The placeholders alone raised the cache hit rate from 38.7% to 79.2% on PinchBench and from 67.2% to 83.1% on Claw-Eval [^1].

Why it matters

When cached tokens are billed at a tenth of the price, the cheapest prompt is not necessarily the shortest one. In the study, text compressors lowered costs but hurt task scores, while dynamic memory systems kept quality but paid for cache misses [^1]. The ablation shows how much layout alone is worth: placeholders cut the isolated PinchBench cost from $8.31 to $4.35 without removing any content [^1]. Timing matters too. Evicting every turn caused early truncation that inflated cache misses, while never evicting let context and cost peak [^1]. And trimming needs a way back: without the recovery tool, the score fell from 80.9 to 77.1 and the cost rose to $4.03, because the agent retried and added fresh tool output [^1].

The gains are not uniform. On Claw-Eval, TokenPilot scored 63.1 against 64.5 for the unmanaged agent in isolated mode and 60.8 against 63.4 in continuous mode, where several baselines also scored higher [^1]. The continuous sessions grouped tasks of the same category, and the authors say mixed task streams may reuse less of the prefix [^1]. They also note that prefix stabilization only helps with providers that support prefix caching, that the thresholds may need tuning and that the estimator can misjudge segments [^1]. All results come from a single agent model. TokenPilot has been integrated into the team's LightMem2 project [^1].

Practical example

Say Amara runs a small IT-automation startup whose agent works through support tickets in long sessions, reading logs, running shell commands and fetching web pages. Every model call starts with a system prompt that includes the current working directory and a timestamp, followed by the tool definitions for the current task. Because those values change between tasks, the opening of the prompt differs, and tokens that could be billed at the cached rate are billed in full [^1].

Amara first tries a common fix: summarize the history once it gets long. The prompt shrinks, but each summary rewrites part of the context, and the cache misses again [^1]. Following TokenPilot, Amara's team instead swaps the directory path and timestamp for fixed placeholders and moves the real values and the tool definitions to the end of the system prompt [^1]. Long shell output is cut down to its first 600 and last 400 characters, which the agent can expand through a recovery tool when it needs the rest [^1]. A log the agent has already analyzed stays in context while later tasks may still need it, and is dropped in a batch rather than at the next turn [^1].

In the paper, the placeholder step alone cut one benchmark's cost from $8.31 to $4.35 [^1]. Amara still checks task quality on the startup's own tickets, since on Claw-Eval TokenPilot scored slightly below the unmanaged agent [^1].

Related gear

We recommend this book because it covers agents and inference optimization for applications built on foundation models, the two areas TokenPilot sits between.

AdvertisementAmazon

AI Engineering: Building Applications with Foundation Models

Sources

  1. [1]arXiv — TokenPilot: Cache-Efficient Context Management for LLM Agents
  2. [2]Anthropic — Prompt caching with Claude