Author’s note: This article breaks down the Token bill for long-running Agent tasks, explains why input Tokens account for most of the usage, and shows how to structure context to improve prefix cache reuse.
Many teams discover a problem only after integrating multi-step Agents into their products and checking the bill at the end of the month: a single user request can consume dozens of times more Tokens than an ordinary conversation. The first reaction is often “the model’s output is too verbose,” so they try to shorten it. In reality, most of the usage is on the input side, largely from historical context that has to be sent again on every turn.
This isn’t just a guess. Recent public engineering reports offer some concrete numbers: in production scenarios in China, a user request triggers an average of about 7.2 model invocations. In a 50-step invocation chain, input Tokens account for about 99.4% of usage, while output accounts for just 0.6%. Inference engine teams describe Agent workloads in similar terms: long prefixes, short outputs, and repeated cycles.
What you’ll learn: By the end of this article, you’ll understand the Token bill for long-running Agent tasks, why prefix caching reduces costs, and five ways to organize context that you can implement on the application side.
Key Takeaways on Token Costs for Long-Running Agent Tasks
Here’s a quick summary of the article’s conclusions. We’ll unpack each point in the sections below.
| Key point | What it means | Why it matters to you |
|---|---|---|
| Calls multiply quickly | A single user request averages about 7.2 model invocations, and complex tasks can easily exceed 50 steps | Estimate costs per task, not per request |
| Input dominates the bill | In a 50-step call chain, input accounts for about 99.4% of tokens, versus 0.6% for output | Prioritize reducing input; cutting output has limited impact |
| Repeated context is the main driver | Every step includes the conversation history again, so the same content is billed repeatedly | Identify where costs really come from |
| Prefix caching enables reuse | When an identical prefix hits the cache, repeated prefill computation can be avoided | Stable context structures cost less |
| Structure determines the hit rate | Keep prompts and tool definitions stable, put changing information at the end, and compress history when appropriate | Lower costs without switching models |
A Closer Look at Token Costs for Long-Running Agent Tasks
The biggest difference between an agent and ordinary chat is the loop. Take creating a PowerPoint presentation: the system needs to read the source material, plan the content, generate each slide, use tools to build the presentation, and check the results. To the user, that’s one task. Behind the scenes, it takes multiple rounds of reasoning and tool calls. Each step may reread the context, and if the results aren’t good enough, the agent has to go back and revise them.
On every call, the model receives the system prompt, tool definitions, the full history so far, and the new information for that round. The history keeps growing, while each round adds relatively little. Over dozens of rounds, the same prefix is billed dozens of times. That’s how input accumulates to 99.4% of the total.
Why Long-Running Agent Tasks Use Far More Input Tokens Than Output Tokens
The pattern becomes clearer from the inference engine’s perspective. TokenSpeed uses a coding agent as an example: the model keeps thinking, calling tools, receiving results, and then thinking again. The whole process consists of repeated short requests. The first request might already carry tens of thousands of tokens of prefix context, while each subsequent cycle often produces only a few hundred tokens.
The table below compares chat and agent workloads to help you figure out which category your application falls into.
| Dimension | Traditional Chat Workload | Long-Running Agent Workload |
|---|---|---|
| Request pattern | One question and one complete response | Repeated short requests interspersed with tool calls |
| Prefix length | Usually short | Can start at tens of thousands of tokens and grow with each cycle |
| Output per cycle | Relatively long | Often just a few hundred tokens |
| Context reuse | Loaded once, then released | The same prefix is repeatedly included throughout the task |
| Main cost driver | Both input and output are significant | Input, especially repeated prefixes, accounts for most of the cost |
How Prefix Caching Works in Long-Running Agent Tasks
An inference engine processes a request in two phases: during prefill, it computes the entire input and generates a KV Cache; during decode, it generates output tokens one at a time. For agent workloads, a prefix of tens of thousands of tokens needs to be prefilled in every cycle, and that computation is where much of the cost comes from. If the same prefix has already been computed and the result is still in the cache, the engine can reuse it and skip that prefill work.
Inference engine teams consider the KV Cache reuse rate a key factor in inference costs. According to the presentation, DeepSeek reportedly achieves a cache reuse rate above 99%, while most systems in the industry currently fall between 70% and 80%. For agent workloads, prefix cache hit rates during prefill are typically above 90%; after a hit, the engine only needs to compute a few hundred additional tokens.
There’s another easy-to-miss detail: prefix caching matches segments sequentially from the beginning. Early on, TokenSpeed took inspiration from SGLang and used a Radix Tree to manage its prefix cache. Each node in the tree represented a token sequence, and requests were matched against the tree starting at the root. Although TokenSpeed later switched to a Flat KV design, reuse still operates in fixed-size logical blocks of 128 tokens. For application developers, the takeaway is straightforward: if even one token differs near the beginning, none of the content after it can be reused.
How to Organize Context for Better Prefix Cache Hit Rates in Long-Running Agent Tasks
Once you understand how cache matching works, the optimization strategy is clear: make the beginning of each request as long and stable as possible. The WorkBuddy team keeps prompts and tool structures fixed, puts changing information at the end of the context, and compresses history once the context grows too long. Here are five practical ways to do that.
| Practice | What to do | Common mistake |
|---|---|---|
| 1. Keep the system prompt fixed | Keep the system prompt identical, and roll out version changes together | Insert the current time, username, or random examples at the beginning |
| 2. Keep the tool structure fixed | Keep the tool list order and parameter descriptions unchanged | Add or remove tools as needed each round, or use an inconsistent serialization order |
| 3. Put changing information at the end | Append timestamps, observations from the current round, and dynamic state at the end | Put “current progress” at the top of the system prompt |
| 4. Append to history without rewriting it | Keep previously sent history unchanged and add new content only at the end | Reformat or reorder previous messages every round |
| 5. Compress history periodically | Compress history once at a checkpoint when the context gets too long, then continue appending | Regenerate the summary every round, causing the prefix to change constantly |
Keeping Prompts and Tool Structures Fixed for Long-Running Agent Tasks
The first two practices are easy to overlook. Many frameworks insert the current date into the system prompt or dynamically select tools based on the current step. Both approaches change the beginning of every request, causing a prefix cache miss from the first token. A safer approach is to put only rules and roles in the system prompt, then add information like the date as a regular message at the end. List all available tools at once, and use the prompt to specify which ones to use at each stage instead of changing the tool list itself.
Serialization matters too. If the JSON field order for the same tool definition differs across processes, the resulting byte sequences are different prefixes. Sort tool definitions consistently before sending them, and log a hash of the prefix to help diagnose cache misses when nothing appears to have changed.
Balancing History Compression and Cache Reuse in Long-Running Agent Tasks
The fifth practice involves a trade-off. Compressing history reduces input length, but it also changes the prefix, so the current request is guaranteed to miss the cache. If you compress every round, you give up cache reuse every round. A better approach is to set a threshold—for example, compress once when the context reaches a certain length or when a subtask ends. Then use the compressed version as the new stable prefix, continuing to append without rewriting.
The two layouts above look almost identical when you consider a single request, but over dozens of rounds in a long-running task, the difference in cost can be significant. The most direct way to see which pattern your application follows is to check how input tokens change from round to round in your invocation logs. APIYI apiyi.com offers unified billing, usage statistics, and invocation logs, so you can compare data across multiple models in one place and diagnose this kind of billing breakdown.
Quick Start: Optimizing Token Costs for Long-Running Agent Tasks
This section doesn’t involve any code. Just follow these practical troubleshooting steps in order.
Step 1: Break Down Your Token Bill by Task
- Generate a task ID for each user task, and include it in the logs for every model invocation within that task.
- Record the input and output token counts from each response’s
usageinformation. Some providers also return the number of tokens served from cache; check their official documentation for the exact field names. - Aggregate three metrics by task: number of invocations, input-to-output ratio, and cache hit percentage.
If you use models from multiple providers, their logging formats often differ. With APIYI’s unified integration, you can view usage details and invocation records for all your models in one console. Combine those with your own task IDs to see how much each task costs and where the money goes.
Step 2: Find Where the Prefix Changes
- Take two consecutive requests from the same task and compare their system prompts, tool definitions, and message history section by section to see whether they match exactly.
- Pay particular attention to where dynamic fields, such as timestamps, random numbers, and user state, appear.
- Check that tool definitions and messages are serialized in a consistent order across requests.
Step 3: Apply the 5 Practices and Retest
- Move dynamic content to the end, and keep the system prompt and tool list fixed.
- Set a threshold for compressing history, and compress it at checkpoints rather than after every turn.
- Run the same set of test tasks on the versions before and after your changes. Compare input tokens and cache hit percentage per task.
🚀 Getting started tip: Use the same set of tasks and the same model when comparing results before and after your changes. Otherwise, the comparison won’t be meaningful. With APIYI’s unified interface, you don’t have to adapt to each provider’s interface when retesting with a different model. Usage statistics are also centralized, making it easier to distinguish the effects of structural changes from differences between models.
Limits of the Cost Estimate
WorkBuddy published an estimate based on its task sizes and TokenHub’s flagship-tier list prices: at a 90% cache hit rate, the estimated cost per task could drop from about 21.06 yuan to about 3.36 yuan. These figures depend on specific context lengths and pricing conditions. The full assumptions weren’t disclosed, so the calculation can’t be reproduced, and you can’t apply it directly to your application.
| Factor | Why it changes the result | What to check |
|---|---|---|
| Context length | The longer the prefix, the more you save by reusing it from cache | The average prefix length of your tasks |
| Cache pricing | Providers offer different discounts for cached tokens | Cache prices in the target model’s official documentation |
| Cache trigger conditions | Rules such as minimum length and expiration vary | Trigger and expiration rules in the official documentation |
| Time between invocations | The cache may expire if too much time passes between turns | Whether tool execution time fits within the cache lifetime |
| Number of task steps | More steps mean more repeated prefixes accumulate | The distribution of step counts in real tasks |
Complementary Ways to Reduce Token Costs for Long-Running Agent Tasks
Beyond context structure, the same presentation mentioned two application-side strategies that can work alongside prefix caching.
- Choose models by step: The WorkBuddy team shared that roughly 80% of simple steps can be handled by lightweight models, reducing per-task costs by about 57%. Steps like organizing information and converting formats don’t necessarily need a flagship model.
- Handle failures based on their cause: When a request times out, is rejected, or encounters a provider outage, limit retries based on the cause, adjust the context, or switch models. Unlimited retries can result in the same long prefix being billed repeatedly.
Keep in mind that switching models means switching to a different cache, so prefix reuse has to start over. When designing your routing, try to keep consecutive steps within the same subtask on the same model, and switch models at subtask boundaries.
Frequently Asked Questions
Q1: What is prefix caching in long-running Agent tasks?
Prefix caching means an inference service saves an already computed input prefix (KV Cache) and reuses it when a later request starts with the same prefix, avoiding prefill computation for that portion. Agents carry a large amount of the same history in each round, making them especially well suited to reuse.
Q2: How is prefix caching different from compressing historical context?
Prefix caching doesn’t reduce the input; it simply makes repeated portions cheaper to process. Compressing history shortens the input directly, but it changes the prefix and invalidates the cache for that round. The two approaches work best together: use append-only updates to maintain cache hits, then compress at checkpoints.
Q3: Does the 99.4% input share apply to all Agents?
Not necessarily. That figure came from a shared example of a 50-step invocation chain. Tasks with fewer steps or longer outputs per round will have a lower input share. But input usually accounts for most tokens in tasks with a “long prefix, short output, multi-round loop” structure. It’s best to check your own logs.
Q4: Which Agent scenarios benefit most from prefix cache optimization?
– **Coding Agents**: Repository context is long, with frequent cycles of reading files, editing code, and running tests
– **Document or PPT generation**: After reading the source material, content is generated page by page, with highly repetitive prefixes
– **Data analysis Agents**: Table schemas and metric descriptions stay fixed, while only query results are appended each round
Q5: What’s a quick way to understand my application’s input/output token breakdown?
The simplest approach is to aggregate usage data for each round by task ID. If your application uses multiple models, you can access them through APIYI at apiyi.com. The platform provides usage statistics and invocation logs, making it easier to view the input/output breakdown by model and time period. For prompt caching support and pricing for each model, check the platform’s documentation.
Q6: What are the known limitations of prefix caching?
– Trigger conditions vary by provider, including minimum length and expiration time, so check each provider’s official documentation
– If tool execution takes too long, the cache may expire before the next round
– Switching models or changing the system prompt invalidates the entire cache
– Discounts on cached portions vary by model, so measure the savings in practice
Key Takeaways: Token Costs for Long-Running Agent Tasks
- Calculate costs per task: A single request averages around 7.2 calls, and long-running tasks can exceed 50 steps.
- Input is the main cost driver: In a 50-step workflow, input accounts for about 99.4% of tokens, so reducing output has limited impact.
- Cache matching starts at the beginning: If even one token differs near the start, the content that follows can’t be reused.
- Structure matters more than the model: Keep prompts and tool definitions fixed, and put changing information at the end.
- Compress at the right intervals: Compress once at checkpoints rather than rewriting the prefix on every round.
- Don’t extrapolate cost estimates: Published figures depend on specific lengths and prices, so test them against your own workload.
- Observe before optimizing: Use APIYI’s usage statistics and invocation logs to break down your bill.
Summary
The main takeaways for token costs in long-running Agent tasks:
- Cost structure: Most costs come from repeatedly passing historical context into each round. Input tokens make up the vast majority.
- How caching reduces costs: Prefix caching avoids prefill computation for repeated prefixes. The cache hit rate depends on how context is structured.
- How to put it into practice: Keep prompts and tools fixed, put changing information at the end, only append to history, compress periodically, and choose models based on each step.
Start by spending a week breaking down logs by task to understand your input-token share and cache hit rate. Then apply the five practices in this article one by one. If you’re using multiple models, APIYI at apiyi.com provides unified access and a central view of usage and invocation records, so you can use the same data to measure the actual impact of each optimization.
📚 References
- Building the Next-Generation LLM Inference Engine for Agentic Reasoning: A summary of TokenSpeed’s talk at AICon 2026 Shenzhen
- Link:
infoq.cn/article/YwAayF1ylWb7oN4P0lyC -
Details: Covers the “long prefix, short output” characteristics of agent workloads, KV cache reuse rates, and prefix cache architecture
-
From Model Invocation to Task Delivery: How MaaS Enters Agent Production: A summary of Tencent Cloud’s MaaS session
- Link:
infoq.cn/article/zZutPTX8bsr6NWQYNC3A - Details: Includes WorkBuddy’s data on invocation counts, input share, prefix caching, and model routing
Author: APIYI Technical Team
Technical discussions: For questions about multi-model invocation and usage analysis, contact us through APIYI at apiyi.com. For more resources, visit the documentation center at docs.apiyi.com.
