Author’s note: Starting from the cost structure of inference services, this piece explains why Token services tend to lose money, then gives you an actionable checklist for evaluating model API providers.
Tokens are already being sold in livestreams, but inference providers like SiliconFlow still aren’t looking great financially. In another news story, a company found Claude too expensive, switched to GLM, used 56 B200s to support 2,000 engineers, and cut AI costs by about 90%. Put those two things together, and the cost structure of Token services has its own logic. This article starts from the cost structure of inference services, explains why Token services lose money, and then gets to the question developers actually care about: how do you tell whether a model API provider is reliable?
This isn’t another industry-gossip explainer. The breakdown here is built around technical variables: GPU depreciation, utilization, the cost gap between Prefill and Decode, KV Cache, cache hit rate, and subsidy strategy. Once you’ve read it, you can use it directly in vendor selection.
Key value: By the end, you’ll understand what sits behind Token pricing, and you’ll have a six-dimension method for evaluating providers. You’ll also know how to use real invocation data to check whether billing is transparent.
Why Token Services Lose Money: Core Points of the Inference Cost Structure
Let’s start with the conclusion. Most of the cost of inference services is fixed cost, while revenue is billed by Token volume. Utilization is what connects the two. If utilization doesn’t improve, or if competition pushes prices down near marginal cost, the books will show a loss.
| Cost Element | Description | Impact on the Unit Price per Token |
|---|---|---|
| GPU capital expenditure and depreciation | High-end GPU purchase or long-term rental costs, amortized over their useful life | Fixed cost; the lower the utilization, the higher the per-Token allocation |
| Data center, power, and networking | Colocation, cooling, bandwidth, and cross-region disaster recovery | Semi-fixed cost; can only be diluted at scale |
| GPU utilization | Share of time actually running inference and batch saturation | Core profit-and-loss lever; the peak-valley spread directly eats into profit |
| Prefill / Decode structure | Input is processed in parallel; output is generated Token by Token | Output Tokens are more expensive; long-output scenarios put more pressure on costs |
| Long context and KV Cache | VRAM is occupied by cache, reducing concurrency capacity | Long-context requests reduce the number of users that can be served |
| Subsidies and customer acquisition | Free quota, low-price traffic acquisition, and channel revenue sharing | Drives volume in the short term, lowers gross margin in the long term |
Key Details of the Token Service Cost Structure
The mismatch between fixed costs and usage-based revenue is the first reason. Whether or not there are requests, a fully configured GPU server keeps incurring depreciation, electricity, and data center costs. Users pay only for the Tokens they actually consume. Daytime peaks require enough reserved compute to guarantee latency; at night, that compute sits idle. The cost of idle periods ultimately gets spread across the Tokens sold during peak periods. If pricing only references “theoretical cost at full load,” real operations will almost certainly lose money.
Price competition pushes unit prices toward marginal cost is the second reason. Open-source model weights are available to everyone, and multiple inference providers deploy the same model. With little room for differentiation, price naturally becomes the main competitive tool. Putting Tokens into livestream promotions shows that this kind of product is already close to a “standardized commodity.” In this situation, providers find it hard to cover fixed costs through premium pricing; they can only compete on scale and scheduling efficiency.
Breaking Down Inference Service Costs: How Token Prices Are Actually Calculated
To understand why Token services lose money, you need to split a single inference request into two stages. Different stages consume different resources, so they’re billed on different bases. That’s exactly why many providers price input, output, and caching separately.
Prefill vs. Decode: Why Output Tokens Cost More
A request first enters the Prefill stage. The model processes all input Tokens in parallel — compute-intensive but efficient, so GPU compute is fully utilized. Then comes the Decode stage, where the model generates one Token at a time and repeatedly reads the previous KV Cache. This stage is more constrained by memory bandwidth, and GPU compute is hard to saturate.
So for the same number of Tokens, the unit cost of output is usually noticeably higher than input. Mainstream providers also generally price output higher than input. If your workload is output-heavy — code generation, long-form writing, or multi-turn Agent reasoning — costs concentrate on the output side. When evaluating providers, look at output unit price and output throughput first.
KV Cache and Long Context: The Invisible VRAM Tax
The cost of long-context requests isn’t just “more input.” KV Cache grows with context length and consumes more VRAM, so the number of requests a single card can serve at once drops. In other words, one ultra-long-context user can hog the concurrency slots of several ordinary users. That cost isn’t necessarily fully reflected in the unit price — providers can only balance it through tiered pricing, rate limiting, or context length caps.
Cache Hits: Both a Cost Saver and a Pricing Trap
Prompt Caching lets repeated prefixes skip recomputation, with the hit portion billed at a lower cache-read price. For Agent and RAG workloads with long, repetitive system prompts, the cache hit rate significantly affects the final bill. Note that providers differ in how they bill cache writes, cache reads, and cache TTL — some require you to enable it explicitly. When evaluating, always confirm whether the returned usage field includes cache-related Token statistics.
| Billing Dimension | Resource Consumption Characteristics | What to Confirm When Evaluating |
|---|---|---|
| Input Token | Prefill runs in parallel, relatively efficient | Whether pricing is tiered by context length |
| Output Token | Decode generates one at a time, limited by memory bandwidth | Output unit price, output throughput (Token/s) |
| Cache Write | First write to cache; some platforms bill separately | Whether writes carry a surcharge, how long the TTL is |
| Cache Read | Prefix hit, skips recomputation | Discount on the hit portion, whether usage exposes it |
| Reasoning Thinking Token | The internal thinking process of reasoning models | Whether it counts toward output, whether the budget can be capped |
💰 Pricing scope note: The table above only covers billing dimensions, not specific prices. For the input unit price, output unit price, and cache read/write prices of the models discussed here, refer to the actual data on the APIYI pricing page
https://docs.apiyi.com/models. Billing rules may vary by model and by group, so check each item before ordering.
Self-hosted inference vs. calling APIs: how do you calculate token costs?
The case of “56 B200 GPUs supporting 2,000 engineers and cutting costs by about 90%” rests on a few preconditions. The company’s load comes from internal engineers, with high and relatively stable request volume. Usage is concentrated during working hours and highly predictable. And the model was swapped for GLM, an open-source model that can be self-hosted. These three factors let fixed costs be spread out sufficiently, neatly avoiding the utilization problem that public inference providers struggle with most.
Conversely, this case doesn’t mean every team should self-host. If request volume is small and volatile, or you need to switch frequently among multiple closed-source models, the idle cost of a self-built GPU cluster will quickly exceed your API bill. The value of public inference providers is precisely that they pool load from many customers to smooth peaks and fill valleys, achieving higher overall utilization.
| Option | Cost structure | Best for | Main risks |
|---|---|---|---|
| Self-hosted GPU cluster | High fixed cost, low marginal cost | Large teams with high usage, stable load, and a fixed model | Idle waste, ops staffing, hardware depreciation as generations turn over |
| Official API for a single model | Pure pay-as-you-go | Businesses that depend heavily on one closed-source model | Single price point, high switching cost, limited regional availability |
| Multi-model API aggregation platform | Pay-as-you-go, unified interface | Teams that need to compare, switch, and mix multiple models | Need to verify billing transparency and stability |
| Hybrid architecture | Self-host high-frequency tasks; route long-tail tasks to APIs | Mid-to-large businesses with clear model tiering | Higher architectural complexity |
Benchmarking the self-hosted option: Self-hosting has an advantage in the lower bound of unit cost, but only if utilization is high enough and the team can handle deployment, quantization, scaling, and failure recovery. By comparison, the unit cost of an API approach is more predictable and better suited to teams with volatile load that need to validate multiple models in parallel.
Benchmarking the aggregation platform option: Official APIs have an edge in how quickly new models launch, but in most cases one account only covers one vendor’s models. The benefit of a multi-model aggregation platform is that one interface connects to multiple models, making A/B comparisons and cost tiering easier. Platforms like APIYI apiyi.com are compatible with both OpenAI and Anthropic API formats, so you can switch models with very little change to existing code.
How to Evaluate Model API Providers: A 6-Dimension Quick-Start Checklist
Once you understand the cost structure, you can work backward to a conclusion: prices below cost can’t be sustained over the long term. Providers are either skimping somewhere or leaning on subsidies for a while. When evaluating a provider, don’t just look at the list price—look at these six dimensions together.
| Dimension | Core question | How to verify |
|---|---|---|
| 1. Price transparency | Are input, output, and cache pricing listed separately? | Cross-check the pricing page and record the basis for each item |
| 2. Billing consistency | Does the actual charge match usage? |
Check bill details after a small batch of calls |
| 3. Model authenticity | Is there downgrading, mixing, or over-quantization? | Compare output quality on a fixed test set |
| 4. Performance stability | How are first-token latency, throughput, and error rate? | Run one test at peak and one at off-peak |
| 5. API compatibility | Is it compatible with OpenAI / Anthropic formats? | Use your existing SDK, change only base_url, and test |
| 6. Sustainability | Is the price clearly below a reasonable cost range? | Judge subsidy risk based on the cost structure |
Step 1: Connect via a compatible API and make a minimal call
You can explain the test process clearly without writing code. If your project already uses the OpenAI SDK, you usually only need to change two configuration items: replace api_key with the key provided by the provider, and replace base_url with the provider’s API endpoint. Take APIYI as an example: for the OpenAI-compatible format, the base_url is https://api.apiyi.com/v1; chat requests go to the /v1/chat/completions path, and you just put the target model name in the model field.
If your project is based on the Anthropic SDK—for example, using Claude-series models or Claude Code-style tools—you can likewise replace the API endpoint and use the Anthropic native format at the /v1/messages path. For specific addresses and model names, refer to the APIYI documentation at https://docs.apiyi.com. When making a minimal call, it’s a good idea to also enable stream streaming output, so you can measure first-token latency later.
Step 2: Verify the billing basis and usage fields
In the response of each call, the OpenAI format returns usage.prompt_tokens and usage.completion_tokens, and some models also provide cached_tokens for cache hits. The Anthropic format corresponds to input_tokens, output_tokens, and cache read/write fields. You can use a simple formula to estimate the cost of a single call: the number of input tokens multiplied by the input unit price, plus the number of output tokens multiplied by the output unit price, plus the number of cache read/write tokens multiplied by the corresponding cache unit price.
Compare the estimated value with the platform’s billing details; any discrepancy should be explainable. If “the bill is consistently higher than what usage suggests” over time, or if cache hits are never reflected in billing, that’s a red flag. For the unit prices used in the estimate, refer to the data published on the APIYI pricing page at https://docs.apiyi.com/models.
Step 3: Test performance, quality, and peak hours
For performance testing, it’s recommended to record three metrics: time to first token (TTFT), output throughput (tokens per second), and error rate (timeouts, 429 rate limiting, 5xx). Test one round during weekday peak hours and another during the nighttime off-peak period; the smaller the fluctuation, the more adequate the provider’s capacity planning. For quality testing, use a fixed set of prompts covering code, reasoning, and long text, and cross-compare the outputs with the official channel or other platforms—this helps detect whether a model has been downgraded or substituted.
| Test Item | Key Parameters / Fields | Acceptable Reference |
|---|---|---|
| Time to first token | Enable stream, record the time of the first chunk |
The difference between peak and off-peak isn’t too large |
| Output throughput | completion_tokens ÷ generation time |
Close to other channels of the same model |
| Billing verification | Each usage field × unit price on the pricing page |
Discrepancies with billing details are explainable |
| Cache hits | cached_tokens or cache read fields |
Repeated requests with long prefixes show hits |
| Stability | Number of 429 / 5xx / timeouts | Low error rate with retry documentation |
🚀 Quick Start: If you want to compare the cost and performance of multiple models under the same set of interfaces, you can first register through APIYI at apiyi.com and get test credits. The platform provides API access to the mainstream models covered in this article, compatible with both OpenAI and Anthropic interface formats. Just change
base_urltohttps://api.apiyi.com/v1to start validating. Model prices are subject tohttps://docs.apiyi.com/models.
FAQ
Q1: Why do token services lose money?
Inference services are dominated by fixed costs like GPU depreciation and data center power, yet revenue is billed by token volume. You need to reserve compute for peak hours, which sits idle during troughs, so utilization is hard to keep high over the long run. Add in the price wars and free-credit subsidies driven by homogenized open-source models, and prices easily get pushed below cost.
Q2: Why are output tokens usually more expensive than input tokens?
Inputs are computed in parallel during the Prefill stage, so GPU utilization is high. Outputs are generated one at a time during the Decode stage, which requires repeatedly reading the KV Cache and is mainly limited by memory bandwidth — so the unit cost is higher. That’s why businesses with long outputs should pay close attention to output pricing and throughput.
Q3: Is self-hosting GPUs always cheaper than calling an API?
Not necessarily. Cases like “56 B200s supporting 2,000 engineers” depend on large-scale, stable, and predictable internal workloads. Teams with small or highly fluctuating usage often end up with serious idle capacity after self-hosting, making the unit cost even higher. A better approach: run on an API first to map out your real usage curve, then evaluate the break-even point.
Q4: Can I use an API provider with extremely low prices?
You can try it, but verify thoroughly. Prices clearly below the reasonable cost range usually mean one of the following: subsidies, rate limiting, model downgrades, or a discount on stability. Running through the 6-dimension checklist in this article to verify quality and billing is more reliable than just looking at the sticker price.
Q5: How can I compare API performance across multiple models at low cost?
We recommend using a multi-model platform compatible with mainstream API formats for unified testing:
1. Register an account on APIYI apiyi.com and get an API Key
2. Configure the OpenAI- or Anthropic-compatible endpoint according to the docs at `https://docs.apiyi.com`
3. Use a fixed test set and rotate the `model` field, recording latency, quality, and `usage`
You can complete side-by-side comparisons with the same set of code, so switching costs are very low.
Q6: How do I check cache pricing? Is it defined the same way across platforms?
No, it isn’t. Some platforms bill cache writes separately, while others only discount reads; validity periods and how caching is enabled also differ. When calling through APIYI, refer to the actual cache read/write prices on the pricing page at `https://docs.apiyi.com/models`, and cross-check the charges against the cache fields in the returned `usage`.
Q7: What’s the most overlooked metric when evaluating a provider?
Billing consistency and peak-hour stability. Many teams run a single test during off-peak hours and then go live, only to hit frequent 429s or timeouts during peak periods. Plenty of teams also never verify that `usage` lines up with their bill. Both should be part of your standard pre-launch checks.
Token Service Costs and API Provider Evaluation: Key Takeaways
- Where losses come from: high fixed costs, usage-based revenue, and fluctuating utilization — the mismatch between these three makes Token services easy to run at a loss
- Output costs more: decode is limited by memory bandwidth, so any workload with long outputs needs to watch output unit price and throughput closely
- Long context carries hidden costs: KV Cache eats into GPU memory and lowers concurrency, which leaves providers balancing things with tiered pricing and rate limiting
- Self-hosting has a barrier to entry: only large-scale, steady workloads can amortize GPU fixed costs — small teams are better off starting with APIs
- Evaluate on six dimensions: price transparency, billing consistency, model authenticity, performance stability, interface compatibility, and sustainability
- How to get started: run quick real-world tests through APIYI’s OpenAI/Anthropic-compatible formats; for pricing, refer to
docs.apiyi.com/models
Summary: Understand Token Services Through Their Cost Structure, and Pick the Right Model API Provider
Why do Token services lose money? It comes down to the cost structure of inference serving:
- Cost side: GPU depreciation, data center power, and facility costs are fixed. Utilization is the profit-and-loss lever, and output tokens plus long context push costs even higher
- Revenue side: open-source models have become commoditized, making price wars hard to avoid — subsidies bring volume but squeeze gross margin
- Selection side: a low price doesn’t mean a good deal. Judge providers across six dimensions: price transparency, billing consistency, model authenticity, performance stability, interface compatibility, and sustainability
For developers, rather than guessing which provider will last, it’s better to let real invocation data do the talking. Start with a small volume of traffic to verify billing accuracy and compare quality, then decide on your primary model and backup channels.
I recommend validating quickly through APIYI at apiyi.com. The platform offers API access to the models covered in this article and is compatible with both OpenAI and Anthropic interface formats. Integration docs are at https://docs.apiyi.com. For input, output, and cache pricing across all models, the actual data on the pricing page https://docs.apiyi.com/models is the authoritative source.
📚 References
- 36Kr – Tokens Are Being Sold in Li Jiaqi’s Livestream, So Why Are “SiliconFlow”-Type Companies Still Losing Money?: A report on the profitability struggles of inference service providers
- Link:
36kr.com/p/4055385238017664 -
Note: Background on the commoditization of tokens and why inference service providers are operating at a loss
-
36Kr – Someone Was Burning $10,000 a Month and Decided Claude Was Too Expensive, So This Company Switched to GLM: A case study on enterprises cutting costs with self-hosted inference
- Link:
36kr.com/p/4055383551038595 -
Note: A self-hosted inference case where 56 B200 GPUs support 2,000 engineers — useful for understanding the cost boundary between self-hosting and API access
-
APIYI Documentation Center: API integration and model pricing
- Link:
docs.apiyi.com - Note: See the OpenAI / Anthropic-compatible integration methods; for model pricing, refer to
docs.apiyi.com/models
Author: APIYI Technical Team
Technical Discussion: Have questions about model selection or token cost evaluation? Feel free to reach out through the APIYI official website at apiyi.com. For more integration resources, visit the docs.apiyi.com documentation center.
