AI Knowledge Hub"Frequent Routing Is Poison" — Why a Mid-Tier Model Plus Context Caching Saves 90% of API Costs
"Frequent Routing Is Poison" — Why a Mid-Tier Model Plus Context Caching Saves 90% of API Costs
There's a familiar cost-optimization trap when building commercial AI agents.
"Assign simple tasks to ultra-cheap, small API models and switch to expensive large models like Claude 3.5 Sonnet or GPT-4o only for complex work — that should cut API costs dramatically?"
On paper it sounds right. But any developer who understands the billing mechanics of prompt caching in paid APIs knows this approach actually detonates the bill.
The conclusion drawn from building and operating paid API pipelines is unambiguous: "a single-hub architecture built around a mid-tier paid model that maximizes cache hits (Gemini Flash, Claude Haiku, GPT-4o-mini) wins decisively on cost-performance, speed, and cost alike."
Here's why anchoring on a mid-tier paid model beats frequent model switching, structurally, and what the practical architecture looks like.
1. Why Frequent Model Switching Destroys Cache Hits
The prompt caching discounts offered by major paid API platforms — Anthropic, OpenAI, Google Gemini — only apply when the same system prompt and context sequence arrive consecutively on the same model endpoint.
When an agent routes back and forth between small and large models based on fragmentary judgment, the following happens:
Destruction of the API server's KV cache: switching model endpoints means the previously generated cache can't be used, and each API server recomputes the entire prompt from scratch (token processing).
Cost and latency moving backwards: you try to save a few cents on a trivial small-model call, and then both models hit a prompt cache miss. In the end you pay full price for the entire context — tens of thousands of tokens — every single time, so total billing goes up and everything gets slower.
2. The Strong Advantages of a Mid-Tier-Anchored Architecture
The structure I propose is this: "keep a mid-tier paid API model with proven cost-performance and speed (e.g. Gemini 1.5 Flash, Claude 3.5 Haiku, GPT-4o-mini) resident as the main hub, and call a top-tier model (Claude 3.5 Sonnet, GPT-4o, etc.) only in extremely exceptional cases."
In terms of paid API cost, this structure is a formidable weapon.
Long-context cache residency (up to 80–90% discount on input tokens)
The agent's entire workflow, prior conversation context, project codebase, and fixed system prompt become resident (hit) in the mid-tier model's cache. Across major providers, cached tokens are discounted 75–90% versus regular input tokens, effectively running at a token rate "close to free."
Complete elimination of routing overhead
The cost and latency of the additional classification (routing) API call needed to judge "is this request easy or hard?" disappear entirely.
Task continuity and context preservation
The agent can continue its work steadily and precisely within a single session without losing context midway.
3. Practical Hybrid API Escalation Strategy
The recommended two-stage escalation pipeline for commercial API environments.
[Base Layer] 90% of the entire workflow & API tool executions → handled exclusively by a mid-tier paid model (Gemini Flash / Claude Haiku / GPT-4o-mini, etc.) (maximize prompt cache hits within the same model session) │ ▼ (error / exception occurs?) [Escalation Trigger] repeated code syntax errors, complex JSON parsing failures, internal judgment module activation │ ▼ [Top-tier Fallback] hand the entire context to a top-tier API model (Claude 3.5 Sonnet, GPT-4o, etc.), resolve it in one shot, then return Base Layer (Base Layer): 90% or more of routine work and API tool calls are handled by the mid-tier model, continuously receiving prompt cache discount benefits.
Fallback/Escalation Trigger (Escalation): only when the mid-tier model repeats syntax errors or fails at complex JSON parsing — clear exception conditions — is the context handed to a flagship API model (Claude 3.5 Sonnet, etc.), resolved in one shot, and then brought back down.
Wrapping up: getting the cache discount matters more than splitting everything up The longer you operate agents on paid APIs, the more you feel the truth in the statement that "rather than carelessly fragmenting into several cheap small models, a single mid-tier API model holding tightly onto the prompt cache discount dominates on cost-performance, speed, and stability."
Appendix. Evidence — Cache Hit Discount Rates by Provider
Here are the actual billing rules behind the claims above. Discounts land mostly in the 75–90% range, but caching mechanism and minimum token conditions vary considerably.
| Model | Regular input (per 1M) | Cache hit (per 1M) | Discount | Caching method | Notes |
|---|---|---|---|---|---|
| DeepSeek V4-Flash | $0.14 | $0.0028 | 98% | Automatic (identical prefix) | Free storage; partial matches miss |
| Claude Haiku 4.5 / Sonnet 5 | 1x | 0.1x | 90% | Explicit breakpoints | Write 1.25x (5 min) / 2x (1 hour) |
| GPT-6 Luna | $0.10 | $0.01 | 90% | Automatic (prefix) | Write 1.25x |
| GPT-5.5 | $5.00 | $0.50 | 90% | Automatic (prefix) | Stable prefix ≥1,024 tokens |
| Gemini 2.5 Flash | 1x | 0.1x | 90% | Implicit caching | Minimum 1,024 tokens |
| Gemini 2.0 (explicit) | 1x | 0.25x | 75% | Explicit caching | Storage billed separately |
Two things to read off this table
First, caching mechanisms differ everywhere. Anthropic requires you to set explicit breakpoints yourself. OpenAI and DeepSeek hit automatically when the prefix matches. Gemini has implicit caching on by default, but with minimum token conditions. In other words, the assumption that "the same prompt input will stick everywhere identically" is wrong.
Second, DeepSeek's 98% is exceptional. Everyone else sits in the 75–90% band, while DeepSeek alone reaches 98%. Running DeepSeek and MiMo in production for three months, I measured that 90.4% of all tokens were cached input hits — cache dependence had formed that deeply. That experience is also where this article starts.
Where this differs from the original text
The model names mentioned in the body (Gemini 1.5 Flash, Claude 3.5 Haiku, GPT-4o-mini, etc.) reflect the state at that time. Providers update their cache billing rules alongside generational changes, so when building an actual budget it's always safer to check each provider's current pricing page directly. The table above is meant to show the relative scale of discounts; absolute unit prices change over time.
Source: each provider's official pricing and documentation pages (Anthropic Prompt Caching, OpenAI API Pricing, Google Cloud Context Caching, DeepSeek API Docs)
Comments (1)
The discount rates you gathered into the appendix table are sufficient evidence for the claim. The one thing I'd add is that "routing is poison" is a conditional conclusion, and that condition deserves a line.
98% and 90% are not the same number
You already flagged DeepSeek V4-Flash's 98% as exceptional in your own earlier post. What the table is missing is that there is an anchor cost attached.
A 98% cache hit discount does not mean spending drops by 98%. Here's the actual arithmetic. Assume a 6,000-token system prompt with eight calls per session:
That's a 59% reduction. The 98% rate applies only to the repeated portion. Fresh user input that changes every call is still billed at full price, and in a real agent session with roughly 500 new tokens per turn, that's where the math bites.
So there is a legitimate basis for the "90% savings" headline: the reduction rate scales with context reuse. If 90% of your prefixes are repeated, actual savings converge to the high 80s. A 90% cache discount and a 90% real reduction are different numbers, and keeping them apart is the point.
The routing-decision cost may be smaller than you think
The post says the cost of the classification call disappears. There's a counterexample: if you run the classifier on the cached mid-tier model, that classification call hits the cache itself, so its cost collapses to roughly the first call in the session.
So "no routing at all" isn't quite the framing. Separating the decision layer from the execution layer is more accurate — cached mid-tier classifies once, then the cached context does the actual work. And that holds for both batch and interactive use.
The variable missing from the table: TTL
The table has discount rates but no cache lifetime, and that gap matters.
If a user chats every 30 seconds, Anthropic's 5-minute TTL is plenty. But batch jobs or overnight crawlers processing one item every 10 minutes expire that cache every time, paying the write cost (1.25x) repeatedly. In that scenario the "anchor on mid-tier" strategy is actively a loss.
A provider like DeepSeek with automatic caching and free storage sidesteps this entirely. Which means the provider choice is itself part of the caching strategy.
Refined conclusion
I agree, with one refinement to the wording:
Adding a "caching method" column to the appendix was the right call. Two more worth adding: TTL, and cache anchor token size (how many tokens the system prompt plus tool definitions actually occupy). Without those two numbers you can't convert a 90% discount into a real reduction rate.
That said, the central argument holds. On caching, holding onto one thing beats fragmenting every request.