---
title: "\"Frequent Routing Is Poison\" — Why a Mid-Tier Model Plus Context Caching Saves 90% of API Costs
date: 2026-10-15
time: 09:20
model: admin
category: knowhow
author_type: human
summary: "Why routing simple tasks to small models and hard tasks to large models actually breaks prompt caching and increases cost. A structural explanation with provider-by-provider cache discount evidence and a practical hybrid escalation architecture."
tags: prompt caching, AI agents, API costs, LLM routing, token optimization, cost optimization
---

# "Frequent Routing Is Poison" — Why a Mid-Tier Model Plus Context Caching Saves 90% of API Costs

There's a familiar cost-optimization trap when building commercial AI agents.

"Assign simple tasks to ultra-cheap, small API models and switch to expensive large models like Claude 3.5 Sonnet or GPT-4o only for complex work — that should cut API costs dramatically?"

On paper it sounds right. But any developer who understands the billing mechanics of prompt caching in paid APIs knows this approach actually detonates the bill.

The conclusion drawn from building and operating paid API pipelines is unambiguous: "a single-hub architecture built around a mid-tier paid model that maximizes cache hits (Gemini Flash, Claude Haiku, GPT-4o-mini) wins decisively on cost-performance, speed, and cost alike."

Here's why anchoring on a mid-tier paid model beats frequent model switching, structurally, and what the practical architecture looks like.

## 1. Why Frequent Model Switching Destroys Cache Hits

The prompt caching discounts offered by major paid API platforms — Anthropic, OpenAI, Google Gemini — only apply when the same system prompt and context sequence arrive consecutively on the same model endpoint.

When an agent routes back and forth between small and large models based on fragmentary judgment, the following happens:

Destruction of the API server's KV cache: switching model endpoints means the previously generated cache can't be used, and each API server recomputes the entire prompt from scratch (token processing).

Cost and latency moving backwards: you try to save a few cents on a trivial small-model call, and then both models hit a prompt cache miss. In the end you pay full price for the entire context — tens of thousands of tokens — every single time, so total billing goes up and everything gets slower.

## 2. The Strong Advantages of a Mid-Tier-Anchored Architecture

The structure I propose is this: "keep a mid-tier paid API model with proven cost-performance and speed (e.g. Gemini 1.5 Flash, Claude 3.5 Haiku, GPT-4o-mini) resident as the main hub, and call a top-tier model (Claude 3.5 Sonnet, GPT-4o, etc.) only in extremely exceptional cases."

In terms of paid API cost, this structure is a formidable weapon.

Long-context cache residency (up to 80–90% discount on input tokens)

The agent's entire workflow, prior conversation context, project codebase, and fixed system prompt become resident (hit) in the mid-tier model's cache. Across major providers, cached tokens are discounted 75–90% versus regular input tokens, effectively running at a token rate "close to free."

Complete elimination of routing overhead

The cost and latency of the additional classification (routing) API call needed to judge "is this request easy or hard?" disappear entirely.

Task continuity and context preservation

The agent can continue its work steadily and precisely within a single session without losing context midway.

## 3. Practical Hybrid API Escalation Strategy

The recommended two-stage escalation pipeline for commercial API environments.

[Base Layer]
90% of the entire workflow & API tool executions
→ handled exclusively by a mid-tier paid model (Gemini Flash / Claude Haiku / GPT-4o-mini, etc.)
  (maximize prompt cache hits within the same model session)
       │
       ▼ (error / exception occurs?)
[Escalation Trigger]
repeated code syntax errors, complex JSON parsing failures, internal judgment module activation
       │
       ▼
[Top-tier Fallback]
hand the entire context to a top-tier API model (Claude 3.5 Sonnet, GPT-4o, etc.),
resolve it in one shot, then return
Base Layer (Base Layer): 90% or more of routine work and API tool calls are handled by the mid-tier model, continuously receiving prompt cache discount benefits.

Fallback/Escalation Trigger (Escalation): only when the mid-tier model repeats syntax errors or fails at complex JSON parsing — clear exception conditions — is the context handed to a flagship API model (Claude 3.5 Sonnet, etc.), resolved in one shot, and then brought back down.

Wrapping up: getting the cache discount matters more than splitting everything up
The longer you operate agents on paid APIs, the more you feel the truth in the statement that "rather than carelessly fragmenting into several cheap small models, a single mid-tier API model holding tightly onto the prompt cache discount dominates on cost-performance, speed, and stability."

---

## Appendix. Evidence — Cache Hit Discount Rates by Provider

Here are the actual billing rules behind the claims above. Discounts land mostly in the 75–90% range, but caching mechanism and minimum token conditions vary considerably.

| Model | Regular input (per 1M) | Cache hit (per 1M) | Discount | Caching method | Notes |
|---|---:|---:|---:|---|---|
| DeepSeek V4-Flash | $0.14 | $0.0028 | **98%** | Automatic (identical prefix) | Free storage; partial matches miss |
| Claude Haiku 4.5 / Sonnet 5 | 1x | 0.1x | **90%** | Explicit breakpoints | Write 1.25x (5 min) / 2x (1 hour) |
| GPT-6 Luna | $0.10 | $0.01 | **90%** | Automatic (prefix) | Write 1.25x |
| GPT-5.5 | $5.00 | $0.50 | **90%** | Automatic (prefix) | Stable prefix ≥1,024 tokens |
| Gemini 2.5 Flash | 1x | 0.1x | **90%** | Implicit caching | Minimum 1,024 tokens |
| Gemini 2.0 (explicit) | 1x | 0.25x | **75%** | Explicit caching | Storage billed separately |

### Two things to read off this table

**First, caching mechanisms differ everywhere.** Anthropic requires you to set explicit breakpoints yourself. OpenAI and DeepSeek hit automatically when the prefix matches. Gemini has implicit caching on by default, but with minimum token conditions. In other words, the assumption that "the same prompt input will stick everywhere identically" is wrong.

**Second, DeepSeek's 98% is exceptional.** Everyone else sits in the 75–90% band, while DeepSeek alone reaches 98%. Running DeepSeek and MiMo in production for three months, I measured that 90.4% of all tokens were cached input hits — cache dependence had formed that deeply. That experience is also where this article starts.

### Where this differs from the original text

The model names mentioned in the body (Gemini 1.5 Flash, Claude 3.5 Haiku, GPT-4o-mini, etc.) reflect the state at that time. Providers update their cache billing rules alongside generational changes, so when building an actual budget it's always safer to check each provider's current pricing page directly. The table above is meant to show the **relative scale** of discounts; absolute unit prices change over time.

Source: each provider's official pricing and documentation pages (Anthropic Prompt Caching, OpenAI API Pricing, Google Cloud Context Caching, DeepSeek API Docs)