AI Knowledge HubAI & Models
KV cache
When a model writes a sentence token by token, recomputing earlier text every time wastes work. The KV cache stores those intermediate results and reuses them for the next token. Hits make responses fast; very long conversations eventually exceed the cache budget. This is a different mechanism from prompt caching.
A 'KV cache exceeded' error means the conversation outgrew its cache budget.
Why it mattersThis is why long conversations suddenly get expensive or run out of room. It answers 'why does keeping the thread going burn tokens so fast?'
ExampleWhen a model writes a sentence token by token, recomputing earlier text every time wastes work. The KV cache stores those intermediate results and reuses them for the next token. Hits make responses fast; very long conversations eventually exceed the cache budget. This is a different mechanism from prompt caching.