GPU VRAM Allocation Structure and the KV Cache Bible: A Complete Breakdown of Real Usage by Model

Bottom Line First

If you load a 6GB model on an 8GB GPU and your TPS (tokens per second) suddenly halves, the problem is not the model size but the KV cache. The KV cache grows exponentially as the context gets longer, and when this area exceeds the GPU limit it offloads to the CPU and the speed collapses. This piece analyzes the real VRAM usage of three models — Qwen3-4B, K2-Horizon-3.7B, and Gemma 4 E4B — with benchmark figures.

1. GPU VRAM Allocation Structure: Static vs Dynamic

+--------------------------------------------------------+
|                   Total VRAM (8 GB)                    |
+---------------------------+----------------------------+
|  Static area              |     Dynamic area           |
|  - Model weights (6 GB)   |     - KV cache + activation|
|                           |       (available: ~2 GB)   |
+---------------------------+----------------------------+

Static allocation: the fixed area taken up by the model's quantized weights. With Q4_K_M quantization, the file size roughly equals the VRAM footprint.

Dynamic allocation: the area where the activation tensors and KV cache live, changing in real time during context input and token generation. This is the core of the problem.

2. What the KV Cache Is

A transformer's self-attention has to compute the relationship with every previous token. Recomputing from scratch for every token would be inefficient, so the Key and Value tensors of past tokens are cached in memory.

KV cache size formula (for a GQA model):

KV_cache_bytes = 2 × L × H_kv × D_head × C × B

• L: number of layers
• H_kv: number of KV attention heads (fewer than query heads under GQA)
• D_head: head dimension
• C: context window (number of tokens currently processed)
• B: data precision (FP16 = 2 bytes, INT8 = 1 byte)

Key point: because a GQA model has H_kv smaller than H_query, its KV cache is 4x smaller than MHA (full attention) even at the same context.

3. Real VRAM Usage Benchmark by Model

Qwen3-4B (36 layers, 32 query heads, 8 KV heads, GQA 4:1)

On 8GB VRAM with Q4_K_M, headroom runs out from a 32K context (5.7GB), and at a 41K context it reaches 6.5GB, close to the threshold. Q6_K tightens from 8K (4.9GB) and 32K (6.6GB).

K2-Horizon-3.7B (36 layers, 32 query heads, 8 KV heads, GQA 4:1)

K2-Horizon's native context is 512K tokens. At Q4_K_M, a 512K context puts the KV cache alone at 34GB. On an 8GB setup you must force the context down to 32K or less.

Gemma 4 E4B (42 layers, 8 query heads, 2 KV heads, GQA 4:1, hybrid attention)

Gemma 4 E4B uses a hybrid attention structure (36 sliding-window layers + 6 global layers). The sliding-window layers reuse KV across the window, so the real KV cache is smaller than the formula suggests. On 8GB it runs a 131K context stably at 4.5GB (Q4_K_M).

4. The Mechanism by Which GQA Cuts the KV Cache 4x

Without GQA (full MHA, 32 KV heads):

• Qwen3-4B at 41K context: KV cache 14.4GB (currently 3.6GB)
• K2-Horizon at 512K context: KV cache 136GB (currently 34GB)

GQA is the key technology that makes long-context inference realistic on consumer GPUs.

5. CPU Offloading and the Bottleneck

Bandwidth difference:

• GPU VRAM bandwidth: 300-400 GB/s for GDDR6
• PCIe Gen4 x16: 32 GB/s (about 10x slower than the GPU)

If the weights and KV cache cannot both fit in GPU VRAM, the overflow offloads to CPU RAM. Tokens are then exchanged over the PCIe bus, and TPS plunges.

Concrete example (Qwen3-4B Q8_0, 8GB VRAM):

• 8K context: 5.6GB used -> runs entirely on GPU -> TPS 45-60
• 32K context: 7.3GB used -> near the threshold -> TPS 35-45
• 41K context: 8.1GB used -> 100MB offloaded -> TPS 25-30 (about 40% drop)

6. Optimization Options to Overcome Hardware Limits

Option 1: Limit the context window

llama-server -m model.gguf --ctx-size 8192

Safe context ceilings on an 8GB setup:

• Qwen3-4B Q4_K_M: 32K (5.7GB)
• K2-Horizon Q4_K_M: 32K (4.8GB)
• Gemma 4 E4B Q4_K_M: 131K (4.5GB)

Option 2: Enable Flash Attention

llama-server -m model.gguf --flash-attn --ctx-size 32768

Flash Attention computes the attention matrix tile by tile instead of fully materializing it, cutting KV cache memory by 30-50%.

Option 3: KV cache quantization

llama-server -m model.gguf \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --ctx-size 32768

Storing the KV cache as Q8_0 instead of FP16 halves the memory. Quality loss is under 1%.

Option 4: Flash Attention + KV quantization (the optimal combination)

llama-server -m model.gguf \
  --n-gpu-layers 999 \
  --flash-attn \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --ctx-size 131072

Applying Flash Attention and Q8_0 KV cache together cuts context VRAM by 70-80% versus the default. A 128K context becomes realistic even on a 24GB GPU.

7. Practical Deployment Recommendations by Model

8GB VRAM (RTX 3060, RTX 4060 Ti, etc.)

16GB VRAM (RTX 4080, RTX 5070, etc.)

24GB VRAM (RTX 3090, RTX 4090, RTX 5090, etc.)

8. Key Summary

1. The KV cache grows in proportion to context length: multiplying the context 8x from 4K to 32K increases the KV cache about 8x too.
2. GQA is essential: Qwen3-4B, K2-Horizon, and Gemma 4 E4B all use GQA 4:1 to cut the KV cache 4x.
3. Flash Attention + KV quantization = the optimal combination: cutting context VRAM by 70-80% makes long-context inference possible even on a consumer GPU.
4. On an 8GB setup it is safe to limit the context to 8K-16K at Q4_K_M quantization.
5. Avoid CPU offloading: the PCIe bandwidth bottleneck drops TPS by more than 40%.