---
title: "GPU VRAM Allocation Structure and the KV Cache Bible: A Complete Breakdown of Real Usage by Model"
date: 2026-09-23
time: "17:30"
model: admin
category: knowhow
summary: "Why a local LLM suddenly slows down on 8GB of VRAM, what the KV cache is, and the real VRAM usage per model, laid out with benchmark figures."
tags: GPU,VRAM,KV cache,local LLM,quantization,Flash Attention,GQA,llama-server
---

## Bottom Line First

If you load a 6GB model on an 8GB GPU and your TPS (tokens per second) suddenly halves, the problem is not the model size but the **KV cache**. The KV cache grows exponentially as the context gets longer, and when this area exceeds the GPU limit it offloads to the CPU and the speed collapses. This piece analyzes the real VRAM usage of three models — Qwen3-4B, K2-Horizon-3.7B, and Gemma 4 E4B — with benchmark figures.

---

## 1. GPU VRAM Allocation Structure: Static vs Dynamic

```
+--------------------------------------------------------+
|                   Total VRAM (8 GB)                    |
+---------------------------+----------------------------+
|  Static area              |     Dynamic area           |
|  - Model weights (6 GB)   |     - KV cache + activation|
|                           |       (available: ~2 GB)   |
+---------------------------+----------------------------+
```

**Static allocation**: the fixed area taken up by the model's quantized weights. With Q4_K_M quantization, the file size roughly equals the VRAM footprint.

**Dynamic allocation**: the area where the activation tensors and KV cache live, changing in real time during context input and token generation. This is the core of the problem.

---

## 2. What the KV Cache Is

A transformer's self-attention has to compute the relationship with every previous token. Recomputing from scratch for every token would be inefficient, so the Key and Value tensors of past tokens are cached in memory.

**KV cache size formula (for a GQA model):**

```
KV_cache_bytes = 2 × L × H_kv × D_head × C × B
```

- `L`: number of layers
- `H_kv`: number of KV attention heads (fewer than query heads under GQA)
- `D_head`: head dimension
- `C`: context window (number of tokens currently processed)
- `B`: data precision (FP16 = 2 bytes, INT8 = 1 byte)

**Key point**: because a GQA model has `H_kv` smaller than `H_query`, its KV cache is 4x smaller than MHA (full attention) even at the same context.

---

## 3. Real VRAM Usage Benchmark by Model

### Qwen3-4B (36 layers, 32 query heads, 8 KV heads, GQA 4:1)

| Quantization | Weight file | Weight VRAM | +4K ctx | +8K ctx | +32K ctx | +41K (max) |
|--------|-----------|-----------|---------|---------|----------|------------|
| Q4_K_M | 2.41 GB | ~2.9 GB | ~3.2 GB | ~4.0 GB | ~5.7 GB | **6.5 GB** |
| Q6_K | 3.32 GB | ~3.8 GB | ~4.1 GB | ~4.9 GB | ~6.6 GB | ~7.4 GB |
| Q8_0 | 4.02 GB | ~4.5 GB | ~4.8 GB | ~5.6 GB | ~7.3 GB | ~8.1 GB |
| BF16 | 8.05 GB | ~8.5 GB | ~8.8 GB | ~9.6 GB | ~11.3 GB | ~12.1 GB |

On 8GB VRAM with Q4_K_M, headroom runs out from a 32K context (5.7GB), and at a 41K context it reaches 6.5GB, close to the threshold. Q6_K tightens from 8K (4.9GB) and 32K (6.6GB).

### K2-Horizon-3.7B (36 layers, 32 query heads, 8 KV heads, GQA 4:1)

| Quantization | Weight file | Weight VRAM | +4K ctx | +32K ctx | +128K ctx | +512K (max) |
|--------|-----------|-----------|---------|----------|-----------|------------|
| Q4_K_M | 3.03 GB | ~3.33 GB | ~3.5 GB | ~4.8 GB | ~12.7 GB | **~47 GB** |
| Q8_0 | 5.41 GB | ~5.95 GB | ~6.1 GB | ~7.4 GB | ~15.3 GB | ~49.5 GB |
| FP16 | 10.12 GB | ~11.13 GB | ~11.3 GB | ~12.6 GB | ~20.5 GB | ~54.6 GB |

K2-Horizon's native context is 512K tokens. At Q4_K_M, a 512K context puts the KV cache alone at 34GB. On an 8GB setup you must force the context down to 32K or less.

### Gemma 4 E4B (42 layers, 8 query heads, 2 KV heads, GQA 4:1, hybrid attention)

| Quantization | Weight file | Weight VRAM | +4K ctx | +8K ctx | +32K ctx | +131K (max) |
|--------|-----------|-----------|---------|---------|----------|------------|
| Q4_K_M | ~2.8 GB | ~3.0 GB | ~3.1 GB | ~3.2 GB | ~3.5 GB | ~4.5 GB |
| Q6_K | ~3.6 GB | ~3.8 GB | ~3.9 GB | ~4.0 GB | ~4.3 GB | ~5.3 GB |
| Q8_0 | ~4.5 GB | ~4.7 GB | ~4.8 GB | ~4.9 GB | ~5.2 GB | ~6.2 GB |
| BF16 | ~8.5 GB | ~10 GB | ~10.1 GB | ~10.2 GB | ~10.5 GB | ~11.5 GB |

Gemma 4 E4B uses a hybrid attention structure (36 sliding-window layers + 6 global layers). The sliding-window layers reuse KV across the window, so the real KV cache is smaller than the formula suggests. On 8GB it runs a 131K context stably at 4.5GB (Q4_K_M).

---

## 4. The Mechanism by Which GQA Cuts the KV Cache 4x

| Model | Query Heads | KV Heads | GQA ratio | KV cache saving vs MHA |
|------|------------|----------|---------|---------------------|
| Qwen3-4B | 32 | 8 | 4:1 | **4x** |
| K2-Horizon-3.7B | 32 | 8 | 4:1 | **4x** |
| Gemma 4 E4B | 8 | 2 | 4:1 | **4x** |

Without GQA (full MHA, 32 KV heads):
- Qwen3-4B at 41K context: KV cache 14.4GB (currently 3.6GB)
- K2-Horizon at 512K context: KV cache 136GB (currently 34GB)

GQA is the key technology that makes long-context inference realistic on consumer GPUs.

---

## 5. CPU Offloading and the Bottleneck

**Bandwidth difference**:
- GPU VRAM bandwidth: 300-400 GB/s for GDDR6
- PCIe Gen4 x16: 32 GB/s (about 10x slower than the GPU)

If the weights and KV cache cannot both fit in GPU VRAM, the overflow offloads to CPU RAM. Tokens are then exchanged over the PCIe bus, and TPS plunges.

**Concrete example** (Qwen3-4B Q8_0, 8GB VRAM):
- 8K context: 5.6GB used -> runs entirely on GPU -> TPS 45-60
- 32K context: 7.3GB used -> near the threshold -> TPS 35-45
- 41K context: 8.1GB used -> 100MB offloaded -> TPS 25-30 (about 40% drop)

---

## 6. Optimization Options to Overcome Hardware Limits

### Option 1: Limit the context window

```bash
llama-server -m model.gguf --ctx-size 8192
```

Safe context ceilings on an 8GB setup:
- Qwen3-4B Q4_K_M: 32K (5.7GB)
- K2-Horizon Q4_K_M: 32K (4.8GB)
- Gemma 4 E4B Q4_K_M: 131K (4.5GB)

### Option 2: Enable Flash Attention

```bash
llama-server -m model.gguf --flash-attn --ctx-size 32768
```

Flash Attention computes the attention matrix tile by tile instead of fully materializing it, cutting KV cache memory by 30-50%.

| Context | Without Flash Attention | With Flash Attention | Saving |
|---------|---------------------|---------------------|--------|
| 8K | ~2 GB | ~1.2 GB | 800 MB |
| 32K | ~8 GB | ~4.5 GB | 3.5 GB |
| 128K | ~32 GB | ~18 GB | 14 GB |

### Option 3: KV cache quantization

```bash
llama-server -m model.gguf \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --ctx-size 32768
```

Storing the KV cache as Q8_0 instead of FP16 halves the memory. Quality loss is under 1%.

| Setting | 32K context KV cache | 128K context |
|------|---------------------|-------------|
| Default (FP16 KV) | ~8 GB | ~32 GB |
| Q8_0 KV cache | ~4 GB | ~16 GB |
| Flash Attention + Q8_0 | **~2.25 GB** | **~9 GB** |

### Option 4: Flash Attention + KV quantization (the optimal combination)

```bash
llama-server -m model.gguf \
  --n-gpu-layers 999 \
  --flash-attn \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --ctx-size 131072
```

Applying Flash Attention and Q8_0 KV cache together cuts context VRAM by **70-80%** versus the default. A 128K context becomes realistic even on a 24GB GPU.

---

## 7. Practical Deployment Recommendations by Model

### 8GB VRAM (RTX 3060, RTX 4060 Ti, etc.)

| Model | Quantization | Recommended context | Expected TPS | Note |
|------|--------|-------------|---------|------|
| Qwen3-4B | Q4_K_M | 8K-16K | 50-70 | Best value, excellent Korean handling |
| K2-Horizon | Q4_K_M | 4K-8K | 55-75 | Coding-specialized, context limit required |
| Gemma 4 E4B | Q4_K_M | 32K-64K | 40-60 | Long context possible via hybrid attention |

### 16GB VRAM (RTX 4080, RTX 5070, etc.)

| Model | Quantization | Recommended context | Expected TPS | Note |
|------|--------|-------------|---------|------|
| Qwen3-4B | Q8_0 | 32K-41K | 40-55 | High-precision inference |
| K2-Horizon | Q6_K | 16K-32K | 45-65 | Optimal balance |
| Gemma 4 E4B | Q8_0 | 131K | 35-50 | Long-document analysis possible |

### 24GB VRAM (RTX 3090, RTX 4090, RTX 5090, etc.)

| Model | Quantization | Recommended context | Expected TPS | Note |
|------|--------|-------------|---------|------|
| Qwen3-4B | BF16 | 41K | 35-50 | Original-precision inference |
| K2-Horizon | Q8_0 | 64K-128K | 30-45 | Flash Attention required |
| Gemma 4 E4B | BF16 | 131K | 30-45 | Full context utilization |

---

## 8. Key Summary

1. **The KV cache grows in proportion to context length**: multiplying the context 8x from 4K to 32K increases the KV cache about 8x too.
2. **GQA is essential**: Qwen3-4B, K2-Horizon, and Gemma 4 E4B all use GQA 4:1 to cut the KV cache 4x.
3. **Flash Attention + KV quantization = the optimal combination**: cutting context VRAM by 70-80% makes long-context inference possible even on a consumer GPU.
4. **On an 8GB setup it is safe to limit the context to 8K-16K** at Q4_K_M quantization.
5. **Avoid CPU offloading**: the PCIe bandwidth bottleneck drops TPS by more than 40%.