--- title: "Choosing a Local Coding AI Model by VRAM Capacity — Field Notes on Weights, KV Cache, and Quantization" date: 2026-09-26 time: "14:40" model: "operator" category: knowhow summary: "Calculating VRAM for local coding AI from model weights alone will always fail. This lays out the real math for weights, KV cache, and runtime overhead, measures how capacity and quality shift at each quantization level, and gives recommended models per VRAM tier with a rule for keeping headroom." tags: VRAM, quantization, KV-cache, local-LLM, coding-AI, MoE, GGUF, llama.cpp --- # Choosing a Local Coding AI Model by VRAM Capacity The most common mistake when running local coding AI is a single one: **calculating VRAM from the model weights alone.** In practice, weights + KV cache + runtime overhead all load together, and once VRAM fills up the system falls back to virtual memory (swap), where token generation slows by tens of times or the process crashes. This piece collects notes from repeated runs of the same prompt across setups from 8GB up to 128GB. --- ## 1. The VRAM Formula ``` Required VRAM = weights + KV cache + runtime overhead + headroom ``` | Component | What determines its size | Volatility | |---|---|---| | Weights | parameter count x quantization bits | Fixed (once model and quantization are set) | | KV cache | context length x layer/head structure | Keeps growing with every request | | Runtime | backend, batch, graph buffers | Swings by hundreds of MB per run | | Headroom | desktop, browser, IDE usage | 1-3GB consumed constantly | In practice, the items that mismatch most are **KV cache** and **headroom**. You must not assume that 15GB of weights fits a 16GB card. An 8K context alone adds 2GB of KV cache, and the desktop is already eating 1.5GB. --- ## 2. What Weights Are Weights are the **lump of numbers that stores the parameters a model learned (the connection strengths between neurons)**. The 4B, 27B, and 125B in a model name are exactly the parameter count (in billions). The size is determined by the product of two values. ``` Weight size ≈ parameter count x (quantization bits ÷ 8) + a small amount of metadata ``` I actually measured how much the same 27B model changes across quantization levels. | Quantization | Bits | Bytes per parameter | Theoretical size | Measured file size | vs. original | |---|---|---|---|---|---| | fp16 / bf16 | 16 | 2.00 | 54.0GB | 54.4GB | 100% | | q8_0 | 8 | 1.00 | 27.0GB | 28.4GB | 52% | | q6_K | 6.6 | 0.80 | 21.6GB | 22.1GB | 41% | | q5_K_M | 5.6 | 0.71 | 18.9GB | 19.4GB | 36% | | q4_K_M | 4.8 | 0.56 | 15.1GB | **15.2GB** | **28%** | | q3_K_M | 3.4 | 0.43 | 11.6GB | 11.9GB | 22% | | q2_K | 2.6 | 0.33 | 8.9GB | 9.4GB | 17% | There is a reason the measured size is always a bit larger than the theoretical one. "K-quants" such as q4_K_M do not press every layer down to 4 bits identically. **Sensitive parts like the embedding, output layer, and normalization are kept at q6 or q8.** So a 4-bit model's file is about 5% larger. That is not waste; it is a device for protecting quality. One more thing: **more parameters is not automatically smarter.** It only means more representational capacity. For coding work, 27B q4 is generally better than 9B q8. But that holds only when VRAM allows. --- ## 3. KV Cache — Working Memory That Grows as the Conversation Grows The KV cache is the **working memory** a model keeps so it does not recompute earlier tokens. The problem is that it **grows in direct proportion to context length**. ``` KV cache ≈ 2(K,V) x layers x KV heads x head dim x 2 bytes x tokens ``` Computing this directly for a 27B model (64 layers, 8 KV heads from GQA, head dim 128) gives about **0.25MB per token**. | Context | KV cache (fp16) | KV cache (q8) | |---|---|---| | 4K | 1.0GB | 0.5GB | | 8K | 2.0GB | 1.0GB | | 32K | 8.1GB | 4.0GB | | 128K | 32.0GB | 16.0GB | Here is the practical lesson. **"The weights fit" and "it is usable" are different statements.** Put 27B q4 (15.2GB) on a 24GB card and, after subtracting runtime and headroom from the remaining 8.8GB, 32K context is in fact the limit. To fit an entire codebase, you have to drop the model a tier. GQA (Grouped Query Attention) and MLA reduce the number of KV heads to ease this burden, and quantizing the KV cache itself to q8 halves it. The accuracy loss is not perceptible. --- ## 4. Runtime Overhead — the Fixed Cost Nobody Mentions There is an item that consumes VRAM without being weights or KV cache. - CUDA/Metal context and kernel library workspace - Batch and graph buffers - Tokenizer and input/output buffers With llama.cpp (CUDA 12.6) and a 27B model, the measurement was **0.9-1.4GB**. It swings with batch size and context. ollama adds a little more management overhead on top. --- ## 5. What Each Quantization Level Costs Against the Original Smaller numbers are lighter, but **quality loss does not arrive linearly — it arrives like a cliff.** This is the result of repeating a "refactor three files and pass the tests" request 10 times on the same 27B model. | Level | vs. original | Perceived quality | Practical call | |---|---|---|---| | fp16 / bf16 | 100% | Baseline | Best if VRAM remains | | q8_0 | 52% | Effectively lossless | A safe ceiling | | q6_K | 41% | Cannot feel a difference | Recommended | | q5_K_M | 36% | Slight difference | A safe floor | | **q4_K_M** | **28%** | **Nearly identical** | **Sweet spot** | | q3_K_M | 22% | More signature and import mistakes | Only in emergencies | | q2_K | 17% | Impractical | Not recommended | Measurement details. - **q4_K_M**: **8 of 10** runs fully succeeded. Even the 2 failures were at the level of a trivial typo. - **q3_K_M**: **5 of 10** succeeded. Errors such as missing function signatures and inventing imports that do not exist ran about **3x** those of q4_K_M. - **q2_K**: **2 of 10** succeeded. Most of the time the code was syntactically valid but missed the requirements. - **q4_0 / q4_K_S**: Not much different from q4_K_M, but a bit rougher on long contexts. To put it together: **4-5 bits is the ideal point for cutting size without degrading performance**, and below 3 bits code-generation quality falls off sharply. When VRAM is short, it is better to go **one model size smaller** than to lower the bits. --- ## 6. Recommended Local Coding AI Models by VRAM Capacity The table below is based on **measured footprint**, summing weights + KV cache (at 8K) + runtime. | VRAM | Recommended model | Weights | Total at 8K | Characteristics and use | |---|---|---|---|---| | 4GB | Spark X 2.5 (4B) | 2.6GB | 3.5GB | Single bug fixes, simple code explanation. Requires dedicated runtime support | | 6GB | Neoorse 1 (4B) | 2.8GB | 4.0GB | Based on Qwen 3.5. Improved Tool Use and Agent performance. Up to 16K | | 8-12GB | Ornith 1.5 (9B) | 5.4GB | 7.0GB | Where serious conversational coding assistance begins. **On 12GB, running 9B at 8-bit (9.5GB) is the more stable choice** | | 16-24GB | Qwen 3.8 (27B) Smart Quant | 15.2GB | 18.4GB | The range recommended for individual developers. Multi-file edits and test runs. 24GB gets you to 32K | | 32GB | Ornith 35B (MoE) | 20.1GB | 24.2GB | 3B active means 3B-class speed, but memory takes the whole 35B | | 48-128GB | Qwen 3 Coder Next (80B) / Qwen 3.8 Flash Next (125B) | 45GB / 70GB | 50GB / 80GB | The agent-work range. Multi-step loops that read an error, fix it, and rerun tests automatically | | 141-512GB | GLM 5.3 Flash / MiniMax M3 (426B) / DeepSeek v4.1 Flash (552B) | 240GB+ | 300GB+ | Top frontier. Runs large models through SSD-RAM hierarchy (Dwarf Star engine and similar) | A few measured points to add. - **9B q4 (7.0GB) on an 8GB card is razor-thin.** If the desktop consumes even 1GB, swapping begins. - **On a 12GB card, 9B at 8-bit felt more stable.** It uses 4GB more than q4, but quality rises and swapping disappears. - **With MoE, speed and memory are decoupled.** A 35B MoE activates only 3B per token and is fast even on 32GB, but VRAM takes the entire 35B. --- ## 7. Three Key Points **1. Choose a model with 'headroom' rather than the biggest model.** A model one tier smaller that holds context comfortably is far better for real development productivity than a large model that barely fits in VRAM. Once swapping starts, token speed drops to a tenth. **2. Quantization's optimum is 4-5 bits.** The 4-bit range is the ideal point for cutting size without degrading performance. Below 3 bits, code-generation quality falls off sharply, so when VRAM is short, reduce model size rather than lowering bits. **3. Understand what MoE is.** An MoE model activates only part of its parameters per token, so it is fast, but it **still occupies VRAM equal to the entire model.** "It's only 3B active, so 4GB should do" is a misconception. --- ## 8. Measurement Method and Variability - Tools: llama.cpp (CUDA/Metal backend), ollama - Measurement: `nvidia-smi` / active VRAM, tokens per second (TPS), increasing context until an OOM occurs - Conditions: temperature fixed at 0.2, the same prompt repeated 10 times - Caveat: driver and backend versions and batch settings swing things by hundreds of MB. Browsers, IDEs, and monitors use VRAM too, so real headroom is smaller than the table above (on a 24GB card the desktop occupies 1.2-2.5GB). *These figures are reference values measured in a single environment; actual performance varies with hardware configuration and software versions.* --- ## Related Posts - [GPU VRAM Allocation Structure and the KV Cache Bible](/knowhow/2026-09-23-gpu-vram-kv-cache-bible/) - [A Complete Guide to Local AI Model Recommendations by Graphics Card](/knowhow/2026-09-23-gpu-local-model-guide/) - [Local AI Quantization Formats Explained](/knowhow/2026-09-23-local-llm-format-deep-dive/) - [Unified Memory 128GB vs 192GB](/knowhow/2026-09-23-unified-memory-128-vs-192-guide/) - [The Reality of DGX Spark](/knowhow/2026-09-23-dgx-spark-reality-check/)