---
title: "The Complete RAG Pipeline Guide — Eight Practical Techniques That Maximize Retrieval Quality"
date: 2026-09-23
time: "21:00"
model: "operator"
category: knowhow
summary: "RAG is not just vector search. From chunking quality, hybrid search, rerankers, and contextual retrieval to late chunking — this lays out why retrieval fails in practice and how to fix it."
tags: "RAG, vector search, embeddings, chunking, hybrid search, reranker, LLM"
---

# The Complete RAG Pipeline Guide — Eight Practical Techniques That Maximize Retrieval Quality

> "Most of the reasons RAG fails come not from generation (Gen) but from the quality of chunking."

These days the phrase "RAG is Dead" is trendy, but most of the data handled in practice is not code but **unstructured text (contracts, wikis, customer support records, and so on)**. For that kind of data, RAG is still essential. The question is **how you implement it**.

---

## 1. The Basic Structure of a RAG Pipeline

```
Document → Chunking → Embedding → Store in vector DB
                                                    ↓
User question → question embedding → similarity search → extract relevant chunks → pass to LLM → generate answer
```

### 1-1. Chunking: Cutting a 100-Page Document into Pieces

**Why is it needed?**
- Putting a 100-page document into an LLM as-is exceeds the context window
- It must be cut into an appropriate size (usually 300-1,000 tokens) to be retrievable

**Practical pitfalls of chunking:**

| Data type | Difficulty | Note |
|------------|--------|------|
| Plain text | Easy | Cutting by sentence/paragraph is enough |
| Includes tables | Hard | If the table structure breaks, meaning is lost |
| Includes images/graphs | Very hard | Text alone loses information |
| Code documentation | Medium | Cutting by function/class unit is key |

> **Most real-world RAG failures come from "chunking quality being garbage."**

### 1-2. Embedding: Turning Text into Numbers

**Analogy:** It is like similar products in a mart (frozen dumplings and frozen gyoza) being placed on similar shelves. Words with similar meanings are placed close together in numerical space.

**Key point:** Even if the exact word is not included, it can find documents that **match by "meaning."** This is semantic search.

| Embedding model | Dimensions | Characteristics |
|------------|------|------|
| OpenAI text-embedding-3-small | 1,536 | Cheap and stable |
| OpenAI text-embedding-3-large | 3,072 | High accuracy |
| BGE-M3 | 1,024 | Multilingual (excellent for Korean) |
| Cohere Embed v3 | 1,024 | Search-specialized |
| nomic-embed-text | 768 | Runs locally (Ollama) |

### 1-3. Vector DB: A Store Dedicated to Similarity Search

| DB | Characteristics | Recommended for |
|----|------|------|
| **ChromaDB** | Lightweight and simple | Prototypes, small scale |
| **Pinecone** | Managed, fast | Production |
| **Qdrant** | Open source, high performance | Self-hosting |
| **Weaviate** | Graph + vector hybrid | Complex relational data |
| **Milvus** | Large-scale, distributed | Enterprise |

---

## 2. Eight Practical Techniques That Maximize Retrieval Quality

### Technique 1: Hybrid Search

**Problem:** Semantic search alone cannot find technical terms.

**Solution:** Run both kinds of search at once and merge the results.

| Search type | Description | Example |
|-----------|------|------|
| **Dense** | Meaning/context-based search | "damages" → can find "indemnity clause" |
| **Sparse** | Exact keyword matching (BM25) | Exact match for "CBD Chamber" |

**Why do you need both?**
In specialized domains such as finance and manufacturing (semiconductors), the **exact match** of special abbreviations like "CBD chamber" or "BWG" often matters more than semantic search.

> In practice, the standard is to run both searches at once and **fuse (hybridize)** the results.

### Technique 2: Reranker

**Analogy:** The first-stage search is the step that picks 10 candidates by "age, job, region." The reranker lays those 10 profiles side by side with the question and re-evaluates, like a second-round interview, **"does this really contain the answer to this question?"**

| Stage | Role | Speed |
|------|------|------|
| First-stage search (vector) | Broad candidate retrieval | Fast |
| **Reranker** | Precise reordering | Slow |
| Final results | Pass top N | - |

**Practical trade-offs:**
- Accuracy: definitely goes up
- Response speed: gets slower (extra computation)
- Personal tools that value reliability: essential
- Hundreds of concurrent commercial users: needs careful design

### Technique 3: Contextual Retrieval

**Problem:** When a 100-page financial report is cut up, if some chunk is left with only "revenue grew 3% versus the previous quarter," the context of **which company and when** disappears.

**Solution (announced by Anthropic, September 2024):**
Before cutting the chunks, use an LLM to automatically prepend a short context like **"this chunk is part of company XX's Q2 2024 report"** to each chunk.

**Effect:**
- Retrieval failure rate reduced by **up to 67%**

**Cost-saving tip:**
Calling an LLM on every chunk is expensive, so running a **binary filter** first ("does this chunk contain meaningful information?") can cut costs dramatically.

### Technique 4: Late Chunking

**Existing approach:** Cut the document first → embed each piece independently (surrounding context is lost)

**Late Chunking:**
1. Feed the entire document as-is into a long-context embedding model
2. Record the overall connectivity as numbers
3. Split into chunks at the end

> **Analogy:** Rather than cutting a movie scene by scene and summarizing each, it is the approach of **"watching the whole movie from start to finish once, then summarizing scene by scene."** The surrounding context is cleanly preserved.

### Technique 5: Tuning the Right Chunk Size

| Chunk size | Pros | Cons |
|-----------|------|------|
| **Small (200-500 tokens)** | Higher retrieval precision | Risk of broken context |
| **Medium (500-1,000 tokens)** | Balanced choice | Suitable for most practical use |
| **Large (1,000-2,000 tokens)** | Better context preservation | Lower retrieval precision |

**Practical recommendation:** 500-1,000 tokens. Leaving **20-50% overlap** between chunks reduces context breaks.

### Technique 6: Metadata Filtering

**Pre-filtering by metadata** before retrieval narrows the search scope and improves both accuracy and speed at once.

```
Search query: "Q2 2024 revenue"
Filter: { "year": 2024, "quarter": "Q2", "type": "finance" }
```

| Metadata | Example |
|-----------|------|
| Date | 2024-01-01 ~ 2024-06-30 |
| Category | Finance, tech, marketing |
| Source | Wiki, contract, report |
| Language | Korean, English |

### Technique 7: Query Expansion

Have an LLM expand the user's short question into several forms and then search.

```
Original question: "What are RAG's downsides?"
Expanded questions:
1. "What are RAG's limitations and problems?"
2. "In what cases does RAG fail?"
3. "Downsides and cautions of Retrieval Augmented Generation"
```

> Searching with **3-5 variant questions** rather than a single question greatly improves search recall.

### Technique 8: Debugging — Inspect Ranks 1-20 Yourself

**The first thing to do when RAG is not working:**

1. Collect 5-10 questions that are frequently used in the real setting
2. **Visually inspect the chunks ranked 1st-20th** returned by vector search
3. Figure out "why is this data ranked 3rd?" and "why is the needed data outside the ranking?"

> **The first step of RAG optimization (context engineering) is this debugging.**

---

## 3. The Real Core of the "RAG is Dead" Debate

### Why is RAG "dead" in the world of code?

- Source code is **structured data** (function names and file paths are exact)
- Finding a specific string directly with `grep` is **more accurate and safer** than vector search
- Coding agents like Cursor and Claude Code replace it with Agentic Search

### Why is RAG alive in the business world?

Most of the data handled in practice:
- Customer support records
- Internal wikis
- Contracts
- Marketing reports

For such **unstructured text**, semantic search (= the domain of RAG) — which finds **semantically connected words** like "indemnity clause" or "penalty" even when the word "damages" is absent — is essential.

---

## 4. Practical RAG Building Checklist

```
□ Chunking: check whether documents with tables/images need manual preprocessing
□ Appropriate chunk size (500-1,000 tokens) + overlap (20-50%)
□ Enable hybrid search (Dense + Sparse/BM25)
□ Decide whether to apply a reranker (accuracy vs speed trade-off)
□ Configure metadata filtering (date, category, source)
□ Apply contextual retrieval (automatically prepend context to chunks)
□ Debug: visually inspect ranks 1-20 with 5-10 real questions
□ Monitor: continuously track retrieval failure rate and response quality
```

---

## Summary: RAG Pipeline Maturity Check

| Level | Composition | Retrieval quality |
|------|------|----------|
| **Beginner** | Basic vector search + simple chunking | 60% |
| **Intermediate** | Hybrid search + metadata filter | 80% |
| **Advanced** | + reranker + contextual retrieval | 90% |
| **Best-in-class** | + Late Chunking + query expansion | 95% |

> **RAG is not dead. Only badly implemented RAG died.**

---

**Related posts:**
- [The Truth About AI Agent Token Costs](/knowhow/2026-09-23-agent-token-cost-truth/)
- [The Complete Web Search MCP Guide](/knowhow/2026-09-23-web-search-mcp-complete-guide/)
- [GPU VRAM Allocation Structure and the KV Cache Bible](/knowhow/2026-09-23-gpu-vram-kv-cache-bible/)