RAG Fundamentals & Chunking

What RAG solves

LLMs know public internet text. They don't know your codebase, runbooks, or last week's ADR. Retrieval-Augmented Generation fetches relevant documents at query time and injects them into the prompt — grounding the model in facts you control.

User query → Embed query → Search vector DB → Retrieve top-k chunks → Inject chunks into prompt → LLM generates grounded answer

RAG is the default pattern for "chat with our docs" and "answer questions about our codebase." It's cheaper and faster to iterate than fine-tuning.

The RAG pipeline

Stage	Options	Key decision
Ingest	Parse PDF, MD, HTML, code	Preserve structure (headings, code blocks)
Chunk	Fixed-size, semantic, recursive	Chunk size vs retrieval precision
Embed	OpenAI, Cohere, open-source (e.g. nomic, bge)	Quality vs cost vs latency
Store	Pinecone, Weaviate, pgvector, Chroma	Scale, filtering, hybrid search support
Retrieve	Cosine similarity, BM25, hybrid	Pure semantic misses exact matches (API names)
Rerank	Cohere reranker, cross-encoder	Improves top-k at small latency cost
Generate	Any LLM with retrieved context	Cite sources; reject if no relevant chunk found

Chunking: the make-or-break step

Bad chunking = bad RAG, regardless of embedding model quality.

Strategy	How	Best for
Fixed-size	Split every N tokens with overlap	Simple baseline; breaks mid-sentence
Paragraph/section	Split on \n\n or headings	Docs, wikis, markdown
Recursive	Split on \n\n → \n → . → char	LangChain default; decent general-purpose
Code-aware	Split on functions/classes	Source code (preserve signatures)
Parent-child	Small chunks for search, large for context	Best of both: precise retrieval, rich context

Rules of thumb:

• Chunks of 256–512 tokens for factual Q&A
• Chunks of 512–1024 tokens for code (keep function bodies intact)
• 10–20% overlap between chunks to avoid boundary splits
• Include metadata (source file, section title, last-modified) for filtering and citation

Embedding model selection

Mid-2026 options:

Model	Strength	Trade-off
OpenAI text-embedding-3-large	Strong general quality	API cost, vendor lock-in
OpenAI text-embedding-3-small	Good enough for most	Lower cost, slightly less precise
Open-source (bge, nomic)	Self-hosted, no API cost	Ops burden, may need fine-tuning for domain

Re-embed when you switch models — embeddings aren't interchangeable across model families.

Failure modes

Failure	Symptom	Fix
Chunk too small	Answer lacks context	Increase chunk size or use parent-child
Chunk too large	Irrelevant filler dilutes answer	Decrease size; add reranking
Stale index	Answers reference deleted code	CI-triggered re-index on merge
Missing content	"I don't know" for known topics	Check ingest gaps; expand sources
Wrong chunk retrieved	Confident wrong answer	Hybrid search; reranking; eval suite
No citation	Can't verify answer	Require source attribution in prompt

Hybrid search

Pure vector search misses exact matches (function names, error codes, config keys). Combine:

Final score = α × semantic_similarity + (1-α) × BM25_keyword_score

Typical α = 0.7 for docs, 0.5 for code. Most production systems use hybrid.

When RAG beats alternatives

Need	RAG	Fine-tuning	Long context
Q&A over docs	✅ Best	Overkill	Expensive at scale
Code style/conventions	⚠️ Partial	✅ Better	⚠️ Partial
Real-time data	✅ Re-index	❌ Stale	❌ Cutoff
New product knowledge	✅ Fast to update	Slow, expensive	Manual paste

Interview framing

"RAG retrieves relevant chunks at query time and injects them into the prompt. The critical engineering decisions are chunking strategy, hybrid search, and eval — not embedding model selection. I'd build a 20-query eval set before tuning anything."

Senior signal: Explain parent-child chunking. Mention you'd re-index on every merge to main. Describe a RAG failure you diagnosed (wrong chunk retrieved) and how you'd fix it.