Fine-Tuning vs RAG vs Prompts

Three levers for adapting LLMs

You can't retrain GPT from scratch. You have three practical levers:

Lever	What it changes	Effort	Iteration speed
Prompting	Input (instructions, examples, context)	Low	Minutes
RAG	Input (retrieved knowledge)	Medium	Hours (re-index)
Fine-tuning	Model weights (behavior, style, format)	High	Days–weeks

Most teams over-invest in fine-tuning and under-invest in prompts and RAG. Start left, move right only when measured need exists.

Decision framework

Need the model to know new FACTS? → RAG (retrieval at query time) Need the model to behave in a specific FORMAT/STYLE consistently? → Prompting first (structured outputs, few-shot examples) → Fine-tuning if prompting can't hit consistency targets Need the model to perform a specialized TASK reliably at scale? → Fine-tuning (with eval suite proving it beats prompting) Need real-time data? → RAG or tool use (not fine-tuning — knowledge goes stale) Need to reduce cost for high-volume identical task? → Distillation (train small model on frontier model outputs)

When prompting is enough

Prompting solves 80% of adaptation needs:

• Output format enforcement (JSON schema, structured outputs)
• Style and tone (via rules and examples)
• Task decomposition (chain prompts)
• Domain context (paste docs, use @-mentions)

Stop here unless evals prove prompting can't meet your quality bar on representative test cases.

When RAG is the answer

RAG when the model needs facts it wasn't trained on:

• Internal documentation, wikis, ADRs
• Codebase knowledge (with code-aware chunking)
• Dynamic data (re-index on change)
• Citation requirements ("show your sources")

RAG does NOT teach behavior/style. If the model formats answers wrong, RAG won't fix that — prompting or fine-tuning will.

When fine-tuning actually pays off

Fine-tuning is justified when ALL of these are true:

1. Prompting + RAG can't meet quality bar (proven by eval, not intuition)
2. You have 100+ high-quality training examples (more is better)
3. The task is repetitive and high-volume (cost savings justify training cost)
4. You have eval infrastructure to measure before/after
5. Behavior is stable (not changing weekly with new product features)

Common good fits:

• Classification (intent detection, severity triage)
• Structured extraction (parse logs into fields)
• Domain-specific code generation (proprietary DSL/framework)
• Consistent formatting the model resists via prompting alone

Common bad fits:

• "Make it know our product" (that's RAG)
• "Make it smarter" (fine-tuning doesn't add reasoning)
• Tasks with < 50 examples (will overfit)
• Rapidly changing requirements (re-training cost)

Cost comparison (rough)

Approach	Setup cost	Per-query cost	Maintenance
Prompting	$0	Highest (long prompts)	Update prompts in git
RAG	Index infrastructure	Medium (retrieval + generation)	Re-index on content change
Fine-tuning	Training run ($10–500+)	Lower (shorter prompts, smaller model possible)	Re-train on behavior change
Distillation	Training + frontier inference for dataset	Lowest	Re-distill on task change

Distillation: the cost hack

Train a small/fast model on outputs from a frontier model:

1. Collect 500+ examples of frontier model doing your task correctly 2. Fine-tune a small model (Haiku, GPT-4.1-nano) on those outputs 3. Deploy small model at 10–20× lower cost 4. Keep frontier as fallback for low-confidence cases

OpenAI's distillation guide covers this formally. It's how teams run high-volume classification/extraction affordably.

The iteration order

1. Prompt engineering + structured outputs 2. Add RAG for factual grounding 3. Build eval suite (20+ cases) 4. Measure pass rate 5. If < target: try better prompts, better retrieval, better chunking 6. If still < target: consider fine-tuning 7. Fine-tune → re-eval → compare to prompt+RAG baseline 8. Only deploy fine-tuned if measurably better (not just different)

Skip steps at your peril. Teams that fine-tune first usually discover prompting would have worked.

Interview framing

"I start with prompting and RAG — they cover 80% of adaptation needs with hours of effort vs weeks for fine-tuning. I fine-tune only when evals prove prompting can't hit the quality bar AND I have 100+ training examples AND the task is high-volume enough to justify training cost. Distillation is my cost optimization play for proven tasks."

Senior signal: Walk through the decision tree for a specific scenario. Explain why fine-tuning is wrong for "make the model know our API docs" and right for "classify support tickets into 12 categories at 10k/day."