RAG Ingestion at Scale

From notebook RAG to production corpus

AI Engineering covers chunking strategies and basic vector search. At scale, the problem becomes a data pipeline:

• Billions of chunks across tenants
• Continuous updates from wikis, tickets, code repos
• Embedding model version changes requiring re-index
• SLA: "merged doc searchable within N minutes"

Ingestion architecture

Source (Git, Confluence, S3) → CDC/Webhook → Parse → Chunk → Embed → Upsert → Index ↓ Dead-letter, replay

Concern	Pattern
Trigger	Webhooks on merge; nightly full reconcile
Idempotency	Content-hash as chunk ID — skip unchanged
Deletion	Tombstone on doc delete; purge vectors by metadata filter
Versioning	doc_id + revision metadata; query latest only

See Databases for CDC patterns — same mechanics, embedding step added.

Chunking at scale

Notebook chunking is manual; production chunking is policy per source type:

Source	Strategy
Code	AST-aware splits, respect function boundaries
Markdown/docs	Header-based sections + overlap
PDFs	Layout-aware parsing (not naive page split)
Tickets	One chunk per comment thread

Re-chunking on policy change = full backfill — version your chunker.

Embedding pipelines

• Batch embed offline for backfill (cheaper, rate-limit friendly)
• Online embed on hot path only for user queries
• Pin embedding model version — mixing versions in one index destroys recall

Backfill playbook:

1. New index with new embeddings (blue)
2. Dual-write or shadow query both
3. Compare recall@k on eval set
4. Cut traffic; delete old index (green)

Freshness vs cost

Every commit ≠ re-embed everything. Use:

• Diff-based incremental updates
• Tiered freshness (hot repos: 5 min; archive: weekly)
• Lazy embed on first query for cold content (trade latency)

Multi-tenant isolation

Enterprise SaaS needs:

• Namespace per customer in vector DB
• Row-level security metadata filters
• Per-tenant ingestion quotas

Cross-tenant leakage is a severity-1 incident — enforce at index and query layers.

Interview framing

"Index 50TB of internal docs":

1. Object store for raw docs, vector DB for embeddings
2. Incremental pipeline with content-hash dedup
3. Blue/green index for embedding model upgrades
4. Freshness SLO per doc class

Senior signal: Mention parse quality — garbage chunks from bad PDF extraction cause more production pain than wrong HNSW ef_search.

Link forward

Vector Index Operations covers query-time performance, hybrid search, and billion-vector ops.