Semantic Caching & Optimization

Caching layers for LLM apps

Cache type	Key	Hit condition	Staleness risk
Exact	Hash(prompt + model + params)	Identical request	Low
Semantic	Embedding of normalized query	Cosine similarity > θ	Medium–high
Prompt prefix	Vendor-side prefix match	Shared system prompt	Low (vendor feature)
Retrieval	Query embedding	Same KB state	Invalidates on corpus update

Semantic caching mechanics

Query → embed → nearest neighbor in cache index → if sim > 0.92, return cached response

Parameters to tune:

• Similarity threshold — higher = fewer false hits, lower hit rate
• TTL — FAQ content: hours; dynamic data: minutes or none
• Normalization — lowercase, strip whitespace, canonicalize entities

GPTCache and Redis + vector extensions are common implementations.

Stale answer risk

Semantic cache will serve wrong answers when:

• Underlying facts changed ("CEO of X")
• User intent differed subtly ("cancel subscription" vs "pause subscription")
• Policy rules updated

Mitigations:

• Never cache high-stakes actions (payments, medical)
• Include corpus_version in cache key metadata
• Shorter TTL for time-sensitive domains

Vendor prompt caching

OpenAI and Anthropic discount repeated input prefixes (system prompt, few-shot examples). Structure prompts as:

[Static prefix — cached] + [Dynamic user context — not cached]

Can cut input cost 50–90% for agent loops with large system prompts.

Model routing for cost-quality

RouteLLM / custom classifiers send easy queries to cheap models:

Query class	Model	$/1M tokens (illustrative)
FAQ, extraction	Small	$0.10
General chat	Medium	$3
Complex reasoning	Frontier	$15

Train router on labeled query difficulty; monitor quality regression on routed traffic.

Prompt compression

Techniques:

• Summarize conversation history instead of full verbatim
• Retrieve top-3 chunks, not top-20
• Use smaller model to compress context before frontier call

Measure quality impact on eval set — compression is lossy.

When not to cache

• Personalized responses (user-specific state)
• Regulated audit requirements (must log fresh inference)
• Adversarial surface (cache poisoning)

Interview framing

"Reduce inference cost 40% without killing quality":

1. Prompt prefix caching for system prompts
2. Semantic cache for top 20 FAQ intents (measure hit rate)
3. Router: 60% traffic to small model with eval gate
4. Trim retrieval from 10k to 3k tokens

Senior signal: Quote expected hit rate and stale risk — not just "add Redis."

Link forward

Shipping & Operations covers rolling out routing and cache config safely with feature flags.