Token Economics & Budgeting

Tokens are the unit of cost

LLM billing is per token (input and output priced separately). Understanding unit economics prevents "we shipped a feature and finance called."

Price model basics

Component	Typical pricing pattern
Input tokens	Prompt + retrieved context + history
Output tokens	Generated completion (often 2–4× input price)
Cached input	Discount when prefix repeats (OpenAI, Anthropic prompt caching)
Tool calls	Hidden tokens in arguments + results

Code and JSON tokenize densely — a 200-line file may be 8k tokens.

Unit economics template

Cost/session = Σ (calls × (input_tokens × $in + output_tokens × $out)) Monthly = sessions × DAU_penetration × MAU × cost/session

Example: 3 calls/session, 2k in + 500 out per call, $3/M in + $15/M out → ~$0.028/session. At 500k sessions/day → $420k/month.

Budget controls

Control	Where
Per-user daily cap	Gateway
Per-tenant quota	Multi-tenant SaaS
Per-feature budget	Feature flag + router
Global kill switch	Incident response

Alert at 80% of monthly budget with week remaining — not on day 31.

Cost optimization levers (ordered)

1. Shrink context — summarize history, trim retrieval (biggest lever)
2. Cap max_tokens — stop runaway generation
3. Route to smaller model — classifier picks tier
4. Cache — prompt caching, semantic cache (next subtopic)
5. Batch offline — non-interactive work off peak

Chip Huyen: measure before optimizing — profile which templates burn tokens.

FinOps dashboard

Engineering + finance shared view:

• Daily spend by feature/team/model
• Cost per successful task (not just per call)
• Forecast vs budget with trend line

Tie to product metrics — $0.50/session is fine if it replaces $5 support call.

Hidden costs

• Failed retries still bill partial tokens
• Eval and shadow traffic (often 10–20% of prod spend)
• Embedding backfills
• Human review labor

Include in TCO, not just API invoice.

Interview framing

"CFO says AI costs doubled — what do you do?"

1. Slice spend by feature deploy timeline
2. Check for retry loops, missing max_tokens, context bloat
3. Identify top 3 prompt templates by total cost
4. Propose routing/caching with projected savings

Senior signal: Talk margin per user action, not absolute dollars — connects engineering to business.

Link forward

Semantic Caching & Optimization implements the highest-ROI technical levers after context trimming.