How LLMs Work (Inference, Not Training)

What you're actually talking to

An LLM at inference time is not a database, search engine, or reasoning engine. It is an autoregressive transformer that predicts the next token given everything before it. Every response — code, prose, JSON — is token-by-token sampling from a learned distribution.

Understanding this single fact explains most production surprises.

The inference stack (what matters to engineers)

Your prompt → Tokenizer → Context window → Transformer layers → Logits → Sampler → Output tokens

Component	What it does	Why you care
Tokenizer	Splits text into subword tokens (BPE/SentencePiece)	Code tokenizes differently than English; costs scale with tokens, not characters
Context window	Fixed-size attention over input + output	Everything must fit; long files get truncated silently
Attention	Each token attends to prior tokens	Order matters; information in the middle gets lost (lost-in-the-middle)
Sampler	Picks next token from probability distribution	Temperature, top-p control creativity vs determinism

You don't need to derive attention math. You need to know context is finite, tokens cost money, and sampling is probabilistic.

Autoregressive generation

The model generates left-to-right. It cannot "go back and fix" an earlier token. This is why:

• It commits to wrong API names early and doubles down
• Multi-step reasoning chains can derail at step 3
• Structured output (JSON) breaks if one brace is wrong — it keeps generating plausible continuation

Mitigation: constrain output format (JSON mode, tool schemas), break tasks into smaller calls, verify outputs programmatically.

Key inference parameters

Parameter	Effect	Production default
Temperature	Higher = more random	0–0.3 for code; 0.7+ for brainstorming
Top-p (nucleus)	Limits token pool to cumulative probability mass	0.9–1.0 typical
Max tokens	Hard cap on output length	Set explicitly; unbounded = runaway cost
Stop sequences	Halt generation on delimiter	Useful for multi-turn tool loops

For code generation and agent workflows, low temperature + schema constraints beats clever prompting.

Context window economics

Mid-2026 frontier models offer 128k–1M token windows, but:

• Latency scales with context length (not just output)
• Cost scales linearly with input tokens on most APIs
• Quality degrades in the middle of very long contexts

Rule: treat the context window as a budget, not a dump truck. Put the most important material at the beginning and end; summarize the middle.

What training gave you (without training yourself)

Pre-training teaches broad language patterns from internet-scale text. Post-training (RLHF, constitutional AI, instruction tuning) teaches helpfulness and format compliance. Neither teaches:

• Your private codebase
• Today's API version
• Your team's conventions

That's why RAG, rules, and MCP exist — covered in later topics.

Interview framing

When asked "how do LLMs work?":

1. Autoregressive next-token prediction over a fixed context window
2. Tokenization determines cost and effective capacity
3. Sampling parameters control determinism
4. No built-in fact-checking — outputs are plausible, not guaranteed true

Senior signal: Say "I'd check token count before sending the whole monorepo" instead of "I'd give it all the context." Name the inference parameter you'd tune for a CI code-review bot vs a brainstorming session.