Observability Reference/Observability Foundations

Cardinality, Sampling & Cost

Label cardinality explosions, head vs tail sampling, retention tiers, and why "log everything" bankrupts you at FAANG scale.

3/5Overview: 30m

Cardinality is the hidden bill

Cardinality = number of unique time series (or indexed log combinations) your labels create.

http_requests_total{method="GET", path="/api/users", status="200", pod="checkout-7f3a", region="us-east-1"}

Each unique label-value combination is a series. Add user_id as a label and cardinality explodes — 10M users → 10M series → storage, query, and ingest costs skyrocket.

Rule of thumb: never put unbounded values in metric labels (user IDs, request IDs, URLs with IDs). Put those in logs or span attributes where full-text or trace backends handle them.

Where cardinality hurts

SystemCardinality pain
Prometheus/MimirMemory per series; scrape and query slowdown
DatadogCustom metric billing per tag combination
Splunk/ESIndex size when every field is indexed
LokiLabel cardinality (Loki rejects high-cardinality labels by design)

Sampling strategies

You cannot afford 100% trace capture at 50k RPS. Strategies:

StrategyHow it worksTrade-off
Head samplingDecide at trace start (probabilistic, rate-limited)Simple; may drop the one slow trace you need
Tail samplingDecide after trace completes (keep errors, high latency)Needs buffering; OTel Collector tail sampler
Parent-basedChild inherits parent's sampling decisionKeeps subtrees coherent

Google SRE: sample enough to detect problems, not everything. Error traces and slow traces deserve higher retention than happy paths.

Retention tiers

Production pattern:

  • Hot (hours–days) — full resolution metrics, sampled traces, searchable logs
  • Warm (weeks) — downsampled metrics, error-only traces, aggregated logs
  • Cold (months) — SLO rollups, compliance archives (S3 + Quickwit/Athena)

Cost optimization is an engineering decision, not FinOps alone.

Log volume economics

"Log everything" fails at scale:

  • Structured JSON at 1 KB × 100k RPS = 100 MB/s ingest
  • Agents, network, indexing, storage — all billed

Mitigations: log levels in prod (INFO default, DEBUG on demand), dynamic sampling (1% success, 100% errors), field discipline (don't duplicate data already in metrics).

Interview answer

"We cap label cardinality — bounded enums only on metrics. High-cardinality dimensions go in traces. We use tail sampling to keep error and p99 traces while probabilistically dropping happy paths. Log retention is tiered; we don't index DEBUG in prod."

Link forward

Logging covers pipeline design; Metrics covers histogram buckets (another cardinality lever); Context Propagation & Sampling dives into trace sampling mechanics.

Further Reading