Cardinality is the hidden bill
Cardinality = number of unique time series (or indexed log combinations) your labels create.
http_requests_total{method="GET", path="/api/users", status="200", pod="checkout-7f3a", region="us-east-1"}
Each unique label-value combination is a series. Add user_id as a label and cardinality explodes — 10M users → 10M series → storage, query, and ingest costs skyrocket.
Rule of thumb: never put unbounded values in metric labels (user IDs, request IDs, URLs with IDs). Put those in logs or span attributes where full-text or trace backends handle them.
Where cardinality hurts
| System | Cardinality pain |
|---|---|
| Prometheus/Mimir | Memory per series; scrape and query slowdown |
| Datadog | Custom metric billing per tag combination |
| Splunk/ES | Index size when every field is indexed |
| Loki | Label cardinality (Loki rejects high-cardinality labels by design) |
Sampling strategies
You cannot afford 100% trace capture at 50k RPS. Strategies:
| Strategy | How it works | Trade-off |
|---|---|---|
| Head sampling | Decide at trace start (probabilistic, rate-limited) | Simple; may drop the one slow trace you need |
| Tail sampling | Decide after trace completes (keep errors, high latency) | Needs buffering; OTel Collector tail sampler |
| Parent-based | Child inherits parent's sampling decision | Keeps subtrees coherent |
Google SRE: sample enough to detect problems, not everything. Error traces and slow traces deserve higher retention than happy paths.
Retention tiers
Production pattern:
- Hot (hours–days) — full resolution metrics, sampled traces, searchable logs
- Warm (weeks) — downsampled metrics, error-only traces, aggregated logs
- Cold (months) — SLO rollups, compliance archives (S3 + Quickwit/Athena)
Cost optimization is an engineering decision, not FinOps alone.
Log volume economics
"Log everything" fails at scale:
- Structured JSON at 1 KB × 100k RPS = 100 MB/s ingest
- Agents, network, indexing, storage — all billed
Mitigations: log levels in prod (INFO default, DEBUG on demand), dynamic sampling (1% success, 100% errors), field discipline (don't duplicate data already in metrics).
Interview answer
"We cap label cardinality — bounded enums only on metrics. High-cardinality dimensions go in traces. We use tail sampling to keep error and p99 traces while probabilistically dropping happy paths. Log retention is tiered; we don't index DEBUG in prod."
Link forward
Logging covers pipeline design; Metrics covers histogram buckets (another cardinality lever); Context Propagation & Sampling dives into trace sampling mechanics.