Structured Logging & Pipelines

Why structured logging

Unstructured: User 12345 failed checkout: timeout

Structured:

Machines parse JSON; humans query fields. Splunk, Loki, Datadog, and Quickwit all index on fields — not regex on free text.

Schema design principles

Field	Guidance
timestamp	ISO 8601, UTC
level	ERROR, WARN, INFO — avoid DEBUG in prod by default
service / version	Deployment correlation
trace_id / span_id	Links to distributed traces
message	Human-readable summary
Business IDs	order_id, request_id — bounded, queryable

Do not log secrets (tokens, PANs, passwords). Hash or truncate PII (user_id_hash).

Correlation IDs

A single request generates logs across services. Propagate:

• W3C traceparent — standard trace context (see Distributed Tracing topic)
• request_id — if traces aren't instrumented yet (legacy bridge)

Every log line from the request path should carry the same ID. Without it, you're grep-ing timestamps and hoping.

Log pipeline architecture

App (stdout JSON) → Agent/sidecar → Buffer → Shipper → Ingest → Index → Query UI

Components:

• Agent (Fluent Bit, Datadog agent, Filebeat) — tails container logs, adds k8s metadata
• Buffer — disk queue when sink is down (backpressure matters)
• Shipper — batching, compression, retry with backoff
• Ingest — parsing, field extraction, sampling rules

12-Factor: apps write to stdout; the platform routes. Don't manage log files inside containers.

Levels in production

Level	Prod default?	When
ERROR	Always	Actionable failures
WARN	Yes	Degraded but serving
INFO	Yes	Key lifecycle events (startup, config load)
DEBUG	No	Enable per-pod or per-request via dynamic flag

"Log level DEBUG globally in prod" is a paging incident waiting to happen.

Backpressure and failure modes

When the log sink is down:

• Agent buffers to disk (bounded)
• App stdout pipe fills → process blocks on write → cascading latency

Mitigations: non-blocking loggers, drop-on-full policies for DEBUG, monitor agent health.

Interview signal

Describe the full path from app to query UI. Mention trace correlation. Explain why you won't put user_id on every metric but will log it (hashed) for support tickets.

What we skip

Wire-level inspection (tcpdump, curl -v) is Networking. This topic is application event streams.

Link forward

Log Search Platforms compares Splunk, Quickwit, Loki, and Elasticsearch query models.