Three Pillars & Observability vs Monitoring

Monitoring vs observability

Monitoring answers known questions: "Is error rate above 1%?" "Is disk 90% full?" You predefine dashboards and alerts.

Observability answers unknown questions: "Why did checkout fail for this user in this AZ on this deploy?" You need rich, high-cardinality telemetry you can slice arbitrarily — not just pre-baked graphs.

Charity Majors' test: can you debug a novel failure without shipping new code? If not, you have monitoring, not observability.

The three pillars

Pillar	What it is	Strengths	Weaknesses
Logs	Discrete events with context	Rich detail, arbitrary fields	Expensive at volume; bad for aggregates
Metrics	Numeric time series	Cheap aggregation, alerting, SLOs	Low cardinality; loses individual request context
Traces	Request-scoped span trees	End-to-end latency, dependency map	Sampling required; instrumentation effort

They are complementary, not competing. Senior engineers connect all three via shared IDs (trace_id, service, deployment).

The four golden signals (Google SRE)

For any user-facing service, instrument:

1. Latency — time to serve a request (distinguish success vs error latency)
2. Traffic — demand (requests/sec, connections)
3. Errors — explicit failures + implicit failures (HTTP 500, slow timeouts)
4. Saturation — how "full" the service is (thread pool, queue depth, CPU throttle)

RED (Rate, Errors, Duration) is the microservice shorthand for 1–3. USE (Utilization, Saturation, Errors) applies to resources — see Metrics topic.

What sibling tracks already cover

• Networking — transport failures, DNS, TLS, packet-level latency. Observability instruments above the socket.
• OS — process scheduling, disk I/O, memory pressure. Metrics here are resource-level; this track focuses on service-level signals.
• Distributed Systems — partial failure, retries, circuit breakers. Observability measures those patterns; implementation lives elsewhere.

Interview framing

When asked "how do you debug production?":

1. Start with user impact (SLO breach? which cohort?)
2. Check golden signals on the failing slice
3. Drill from metrics → traces → logs (not the reverse — metrics are cheaper to scan)
4. Correlate with deploys and config changes

Senior signal

Say "I'd add a span attribute for payment_provider" instead of "I'd check the logs." Name the correlation ID that ties pillars together. Distinguish symptom alerts (SLO burn) from cause alerts (single pod CPU) — paging on symptoms, investigating causes.

Link forward

Cardinality & Sampling explains why you can't log every field at FAANG scale; Distributed Tracing shows how spans carry the context logs and metrics lack.