Grafana LGTM Stack | Observability Reference

LGTM = Loki, Grafana, Tempo, Mimir

Grafana Labs' OSS observability stack:

Component	Pillar	Role
Loki	Logs	Label-indexed log aggregation
Grafana	Visualization	Dashboards, alerting, correlation UI
Tempo	Traces	Trace backend on object storage
Mimir	Metrics	Horizontally scalable long-term Prometheus

Plus Pyroscope for profiling (Production Debugging topic).

Why compose instead of monolith?

Best-of-breed each signal type
CNCF alignment — Prometheus, OTel, Loki all cloud-native
Cost control — S3-backed storage, self-hosted option
No per-host SaaS tax — you pay infra + ops time

Trade-off: you operate it — or pay Grafana Cloud to operate it for you.

Component recap

Loki — label-indexed log streams ({service="checkout"} | json), cheap without full-text body index.

Tempo — trace storage on S3/GCS; OTLP ingest, trace-by-ID lookup, correlates via trace ID in Loki and exemplars in Mimir.

Mimir — horizontally scalable Prometheus; multi-tenant retention, recording/alerting rules, drop-in query API. Alternatives: Thanos, Cortex.

Grafana as the hub

Single UI for:

PromQL on Mimir
LogQL on Loki
Trace search on Tempo
Derived fields — click trace_id in log → open trace

Alerting: Grafana unified alerting routes to PagerDuty/Slack from any datasource.

Operating the stack

Production concerns:

Concern	Component
Ingest scale	OTel Collector, Loki ingesters, Tempo distributors
Storage cost	S3 lifecycle policies, compaction, retention
Query performance	Mimir query frontends, Loki caches
High availability	Multi-AZ, replication factor, object storage durability

Grafana Cloud offloads this — pay per GB/series instead of hiring a platform team.

vs Datadog

Dimension	LGTM	Datadog
Ops burden	High (self-hosted)	Low (SaaS)
Correlation	Good (with setup)	Excellent (native)
Cost at scale	Often lower	Often higher
Integrations	Build/buy	600+ built-in

Interview answer

"We run OTel Collector → Mimir/Tempo/Loki on Grafana. S3 backends for long retention. Grafana unified alerting on SLO burn rates. Chose LGTM for cost and PromQL portability; accept ops investment."

Link forward

Production Debugging ties dashboards, exemplars, and profiling into the daily workflow.