Platform landscape
| Platform | Model | Best for |
|---|---|---|
| Splunk | Index everything, SPL queries | Enterprise, security, complex analytics |
| Elasticsearch/OpenSearch | Inverted index on text + fields | Full-text search, flexible schemas |
| Quickwit | Columnar on object storage (S3) | Cost-effective search on massive cold logs |
| Grafana Loki | Label-indexed streams (like Prometheus for logs) | K8s-native, correlates with Grafana metrics/traces |
No single winner — choice depends on query patterns, retention, and ops appetite.
Splunk essentials
SPL (Search Processing Language):
index=checkout status=500 | stats count by error_code | sort -count
Strengths: mature ecosystem, field extraction, ML toolkit, compliance features. Costs: license by ingest volume — index only what you'll query.
Senior tip: use source types and field extractions at ingest, not rex at search time on every query.
Elasticsearch / OpenSearch
Documents indexed with mappings (field types). Query with DSL or Lucene syntax.
Watch mapping explosions — dynamic mapping on high-cardinality fields (like raw URLs with IDs) bloats the cluster.
Pattern: hot-warm-cold architecture — recent logs on SSD nodes, old logs on cheaper tiers.
Quickwit
Built for search on object storage:
- Indexes in S3/GCS (cheap)
- Sub-second search on TB-scale archives
- Good fit for compliance logs, post-incident forensics, cost-sensitive retention
Trade-off: not a real-time streaming analytics platform like Splunk. Think "searchable archive."
Grafana Loki
Labels first, full-text second — like Prometheus labels for logs:
{service="checkout", level="error"} |= "timeout"
LogQL — rate(), count_over_time(), metric queries from logs.
Strengths: native Grafana integration, cheap storage (no full-text index on body by default), correlates with Tempo traces and Mimir metrics.
Constraint: low-cardinality labels only — user_id is not a Loki label.
Query patterns that scale
- Start narrow — time range + service + level, then filter
- Aggregate before raw —
stats count/count_over_timebefore pulling lines - Correlate via trace_id — jump from metric spike → trace → logs
- Avoid —
*wildcards on TB indexes, unbounded time ranges
Interview comparison
"Splunk when the org already runs it and analysts need SPL. Loki when you're on Grafana LGTM and want label-native logs cheap. Quickwit when you need S3-backed search on years of archives. ES when full-text on arbitrary fields is the primary query."
Cross-reference
- Networking — when logs aren't enough, packet capture (
tcpdump) shows wire behavior - Observability Platforms — Datadog logs vs Grafana Loki in unified stacks
Link forward
Metrics topic covers when to metricize log-derived counts instead of searching logs repeatedly.