Datadog Unified Observability

Datadog's unified model

Datadog bundles:

Product	Pillar	Key features
Infrastructure	Metrics	Host, container, K8s metrics; integrations
APM	Traces	Distributed tracing, service map, span analytics
Log Management	Logs	Ingestion, pipelines, indexes, log patterns
RUM	Client-side	Browser/mobile real user monitoring
Synthetics	Proactive	API and browser tests from global locations
Profiling	Runtime	Continuous CPU/memory profiling

Unified = same UI, correlated signals, single agent.

The agent architecture

Datadog Agent runs per host / as DaemonSet:

• Collects infrastructure metrics
• Receives APM traces (library sends to agent)
• Tails container logs
• Forwards to Datadog SaaS

Libraries (Java, Python, Go) send traces/stats to local agent — low latency, buffering on network blips.

Trace ↔ log ↔ metric correlation

Datadog's strength: click from metric spike → representative trace → related logs:

• Shared tags: service, env, version, host
• Trace ID injected into logs automatically (with log injection enabled)
• Span Analytics — SQL-like queries on span attributes

Interview talking point: correlation reduces MTTR — you're not manually joining IDs across three UIs.

Custom metrics and cost

Datadog bills on:

• Custom metrics — cardinality × frequency
• Indexed log volume — GB ingested + retention
• APM — ingested spans (host-hour or span-based plans)

Cost controls:

• Tag cardinality limits
• Log exclusion filters at ingest
• APM ingestion rules (drop health checks, sample aggressively)
• Metrics without limits traps — audit quarterly

When Datadog wins

• Org wants one vendor, one bill, one on-call integration
• Non-K8s legacy + cloud mix — 600+ integrations out of box
• Security/compliance features (SIEM add-on, RBAC)
• Team lacks appetite to operate Loki/Tempo/Mimir

When to hesitate

• Cost at scale — FAANG-tier traffic gets expensive fast
• Vendor lock-in — proprietary query languages, migration pain
• OTel-native shops — may prefer OSS backends + Collector

Mitigation: instrument with OpenTelemetry, export to Datadog via OTel exporter — preserves portability.

Splunk and Quickwit positioning

Datadog logs compete with Splunk (enterprise incumbent) and Quickwit (cost-optimized archive). Datadog's pitch: correlation without Splunk's query language learning curve. Quickwit still wins on multi-year cold storage economics.

Interview answer

"We use Datadog for unified correlation — SLO dashboards, APM service map, log injection for trace IDs. OTel SDKs with Datadog exporter for portability. Custom metric cardinality is gated in code review."

Link forward

Grafana LGTM Stack is the OSS alternative with different ops trade-offs.