OpenTelemetry Fundamentals

What is a trace?

A trace is a tree of spans representing one logical request's journey:

[checkout trace abc123] ├─ span: API Gateway (120ms) │ ├─ span: Auth (15ms) │ ├─ span: Cart Service (80ms) │ │ └─ span: Redis GET (2ms) │ └─ span: Payment (95ms) │ └─ span: Stripe API (90ms)

Each span has: trace_id, span_id, parent_span_id, name, start/end time, status, attributes, events.

OpenTelemetry layers

Layer	Role
API	Interfaces (Tracer, Span) — no-op by default
SDK	Sampling, batching, export implementation
Collector	Receive → process → export (vendor-neutral pipeline)
Instrumentation	Auto (Java agent, Python) or manual spans

Vendor-neutral is the point — instrument once with OTel, export to Jaeger, Tempo, Datadog, or Honeycomb.

Auto vs manual instrumentation

Auto-instrumentation (language agents):

• HTTP/gRPC frameworks, DB drivers, message queues — spans appear with minimal code
• Fast adoption; generic span names (HTTP GET)

Manual instrumentation:

• Business spans (process_payment, fraud_check)
• Custom attributes (order_value, payment_provider)
• Required for domain-specific debugging

Senior teams: auto for plumbing, manual for business-critical paths.

OTel Collector pipeline

Receivers (OTLP, Jaeger, Prometheus) → Processors (batch, tail_sampling, attributes) → Exporters (Tempo, Datadog, logging)

Run Collector as DaemonSet (per node) or sidecar. Centralizes sampling and PII scrubbing.

Span attributes vs events

Mechanism	Use for
Attributes	Key-value context (http.status_code=500, db.system=postgres)
Events	Timestamped log lines within a span (cache_miss, retry_attempt)

Don't stuff large payloads into attributes — size limits exist.

Status and errors

Set span status to ERROR with description when operations fail. Unset status on success (not OK explicitly unless needed).

Exceptions: record with record_exception() — captures stack trace as event.

Interview answer

"We standardize on OpenTelemetry. Java services use the agent for HTTP/DB; payment flow has manual spans. Collector does tail sampling and exports to Tempo. Every span carries deployment.version for rollback correlation."

Cross-reference

• Networking — traces show application latency; tcpdump shows wire retransmits
• Logging — share trace_id in structured logs

Link forward

Context Propagation & Sampling covers W3C headers and keeping traces coherent across async boundaries.