When traces aren't enough
Traces show where time went; profiling shows what code consumed it.
Scenario: p99 latency high, spans show time in process_order() but no child spans — CPU-bound logic, lock contention, or GC pause inside the span.
Continuous profiling samples stack traces at low overhead — aggregate into flame graphs.
Flame graphs
████████████████████████ main
████████████████ handle_request
██████████ process_order
██████ calculate_tax ← wide = hot
████ json_serialize
Width = CPU time (or memory allocations). Read bottom-up: who called whom.
Look for: unexpectedly wide frames, lock waits, regex, serialization.
Continuous profiling tools
| Tool | Notes |
|---|---|
| Grafana Pyroscope | OSS, integrates with LGTM, push or pull models |
| Google Cloud Profiler | Low-overhead sampling, multi-language |
| Datadog Profiler | Unified with APM traces — profile linked to span |
| async-profiler (Java) | Production-safe, wall-clock and CPU modes |
Overhead target: < 1-2% CPU — always-on in prod, not just during incidents.
Profile ↔ trace correlation
Best experience: click span → see profile for that time window.
Datadog and Pyroscope support profile linked to trace ID — answers "what was this specific slow request doing?"
eBPF-based observability
eBPF — sandboxed programs in the Linux kernel without kernel modules:
| Tool | Use |
|---|---|
| bpftrace | Ad-hoc kernel probes — latency histograms per syscall |
| BCC tools | biolatency, tcplife, runqlat — production-safe scripts |
| Cilium Hubble | Network observability for K8s (flows, DNS, L7) |
| Pixie (CNCF) | Auto telemetry via eBPF in K8s |
No code changes; kernel/network visibility at lower overhead than strace. Linux-focused, requires privileges.
eBPF vs application observability
| Layer | Sees |
|---|---|
| eBPF | Syscall latency, TCP retransmits, DNS timing, disk I/O wait |
| OTel traces | Service boundaries, business context |
| Logs | Application errors and state |
Networking track covers tcpdump and curl — eBPF is the programmatic, always-on evolution for kernel/network path.
OS track covers generic I/O and scheduling — eBPF observes them in prod without iostat polling.
When to profile vs add spans
| Situation | Action |
|---|---|
| Unknown CPU hot spot | Continuous profiling |
| Missing dependency visibility | Add span around external call |
| Kernel/network anomaly | eBPF tooling |
| GC pauses | JVM flight recorder + profiling |
Don't add a span per function — overhead and cardinality cost.
Interview answer
"We run continuous profiling in prod via Pyroscope. During latency incidents, correlate exemplar traces to flame graphs. For kernel-level issues — runqlat and biolatency via bpftrace — especially when app traces look clean but latency is high."
Senior signal
Know profiling safety (sample rates, overhead budgets) and when eBPF requires platform team partnership. Full loop: instrument → SLO → burn alerts → incident response → dashboards/exemplars/profiling.