What a service mesh adds
Sidecar proxy (Envoy) per pod intercepts all traffic:
- mTLS between services without app code changes
- Retries, timeouts, circuit breaking uniformly
- Traffic split — canary without redeploying app
- Observability — golden metrics per service pair
Istio, Linkerd, Consul Connect — app connects localhost:15001 (sidecar); sidecar routes to real backends.
Data plane vs control plane
| Plane | Role |
|---|---|
| Data | Envoy sidecar — forwards bytes, enforces policy |
| Control | Istiod / Linkerd controller — pushes config via xDS |
mTLS with SPIFFE
SPIFFE ID — spiffe://trust.domain/ns/payments/sa/api — workload identity.
Certs rotated automatically; no shared static API keys between services.
Cost: CPU on encryption; debugging harder (encrypted tcpdump useless without tap).
Retries at mesh layer — danger
Automatic retry on POST duplicates side effects unless:
- Methods are idempotent
- Idempotency keys honored upstream (Communication)
- Retry budgets cap blast radius
Configure per-route: GET retries OK; POST default off.
vs library-based resilience
| Approach | When |
|---|---|
| Client library (Resilience4j, gRPC interceptors) | Few languages, small fleet |
| Mesh | Polyglot, uniform policy, platform team owns SRE |
Small startup: mesh ops cost > benefit. 50+ services: mesh often wins.
vs API gateway
Mesh = east-west. Gateway = north-south. Don't duplicate rate limits without coordination.
Distributed Systems → Consensus — different problem from mesh routing.
Cross-reference: Communication → gRPC for L7 LB requirements; Observability for RED metrics per service pair.
Further Reading
- Linkerd — Automatic mTLSReference15m
- SPIFFE — Secure Production Identity FrameworkReference20m
- Istio — What is a service mesh?Reference20m
Hands-On Tasks (Optional)
Architecture drills and whiteboard exercises. Assumes Communication & Data Transfer and Distributed Systems fundamentals.
- Configure safe mesh retries15m
gRPC unary with 503 from upstream. Define retry budget, which methods are safe to retry, and how idempotency keys interact with automatic retries.