Health Checks & Client-Side LB

Liveness vs readiness

Probe	Question	Fail action
Liveness	Is process deadlocked?	Restart pod
Readiness	Can it accept traffic?	Remove from LB pool
Startup	Slow-init app (JVM)	Delay liveness until ready

Readiness should fail when:

• DB connection pool exhausted
• Dependency down (optional — depends on degradation strategy)
• Draining for shutdown

Don't put liveness on "dependency down" — restart loop won't fix upstream outage.

Graceful shutdown sequence

1. Receive SIGTERM / preStop hook
2. Fail readiness probe
3. Wait deregistration_delay (LB stops sending new connections)
4. Drain in-flight requests (timeout bounded)
5. Close listeners, exit

K8s terminationGracePeriodSeconds must exceed drain time.

gRPC health checking

Standard grpc.health.v1.Health/Check — LB/mesh probes without invoking business RPCs.

Client-side load balancing (gRPC)

Naive round-robin on TCP connections skews load (HTTP/2 multiplexing).

Solutions:

• Pick-first with subchannel refresh
• Round_robin policy per gRPC channel
• xDS — control plane assigns weighted subsets (GKE, Istio)

Staff interview: explain why ALB TCP mode breaks gRPC load spread.

Health check design anti-patterns

• Hitting DB on every liveness probe → DB overload
• Same endpoint for liveness and readiness
• No timeout on health handler → hung probe marks healthy forever

Connection pooling at scale

Sidecar or client maintains warm connections to subset of backends. Subset selection — client tracks 100 backends but connects to 5 (reduces connection count).

Cross-reference: Concurrency → Thread Pools for server-side accept pool sizing.