Layers of flow control
- TCP — receive window, congestion control (Networking)
- HTTP/2 — stream windows
- gRPC — per-stream flow control on HTTP/2
- Application — bounded queues, reactive streams
Failure at layer 4 when layers 1–3 still accept data → OOM. Always bound in-memory buffers.
Reactive Streams contract
Publisher → Subscriber with demand:
- Subscriber calls
request(n)— backpressure signal - Publisher sends at most n items
Implementations: RxJava, Project Reactor, Akka Streams, Node streams pause()/resume().
Load shedding (Google SRE)
Under overload:
- Return 503 fast with
Retry-After— better than slow success for everyone - Drop low-priority traffic (degraded mode)
- Admission control — reject at gateway when queue depth high
User-facing > background jobs when shedding.
gRPC streaming backpressure
Client/server stream handlers must not unbounded onNext — respect isReady / flow control windows. Blocking stub anti-pattern on streaming RPCs.
Queueing theory sketch
If arrival rate > service rate, queue grows without bound unless you shed or scale. Little's Law: L = λW — depth relates to latency.
Staff tie-in: "Our p99 spiked" — check downstream queue depth, thread pool saturation, not just CPU.
Async HTTP
Node/Python async handlers still need semaphores on outbound calls — async ≠ unlimited concurrency.
Cross-reference: Concurrency → Semaphores & Thread Pools for in-process limits; Observability → Metrics for queue-depth alerting; Networking for TCP windowing.