Distributed Systems Reference/Consensus & Coordination

Distributed Locks & Fencing Tokens

Why naive TTL locks fail, Redlock debate at awareness level, fencing tokens, and when to use etcd/Consul/ZK for coordination.

4/5Overview: 20m

Why locks are hard distributed

A lock on one machine is released when the process dies. Over the network:

  • Lock holder pauses (GC) — lock expires, another acquires, two holders
  • Network delay — same problem with TTL-based locks

Concurrency roadmap covers mutexes on one machine; distributed locks add clock and network uncertainty.

Naive approach: TTL lock in Redis

SET key token NX EX 30 — works until pause > TTL. Not sufficient for correctness-critical sections (e.g. writing to shared storage).

Fencing tokens

Monotonically increasing token from the lock service. Storage rejects writes with stale tokens even if old lock holder wakes up.

Lock service issues token=57 Writer must attach token=57 to storage write Stale leader with token=56 → rejected

Kleppmann's critique of Redlock: without fencing, TTL locks on multiple independent Redis nodes still risk split-brain writes.

Leases

Lease — lock with automatic expiry; holder must renew. Fowler's Lease pattern — know renew failure means stop writing.

When to use coordination services

NeedTool pattern
Leader electionetcd / ZK election API
Distributed configetcd / Consul KV
Service discoveryConsul (not deep dive here — Networking covers DNS/LB)

Interview answer

"For correctness under storage writes, I'd use a coordination service with fencing tokens, not a cache TTL lock alone. For best-effort deduplication, TTL lock may be enough."

Redlock awareness

Antirez's Redlock vs Kleppmann debate — know both sides exist; default to fencing + consensus-backed lock for high-stakes paths.

Further Reading