Custom Binary Protocols & Framing

When custom protocols still happen

• Ultra-low latency trading, gaming — UDP + custom framing
• Embedded / IoT — constrained bandwidth
• Legacy mainframe — fixed-width records
• High-throughput internal — when protobuf+gRPC overhead still too high (rare)

Default answer in 2026: don't invent — use gRPC/protobuf unless profiling proves need.

Framing problem on TCP

TCP is a byte stream — no message boundaries. Protocols need framing:

Style	Format
Length-prefix	[4-byte len][payload]
Delimiter	newline-terminated JSON (fragile)
TLV	type, length, value chains
HTTP/2	frames (standard)

Partial read loop: buffer until full frame available; handle EAGAIN.

Versioning binary protocols

• Header includes version byte
• Unknown version → reject connection
• New fields at end with length — forward compatible if readers skip unknown TLV

Same rules as protobuf field numbers.

Protobuf on raw TCP

Pre-gRPC pattern: length-delimited protobuf messages. gRPC standardized this with HTTP/2 headers + trailers.

Cap'n Proto / FlatBuffers

Zero-copy read — no decode step. Wins in games, browsers parsing maps. Ecosystem smaller than protobuf.

Security

Custom TCP protocols need TLS wrapper (or Noise protocol). Roll your own crypto = failure mode.

Interview framing

If asked "design a protocol":

1. Requirements (latency, reliability, browser?)
2. Pick TCP vs UDP (Networking)
3. Framing + schema (protobuf)
4. Auth, versioning, observability
5. Only then optimize

Demonstrates restraint — staff engineers choose boring technology.