Communication & Data Transfer/Async Integration Patterns

Message-Based Integration

When to use broker vs webhook vs sync API, event notification vs state transfer, and AsyncAPI contracts — not Kafka partitions or outbox relay implementation (see Distributed Systems → Transactions & Messaging).

4/5Overview: 35m

Broker vs webhook vs direct HTTP

PatternBest for
WebhookB2B integrations, few subscribers, HTTP-native
Message brokerHigh volume, many consumers, replay, decoupling
gRPC/event streamInternal low-latency fan-out

Distributed Systems → Messaging & Streams owns Kafka partitions, consumer groups, offset commits, and delivery guarantees. Distributed Systems → Transactions owns saga choreography and transactional outbox relay/CDC in depth. This topic: which integration style to expose and how to document it (AsyncAPI).

Transactional outbox — one-paragraph bridge

When a service must atomically persist state and notify others, the outbox pattern (same DB txn + relay) is the standard answer — but the correctness proof, CDC vs polling relay, and interaction with exactly-once claims are Distributed Systems material. Here: recognize the pattern when choosing "HTTP callback vs broker after write" and point implementers to that track.

Event notification vs state transfer

Notification: { "type": "OrderPlaced", "orderId": "123" } — consumer calls API for details. Thin events, chatty.

State transfer: full order in event — fewer calls, fatter coupling, schema evolution harder.

Staff trade-off: notification + cached read models for large payloads; state transfer for high-volume read paths. Schema evolution rules overlap Topic 3 (JSON:API & Contracts) at the API boundary and Databases at persistence — not re-derived here.

AsyncAPI

Like OpenAPI for events: channels, messages, schemas, bindings (Kafka, AMQP, HTTP).

Enables codegen, documentation, and contract tests for event-driven APIs. Pair with Pact (Topic 3) for consumer-driven checks on event payloads.

Idempotent consumers — boundary only

API/event consumers should dedupe by event_id. How to implement idempotent writes (DB unique keys, MVCC conflicts) ties to Databases → Transactions; how brokers redeliver ties to Distributed Systems → Messaging.

Saga / choreography

Multi-service workflows via events — defer compensating transactions and orchestration vs choreography to Distributed Systems → Distributed Transactions.

Choosing async integration

Ask:

  • Can caller proceed without immediate answer?
  • How many subscribers?
  • Need replay for new consumers?
  • Cross-team schema governance?

If all yes → broker + schema registry. If one external partner → webhook.

Further Reading

Hands-On Tasks (Optional)

API design drills and whiteboard exercises — protocol selection, contract design, and bulk-transfer architecture. Assumes Networking and sibling tracks on the hub page (Distributed Systems, Databases, Concurrency, LLD).

  • Pick async integration for three flows

    For: (1) Stripe-style partner webhook, (2) 50 internal consumers on order events, (3) publish-after-DB-write — choose webhook, broker, or outbox and name which sibling track owns the hard correctness details.

    20m