Data Engineering Reference/Deep Cuts (Platform)

Data Mesh & Platform Boundaries

Domain ownership, data as a product, self-serve platform, federated governance — when mesh helps vs when it's overhead.

4/5Overview: 25m

Data mesh (four principles)

Zhamak Dehghani's data mesh:

  1. Domain ownership — teams own their domain data as products
  2. Data as a product — SLAs, discoverability, quality guarantees
  3. Self-serve data platform — paved road for producing/consuming
  4. Federated computational governance — global standards, local execution

Mesh vs centralized lakehouse

CentralizedData mesh
One platform team owns all pipelinesDomain teams publish data products
Faster initial velocityScales organizational complexity
Risk: platform bottleneckRisk: inconsistency, duplication

Reality at FAANG: hybrid — central platform provides Spark/Kafka/dbt infrastructure; domains own marts with contracts.

Data as a product

Each data product has:

  • Owner and on-call
  • SLA (freshness, quality)
  • Discoverable schema in catalog
  • Versioned interface (contract)

Not "dump tables in a shared bucket."

Platform team role

Build self-serve capabilities:

  • Ingest SDKs, CDC templates
  • dbt project templates
  • CI for data contracts
  • Cost attribution per domain

Not: every custom pipeline request.

When mesh is overkill

  • < 50 engineers, one data team
  • No domain-aligned product org
  • Immature catalog/contract tooling

Start centralized; evolve toward federated ownership as pain appears.

Interview answer template

"Central lakehouse platform with domain-owned gold marts. Payments publishes fct_transactions as a data product with contract and lineage. Platform provides medallion templates and Great Expectations CI — domains don't operate Kafka brokers."

Further Reading

Hands-On Tasks (Optional)

Pipeline design drills and whiteboard exercises — DAG sketches, partition plans, backfill strategies. Assumes Databases and SQL fundamentals are in place.

  • Mesh or centralized platform?

    200-person company with one data team vs 5k-person org with 40 domains. Argue for centralized lakehouse vs data mesh and name one risk of each.

    15m