Parquet & Lake Table Formats

Why columnar files

Row-oriented DB pages load all columns when you need one. Parquet (and ORC) store data column-by-column in immutable files — ideal for analytics scanning billions of rows × few columns.

Parquet file ├── Row group 1 │ ├── Column chunk: user_id (dictionary + RLE encoded) │ ├── Column chunk: event_type │ └── Column chunk: timestamp ├── Row group 2 └── Footer metadata (statistics, schema)

Parquet storage details

Feature	Benefit
Row groups	Parallel read units; predicate pushdown per group
Column statistics (min/max)	Skip entire row groups
Dictionary encoding	Low-cardinality strings compress heavily
RLE / bit-packing	Repetitive numeric sequences shrink

Files are immutable — updates = write new file + mark old invisible in table metadata.

The metadata problem on object storage

A directory of part-0001.parquet files has no ACID, no schema evolution, no snapshot isolation. Table formats add a metadata layer:

Format	Metadata store	Notable features
Apache Iceberg	Manifest files + catalog	Hidden partitioning, time travel, schema evolution
Delta Lake	Transaction log (_delta_log)	ACID, Z-order, Databricks integration
Apache Hudi	Timeline + file groups	Incremental processing, upserts

Iceberg snapshot model

Catalog (Glue, REST) → metadata.json (current schema, partition spec) → manifest list → manifests → data files (Parquet)

Snapshot = consistent view of file set at a point in time. Time travel = read older snapshot. Compaction merges small files — storage maintenance like LSM compaction.

Schema evolution

Add/drop/rename columns in metadata without rewriting all data files (defaults/nulls for old files). Contrast with ALTER TABLE rewriting a B-tree heap.

Partitioning vs bucketing

Partition pruning — year=2024/month=06/ path prefix skips irrelevant objects. Risk: too many small partitions → listing overhead. Iceberg hidden partitioning decouples physical layout from query predicates.

Lakehouse positioning

OLTP stays in Postgres; lake holds historical/analytical data in Parquet + Iceberg; query engines (Trino, Spark SQL, Snowflake external tables) compile SQL to file scans. Storage format choice outlives any single query engine.

Senior-level signal

"CSV on S3" vs "Parquet + Iceberg" — articulate column pruning, compression ratio, and snapshot isolation. That's the storage interview, not the Spark API.

Where this goes next

Elasticsearch & Columnar OLAP — interactive search and sub-second aggregation on columnar layouts.