Object & Blob Storage (S3, HDFS)

Object storage model

Object storage (S3, GCS, Azure Blob) treats data as immutable objects in flat buckets:

s3://bucket/key → bytes + metadata (Content-Type, tags) + version ID

No in-place update — overwrite = new object version. No POSIX seek — whole-object or ranged GET.

Properties that matter in design

Property	Implication
Durability	S3 11 nines — replicated across AZs
Consistency	Strong read-after-write for new objects; list eventual consistency historically
Latency	Higher than block storage; fine for analytics, bad for OLTP
Cost model	Cheap storage, egress and request charges dominate

Storage classes (Standard, IA, Glacier) trade access latency for cost — lifecycle policies automate tiering.

HDFS architecture (historical context)

Hadoop HDFS split large files into blocks (128–256 MB) across DataNodes, with a NameNode tracking metadata.

Client → NameNode (block map) → DataNodes (replicated blocks)

Strengths: colocate compute with data for MapReduce era. Weaknesses: NameNode SPOF (mitigated in HA), ops burden, not cloud-native elasticity.

Modern lakes: S3 as the filesystem (s3a://), EMR/Spark read directly — HDFS declining except legacy clusters.

Block vs object vs file

	Block (EBS)	File (NFS)	Object (S3)
Unit	Sector-aligned	Path hierarchy	Key in bucket
Mutate	Random write	Random write	Replace whole object
Typical use	DB data files	Shared config	Data lake, backups

RDBMS data files live on block storage; analytical dumps land in object storage.

Data lake pattern

Raw zone (immutable append), curated zone (cleaned Parquet), serving zone (warehouse external tables). Storage is decoupled from compute — spin up query engines against the same bucket.

Not covered here

Distributed filesystem theory (inodes, dentry cache) → OS → Filesystems. Pipeline orchestration (Airflow, Spark jobs) — out of scope for this track.

Where this goes next

Parquet & Lake Table Formats — columnar files and metadata layers that turn a bucket of objects into queryable tables.