Storage & table formats

Iceberg turns thousands of immutable Parquet files in object storage into something that behaves like a database table — atomic commits, time travel, schema evolution. It does it with a tree of metadata files and one atomic pointer swap. A walk through the actual mechanics.

Tabular data is rows and columns — but how you serialize it to a file matters more than people think. CSV and TSV and their traps, JSON and JSONL, and why columnar binary formats like Parquet exist at all.

Parquet is the file format under almost every modern data lake. It stores data by column instead of by row, which is why analytics queries on it are fast. A look at the actual file anatomy — row groups, column chunks, pages, and the footer that makes predicate pushdown possible.

Apache Hudi is an open table format built at Uber for fast upserts and incremental processing on a lake — strong write path, lost the format war on neutrality.

Avro is a row-oriented binary format with a schema attached — built for moving and evolving records. Parquet is columnar for analytics at rest. They solve opposite problems.

Partitioning splits a dataset into chunks by a key so queries can skip the parts they don't need. Done right it's the biggest scan win; done wrong it makes tiny files.

Delta Lake is an open table format that adds ACID, time travel, and schema enforcement to a data lake. How it compares to Apache Iceberg.