What is Apache Hudi?

Apache Hudi is an open table format that turns a directory of Parquet¹ files on object storage into a database-like table you can update, delete, and incrementally consume. It was built at Uber, and that origin shows: where the other open formats started from "make the lake queryable," Hudi started from "make the lake writable" — fast upserts and incremental processing were the founding problem, not a later feature. It is one of the three open table formats that define a data lakehouse², alongside Apache Iceberg³ and Delta Lake⁴.

🔗 Learn more — ¹ How Parquet works: columnar storage explained

🔗 Learn more — ² What is a data lakehouse?

🔗 Learn more — ³ How Apache Iceberg actually works

🔗 Learn more — ⁴ What is Delta Lake (and how does it compare to Iceberg)?

Built for the write path

Uber's problem was ingesting a firehose of updates — trips changing state, late-arriving corrections — onto a lake that was append-only by default. Rewriting whole partitions to apply a handful of changed rows is hopeless at that scale. Hudi's answer was the upsert: insert-or-update a record by key, applied efficiently, so a CDC⁵ stream from an OLTP⁶ database can land on the lake as it changes upstream.

🔗 Learn more — ⁵ What is Change Data Capture (CDC)?

🔗 Learn more — ⁶ OLTP vs OLAP: two opposite jobs

Two design choices make that work. The first is a record-level index that maps each record key to the file that holds it, so an incoming update goes straight to the right file instead of scanning the whole table. The second is the timeline — an ordered log of commits (and compaction, cleaning, and rollback actions) that is the table's source of truth. Every write produces a commit; readers see a consistent snapshot as of a commit; a failed write rolls back. The same timeline powers incremental queries: ask Hudi for "everything that changed since commit X" and it hands you just those records, which is exactly the primitive a downstream pipeline needs to avoid reprocessing the whole table.

Copy-on-write vs merge-on-read

Hudi makes you choose, per table, how updates physically land — and this is its most useful knob.

Copy-on-write (CoW) rewrites the affected Parquet file when a record in it changes. Writes cost more (you rewrite a whole file to change a few rows), but reads are plain Parquet scans with no merge step. CoW suits read-heavy, update-light tables.

Merge-on-read (MoR) writes updates to compact row-based delta logs next to the base Parquet, then merges them at read time (or asynchronously compacts them into new base files). Writes are cheap and low-latency — good for high-frequency ingestion — at the cost of readers doing merge work until compaction catches up. MoR is the choice when you are streaming many small updates and want fresh data fast.

flowchart TD
  WRITE["Incoming upsert keyed by record id"] --> IDX["Record-level index finds the file"]
  IDX --> COW["Copy-on-write rewrites the Parquet file"]
  IDX --> MOR["Merge-on-read appends to a delta log"]
  MOR --> COMPACT["Async compaction merges logs into base files"]
  COW --> TL["Commit recorded on the timeline"]
  COMPACT --> TL
  %% color = green: shared write path, amber: CoW choice, grey: MoR choice
  classDef green stroke:#a3be8c,stroke-width:2px
  classDef amber stroke:#ebcb8b,stroke-width:2px
  classDef grey stroke:#7b88a1,stroke-width:2px
  class WRITE,IDX,TL green
  class COW amber
  class MOR,COMPACT grey

The edges are literally true: an upsert is routed by the index, follows either the CoW or MoR path, and both end in a commit on the timeline; MoR's logs are later compacted into base files.

Why the write story is the strongest of the three

Of the three formats, Hudi's write and CDC story is arguably the most complete out of the box. The record-level index, native upserts and deletes, incremental queries, and managed table services (clustering, cleaning, compaction) were there early and integrated, rather than assembled from surrounding tooling. If your workload is mutation-heavy streaming ingestion — landing change data continuously and serving it fresh — Hudi was designed for exactly that shape, and it shows in how little you have to bolt on.

How it lost the format war anyway

Being the best-engineered write path did not win. The lakehouse format war turned on governance and neutrality more than raw capability, and Iceberg won that axis. Iceberg's specification, its broad multi-engine adoption, and its perception as vendor-neutral made it the default that data platforms standardized on — the format every engine and catalog agreed to support. Hudi's tighter coupling to its own write semantics, which is its strength, also made it feel less like a neutral interchange format and more like a system you adopt.

The honest, neutral read: Hudi led on the write path and streaming CDC, Iceberg led on being the format everyone could agree to point at, and the industry optimized for agreement. Hudi did not lose on merit alone. It has since added Iceberg-compatible output so a Hudi-managed table can be read by the Iceberg ecosystem — a pragmatic acknowledgement of where the gravity went, letting you keep Hudi's ingestion strengths while interoperating with the format that won the catalog.

The short version: Apache Hudi is the table format that took upserts, record-level indexing, and incremental processing seriously first — technically excellent on the write path, and a fair example of how standards win on neutrality, not just engineering.

Sources

• Apache Hudi documentation
• Apache Hudi — Wikipedia