Data & lakehouse

Schema-on-write enforces structure at ingest (warehouses); schema-on-read stores raw and applies structure at query (lakes). Flexibility vs guarantees.

SQL databases are relational, schema-enforced, and strongly consistent. NoSQL is a grab-bag of non-relational stores built for scale and flexible schemas. Pick by access pattern.

ACID is the four guarantees a database transaction makes — Atomicity, Consistency, Isolation, Durability — and why they get hard in distributed systems.

Batch processes finite datasets on a schedule for throughput; stream processing handles unbounded events continuously for low latency. When to use each.

A DAG models a pipeline as tasks (nodes) and dependencies (edges) with no cycles — so a valid run order always exists. Why orchestrators rely on it.

A data pipeline moves data from sources through ingest, transform, store, and serve — reliably, on a schedule or as a stream. The stages, batch vs streaming, and where pipelines rot.

CDC streams inserts, updates, and deletes out of a database as they happen — log-based, query-based, or trigger-based — so downstream systems stay in sync.

ETL extracts data, transforms it, then loads it. ELT loads raw first and transforms inside the warehouse. Why cheap cloud compute flipped the order, and where each still fits.

An idempotent step gives the same result whether it runs once or ten times — the property that lets a crashed, re-run pipeline stay correct instead of double-counting.

Apache Hudi is an open table format built at Uber for fast upserts and incremental processing on a lake — strong write path, lost the format war on neutrality.

Avro is a row-oriented binary format with a schema attached — built for moving and evolving records. Parquet is columnar for analytics at rest. They solve opposite problems.

Partitioning splits a dataset into chunks by a key so queries can skip the parts they don't need. Done right it's the biggest scan win; done wrong it makes tiny files.

Delta Lake is an open table format that adds ACID, time travel, and schema enforcement to a data lake. How it compares to Apache Iceberg.

Amazon Kinesis is AWS's managed streaming family: Data Streams (a sharded log), Firehose (managed delivery), and Managed Service for Apache Flink.

Kafka is a distributed, append-only log you can publish events to and replay later. Not a queue and not a database — a durable commit log that decouples the systems producing data from the ones consuming it. How the log, partitions, and consumer groups fit together.

Apache Beam is a unified model for batch and streaming: write one pipeline and run it on a pluggable runner like Dataflow, Flink, or Spark.

Apache Flink is a distributed engine for stateful stream processing: true record-at-a-time streaming, event-time with watermarks, exactly-once state.

Apache Pulsar is a distributed messaging and streaming platform that splits serving (brokers) from storage (BookKeeper), with multi-tenancy built in.

Hadoop launched the big-data era with HDFS, MapReduce, and YARN. Foundational, but largely superseded by Spark, S3, and cloud warehouses.

RabbitMQ is a smart message broker that routes messages through exchanges to queues and deletes them once consumed — unlike Kafka's replayable log.

Redpanda is a Kafka-API-compatible streaming platform in C++ with no JVM and no ZooKeeper, built for lower latency and simpler operations.

Spark is a distributed compute engine that runs one logical query across a cluster of machines. It builds a lazy plan, splits it into parallel tasks, and shuffles data between stages. How the driver, executors, and DAG actually fit together.

A materialized view stores the precomputed result of a query as a real table, so reads are instant — at the cost of keeping it refreshed as the source changes.

A query engine plans and executes queries over data it doesn't own — decoupling compute from storage. Trino and Presto run SQL across lakes and many sources at once.

Apache Druid is a real-time OLAP datastore for sub-second queries on high-volume event data, blending streaming ingestion with columnar storage.

Apache Pinot is a real-time distributed OLAP datastore built for ultra-low-latency, high-concurrency analytics that powers user-facing dashboards.

ClickHouse is an open-source columnar OLAP database in C++ built for very fast analytical queries over large tables via vectorized execution.

StarRocks is an open-source MPP analytical database with vectorized execution and a cost-based optimizer, fast at multi-table joins on its own storage and on open lakehouse tables.

Normalization removes redundancy so each fact lives in one place; denormalization duplicates it to avoid joins. The tradeoff is writes-and-integrity vs reads-and-speed.

Dimension attributes change over time. SCD is how a warehouse handles that: Type 1 overwrites, Type 2 keeps history as versioned rows, Type 3 keeps the previous value.

A star schema puts a central fact table of measurable events around dimension tables of context — denormalized so analytics queries stay fast.

dbt runs the T in ELT inside the warehouse: you write SELECT statements as models, and dbt orders dependencies, materializes tables, tests, and docs.

Dataform is a SQL-based transformation framework, now part of Google Cloud, for managing ELT inside BigQuery with SQLX models, ref dependencies, and tests.

SQLMesh parses your SQL to understand columns and dependencies, and uses virtual environments to preview changes without rebuilding production tables.

BigQuery is Google Cloud's serverless data warehouse: no clusters to manage, queries run across a shared pool, with scan-based pricing to watch.

Databricks is a cloud data and AI platform from the creators of Apache Spark, built around the lakehouse: Spark compute over open table formats.

AWS's columnar MPP data warehouse — a provisioned-cluster design forked from PostgreSQL, now with RA3 managed storage and a serverless option.

Snowflake is a fully-managed cloud data warehouse that separates storage from compute, with per-query virtual warehouses that scale on demand.

A layered lakehouse pattern: bronze holds raw ingested data, silver is cleaned and conformed, gold is business-ready aggregates. Refinement flows upward.

Airflow schedules and monitors pipelines defined as DAGs of tasks in Python. It orchestrates work — deciding what runs, when, and in what order — without doing the heavy compute itself. How the scheduler, DAGs, operators, and executor fit together.

Data lineage is the recorded provenance of data — where a field came from, what transformed it, and what depends on it. It's how you do impact analysis and debug wrong numbers.

Dagster is a Python data orchestrator built on software-defined assets: you declare the data assets you want and it builds the DAG, lineage, and runs.

A data mesh decentralizes data ownership to the domain teams that produce it, treats datasets as products, and federates governance — an org model, not a tool.

Mage is an open-source data pipeline tool with a notebook-style, block-based editor where each runnable block forms the pipeline's DAG.

Project Nessie is an open-source transactional catalog that brings git-like branches, tags, and atomic commits to data lake tables.

Prefect is a Python-native orchestrator: plain functions become flows and tasks via decorators, with dynamic, runtime-determined workflows instead of a static DAG.

Temporal is a durable-execution platform where you write workflows as ordinary code that survives crashes and resumes exactly where it left off.

The Hive metastore maps table names to schemas, partitions, and file locations so engines can treat directories of files as SQL tables.

AWS Glue is Amazon's serverless data-integration service: a managed Hive-compatible Data Catalog, schema crawlers, and serverless Spark ETL jobs.