Orchestration, catalogs & governance

A layered lakehouse pattern: bronze holds raw ingested data, silver is cleaned and conformed, gold is business-ready aggregates. Refinement flows upward.

Airflow schedules and monitors pipelines defined as DAGs of tasks in Python. It orchestrates work — deciding what runs, when, and in what order — without doing the heavy compute itself. How the scheduler, DAGs, operators, and executor fit together.

Data lineage is the recorded provenance of data — where a field came from, what transformed it, and what depends on it. It's how you do impact analysis and debug wrong numbers.

Dagster is a Python data orchestrator built on software-defined assets: you declare the data assets you want and it builds the DAG, lineage, and runs.

A data mesh decentralizes data ownership to the domain teams that produce it, treats datasets as products, and federates governance — an org model, not a tool.

Mage is an open-source data pipeline tool with a notebook-style, block-based editor where each runnable block forms the pipeline's DAG.

Project Nessie is an open-source transactional catalog that brings git-like branches, tags, and atomic commits to data lake tables.

Prefect is a Python-native orchestrator: plain functions become flows and tasks via decorators, with dynamic, runtime-determined workflows instead of a static DAG.

Temporal is a durable-execution platform where you write workflows as ordinary code that survives crashes and resumes exactly where it left off.

The Hive metastore maps table names to schemas, partitions, and file locations so engines can treat directories of files as SQL tables.

AWS Glue is Amazon's serverless data-integration service: a managed Hive-compatible Data Catalog, schema crawlers, and serverless Spark ETL jobs.