Pipelines & ingestion

Batch processes finite datasets on a schedule for throughput; stream processing handles unbounded events continuously for low latency. When to use each.

A DAG models a pipeline as tasks (nodes) and dependencies (edges) with no cycles — so a valid run order always exists. Why orchestrators rely on it.

A data pipeline moves data from sources through ingest, transform, store, and serve — reliably, on a schedule or as a stream. The stages, batch vs streaming, and where pipelines rot.

CDC streams inserts, updates, and deletes out of a database as they happen — log-based, query-based, or trigger-based — so downstream systems stay in sync.

ETL extracts data, transforms it, then loads it. ELT loads raw first and transforms inside the warehouse. Why cheap cloud compute flipped the order, and where each still fits.

An idempotent step gives the same result whether it runs once or ten times — the property that lets a crashed, re-run pipeline stay correct instead of double-counting.