← Learn··Updated 18 Jun 2026·3 min read

What is Prefect?

Prefect is a Python-native orchestrator: plain functions become flows and tasks via decorators, with dynamic, runtime-determined workflows instead of a static DAG.

Data & lakehouse
#data
#orchestration
#ai-assisted

Prefect is a Python-native workflow orchestrator where ordinary Python functions become flows and tasks just by adding a decorator. You write the logic you would write anyway — call an API, transform a dataframe, load a table — wrap the entry point with @flow and the units inside it with @task, and Prefect handles scheduling, retries, state tracking, and observability around that code. There is no separate YAML config, no DSL, and crucially no requirement to declare the whole workflow as a graph up front.

That last point is the design decision that sets Prefect apart from most orchestrators. Where many tools ask you to assemble a DAG — a directed acyclic graph1 of steps — before anything runs, Prefect builds the workflow as it executes. The shape of the run is determined at runtime, from real data, not frozen at definition time.

🔗 Learn more — 1 What is a DAG (and why orchestrators use them)?

The decorator model

In practice a Prefect workflow is just a Python script with two decorators sprinkled in. A function marked @flow is the orchestrated entry point; functions it calls that are marked @task become individually tracked units of work with their own retry and caching behavior. A flow can call tasks, call other flows as subflows, and call plain un-decorated helpers in between. You are not learning a framework's vocabulary for "operators" and "sensors" — you are writing Python, and the decorators are the only seam between your code and the orchestrator.

This matters most for adoption. An existing data pipeline2 written as Python functions can often be brought under orchestration by decorating a handful of functions, rather than being rewritten into a tool-specific node graph.

🔗 Learn more — 2 What is a data pipeline?

Dynamic, runtime-determined workflows

Because the graph is not declared ahead of time, normal Python control flow works inside a flow. You can branch with if/else on a value that only exists at runtime, loop with for or while to fan out a task across however many items a query returned this morning, and spawn more work conditionally based on what earlier tasks produced. The number of tasks and the branches taken can differ on every run.

%% color=green: branch chosen at runtime from real data
flowchart TD
    START["@flow run starts"] --> FETCH["@task fetch list of files"]
    FETCH --> CHECK{"files found?"}
    CHECK -->|"yes, N files"| LOOP["@task process each file (fan-out, N unknown until now)"]
    CHECK -->|"none"| SKIP["@task log empty + exit"]
    LOOP --> LOAD["@task load results"]

    classDef plain stroke:#7b88a1,stroke-width:2.5px
    classDef green stroke:#a3be8c,stroke-width:2.5px
    class CHECK,LOOP green
    class START,FETCH,SKIP,LOAD plain

A statically-declared DAG cannot express "process N files" when N is unknown until the flow is already running, without awkward workarounds. In Prefect this is just a loop.

Retries, caching, and state

Each flow and task tracks its own state — pending, running, completed, failed, retrying — and that state is what the orchestrator reasons about. Retries are a decorator argument: @task(retries=3, retry_delay_seconds=10) reruns on failure with backoff, no boilerplate. Caching lets an expensive task skip recomputation when its inputs (via a cache key) have not changed, so an interrupted run can resume from the last successful point instead of starting over. These primitives — typed states, retries, caching — are the substance of what you get for adding the decorators.

Hybrid execution and the honest caveats

Prefect's deployment model is hybrid: your code runs on your own infrastructure — your containers, your VMs, your Kubernetes — while the Prefect control plane handles scheduling, the API, and observability. Your data and execution stay in your environment; only orchestration metadata talks to the control plane. That suits teams who want managed orchestration without shipping their data through a vendor.

Fairly stated, Prefect is lighter and more genuinely Pythonic than Apache Airflow3, and its dynamic model is a real advantage when workflows branch and fan out on runtime data. But it is less asset- and lineage-focused than Dagster4: Prefect orchestrates function runs, whereas Dagster centers on the data assets a pipeline produces, which some teams want as the first-class concept. And the history is not painless — the 1.x to 2.x rewrite, and the later move to 3.x, changed APIs and concepts enough to be disruptive for early adopters who had to migrate. If you value Python ergonomics and dynamic flows over a strict asset graph, Prefect is a strong fit; if lineage and asset materialization are your center of gravity, weigh Dagster too.

🔗 Learn more — 3 What is Apache Airflow?
🔗 Learn more — 4 What is Dagster?

Sources