This guide is part of Choosing an Orchestrator for Spatial ETL, within the broader Orchestrating Spatial ETL Pipelines reference.

Dagster vs Airflow for Partitioned Raster Workloads

A tiled raster pipeline is the workload where the two tools differ most, because its natural unit of work is a cell in a two-dimensional space and only one of them models that directly.

The Workload

Four hundred tiles, one product per day, each tile rendered from the scenes intersecting it. Roughly 146 000 cells a year. Corrections arrive as individual reprocessed scenes affecting a handful of tiles on one date. That shape — large partition space, sparse targeted rebuilds — is what separates the tools.

How Each Models the Partition Space

The smallest independently selectable unit in each tool On the left Dagster's multi-partition space is drawn as a grid of tile by date cells, any one of which can be selected and materialised alone. On the right Airflow's model is drawn as a run per date containing mapped tasks per tile, so selection happens at the run level and the tile filter is a parameter within it. Dagster · tile × date cells select one cell · materialise one cell status, freshness and lineage are per cell Airflow · runs containing mapped tasks run · 2026-04-18 run · 2026-04-19 select a run · filter tiles inside it status is per run; per-tile state is in the task log

What a Targeted Backfill Costs

One scene is reprocessed. It affects three tiles on one date.

In Dagster, the affected cells are selected directly and materialised: three runs, three tiles rendered. The UI then shows those three cells as fresh and every other cell as untouched, which is also the audit answer.

Rebuild cost for one corrected scene Two bars showing tiles re-rendered to correct a single reprocessed scene affecting three tiles. Cell-level selection renders three. Re-running the whole day's DAG renders four hundred. Parameterising the tile list brings Airflow close to the cell-level figure at the cost of a run whose identity no longer describes what it produced. one scene corrected · tiles re-rendered cell-level selection 3 re-run the day's DAG 400 re-run with a tile filter 3, plus machinery you maintain The third row is the one most teams end up at; the question is whether the machinery is worth building or inheriting.

In Airflow, the smallest natural unit is the DAG run for that date. Clearing and re-running it re-renders all four hundred tiles unless the tile list is parameterised, and parameterising it means the run’s identity no longer describes what it produced. Teams solve this — a tiles parameter, a separate repair DAG, a sensor over a work queue — and each solution is a small amount of machinery that Dagster does not need.

Dagster Airflow
Smallest selectable unit one tile × date cell one DAG run (a date)
Targeted rebuild of 3 tiles 3 cells materialised run re-executed, tiles filtered by parameter
Per-cell freshness in the UI native via task instance history
146 000 partitions per year supported, UI paginates expressed as 365 runs × mapped tasks
Lineage between assets first-class inferred from task dependencies

Where Airflow Is the Better Answer

The partition model is one axis, and it is not always the decisive one.

Operator ecosystem. A pipeline that also moves data between an FTP drop, a legacy Oracle instance and an SFTP handover finds those integrations ready-made in Airflow and hand-written elsewhere.

Existing operational knowledge. A team already running Airflow for twenty non-spatial pipelines has monitoring, alerting, on-call familiarity and deployment tooling in place. Adding a second orchestrator has a cost that no partition model repays quickly.

Scheduling complexity. Cross-DAG dependencies, external triggers and elaborate calendar logic are well-trodden in Airflow and less so in an asset-oriented model that prefers to express dependencies through data.

The honest summary is that Dagster fits the shape of tiled spatial work better, and Airflow fits the organisation better more often than its critics admit.

The Cost of Moving

Migration cost is dominated by how well the spatial code was isolated in the first place. Where download, reproject, mask and write are pure functions over paths — the boundary argued for in orchestrating spatial ETL pipelines — the move is a re-wrapping exercise measured in days, and the same functions run under either tool.

Where geometry logic lives inside operators, reads connection objects from the orchestrator, or depends on XCom shapes, the move is a rewrite. That is a reason to fix the coupling regardless of whether the migration happens, because the same isolation is what makes the pipeline testable.

What decides the cost of switching tools Two codebases compared for portability. The first keeps download, reproject and write as pure functions with the orchestrator only wrapping them, so a migration replaces the wrappers. The second embeds geometry work inside operators and depends on the orchestrator's connection objects and message passing, so a migration rewrites the pipeline. isolated · a port replaces the wrappers orchestrator wrappers — rewritten download · reproject · mask · write pure functions — untouched days coupled · a port is a rewrite geometry inside operators connections read from the scheduler frames passed through XCom all of it rewritten months The same isolation that makes the port cheap is what makes the pipeline testable, which is the argument for doing it whether or not a migration ever happens.

Two things do not port and have to be rebuilt: partition history, which lives in the orchestrator’s own store, and any UI-driven operational habit that a team has grown around. Budget for both, and expect the second to generate more friction than the first.

Troubleshooting the Decision

Symptom of a bad fit What it indicates Consider
Every correction re-renders hundreds of tiles Partition granularity too coarse Cell-level partitions, or a parameterised repair path
Nobody can say which tiles are stale Freshness tracked per run, not per unit Asset-level freshness, or a coverage table you maintain
A repair DAG exists alongside the main one The tool’s unit does not match the work Align the partition key with how errors arrive
The scheduler is slow with many mapped tasks Fan-out too fine for the run model Batch tiles per task; or move the fan-out into the partition space
Two orchestrators in one team Chasing a model rather than a problem Weigh the operational cost honestly

Integration Note

Whichever tool runs it, the partition key should match the storage layout exactly — one cell, one object prefix — as described in partitioning GeoParquet by region and date. That alignment is what makes a rebuild idempotent and a lineage question answerable, and it is worth more than the choice of orchestrator itself.

Reporting a Partial Failure

Twelve tiles out of four hundred fail because a source scene was unavailable. How that is reported is a daily operational concern rather than an architectural one, and the tools differ.

In an asset model the twelve cells stay unmaterialised and remain visibly stale in the catalogue until they succeed. Nothing needs to be remembered by a person: the gap is the state, and a retry targets exactly the missing cells.

In a run model the DAG finishes with twelve failed task instances and a red run. The information is present, and acting on it means reading the task list to recover which tiles failed and then arranging for those to be retried — usually by clearing the individual mapped instances, which works well and is a manual step.

Neither is wrong. The distinction that matters is whether a partial failure leaves behind a machine-readable work list or a human-readable one, because the first can drive an automatic repair pass and the second cannot.

Running Both

A team already invested in Airflow and drawn to asset-level partitions does not have to choose all at once. Two hybrid arrangements are common and neither is a compromise in the pejorative sense.

Airflow schedules, the asset layer materialises. A thin DAG triggers a materialisation for a date’s partitions and waits for it. Airflow keeps the calendar, the connections and the alerting the organisation already knows; the asset layer keeps per-cell state and backfills. The cost is one integration point and two UIs.

Asset semantics without the second tool. Much of what the asset model provides can be built on top of Airflow: a partition table listing the cells and their last successful run, a sensor that enqueues stale cells, and a mapped task that processes exactly the enqueued list. This is a few hundred lines and it is a real system to maintain, but it fits neatly into a stack nobody wants to expand.

The choice between them is mostly about where the team would rather spend maintenance: on an integration with a second tool, or on a small amount of home-grown partition bookkeeping.

Concurrency Against a Rate-Limited Source

Raster pipelines pull from providers with request budgets, and both tools bound concurrency — differently enough to matter when a backfill runs alongside the daily schedule.

Airflow expresses the limit as a pool: a named allowance that every task instance draws from, regardless of which DAG or run it belongs to. That is exactly the right shape for a provider limit, because the constraint belongs to the endpoint rather than to any one pipeline, and a backfill competing with the nightly run shares the same pool automatically.

Dagster expresses it through run and tag-based concurrency limits, which achieve the same bound and are configured per deployment rather than per resource. The practical difference is that a provider allowance shared by several pipelines maps a little more directly onto Airflow’s pool, and a per-asset limit maps more directly onto Dagster’s tags.

Either way the number should come from measurement rather than intuition — the knee in the runtime curve rather than the largest value the client accepts — and it belongs in configuration next to the endpoint definition, so that a provider raising your allowance is a config change rather than a code change.