Orchestrating Spatial ETL Pipelines

A spatial pipeline that runs correctly once from a notebook is not a pipeline — it is a demo. The moment the same logic must run every night against a moving archive of satellite scenes, recover cleanly from a rate-limited download halfway through, and never double-load a tile after a retry, the hard problem stops being the geometry and becomes the orchestration. For the GIS analysts and data engineers running production spatial systems, the orchestrator is what turns a collection of extract, clean, and load scripts into an auditable, restartable, observable data product.

Both the geospatial data ingestion and vector and raster cleaning references assume an orchestration layer underneath them — retries on the Extract step, checkpoints between stages, a scheduler that knows which windows have already been processed. This guide is that layer. It covers how to structure spatial ETL as directed acyclic graphs (DAGs), how to keep tasks idempotent when downloads fail mid-flight, how to partition raster and vector work so backfills are cheap, and how to choose between Airflow, Prefect, and Dagster for a workload defined by tiles, scenes, and coordinate systems rather than rows in a warehouse.

The Orchestrated Spatial Pipeline: A Reference Model

A spatial ETL DAG is not a linear script with a cron trigger bolted on. Each stage is a separately retryable, separately observable unit with an explicit contract for what it consumes and produces. The diagram below shows the canonical shape — the same five stages as any ingestion pipeline, but now every arrow is a durable handoff through object storage and every box is a task the scheduler can retry, skip, or backfill independently.

Orchestrated spatial ETL reference model A scheduler or sensor triggers a DAG of five tasks — Discover, Extract, Transform, Validate, Load — connected left to right. Each task persists its output to object storage before the next begins. The Extract task carries a retry loop; the Validate task branches failed records to a dead-letter store. Scheduler / Sensor Discover STAC · manifest Extract retry + backoff Transform CRS · repair Validate quality gates dead-letter store Load PostGIS · Parquet object storage — durable handoff between every task

Discover — Resolve which units of work exist for this run: STAC items in a time window, tiles intersecting an area of interest, or new files behind an ETag. Discovery emits a manifest, never the data itself, so the graph can be planned before a byte is downloaded. This is where detecting dataset changes with ETag and Last-Modified prevents re-processing unchanged sources.

Extract — Pull raw payloads. This is the only stage that should carry aggressive retries, because network failures are transient; a retry on bad data downstream just wastes compute. Extract writes raw bytes to object storage keyed by a stable identifier.

Transform — Normalize CRS, repair geometry, and align schemas against the raw bytes from the previous stage. Transform is stateless and deterministic so any single unit can be reprocessed in isolation during a backfill.

Validate — Run quality gates and route failures to a dead-letter store instead of crashing the run. A single malformed scene must not block the other 499.

Load — Write validated outputs to the analytical target — PostGIS, DuckDB, or partitioned GeoParquet — using upsert or atomic-replace semantics keyed on the same identifier used during Extract.

The three orchestrators covered here — Airflow, Prefect, and Dagster — express this model differently. Airflow encodes it as DAGs of operators on a fixed schedule; Prefect as Python-native flows with runtime fan-out; Dagster as partitioned, testable data assets. The orchestrator selection guide maps each to the spatial workloads it fits best.

Framework-Specific Orchestration Patterns

The five-stage model is universal; how you express it determines how cheap backfills are, how readable the DAG stays at fifty tasks, and how hard it is to test a single stage in isolation.

The boundary between orchestrator and spatial code Two responsibility columns separated by a contract. The orchestrator handles triggering, retry policy, concurrency limits, backfills and lineage. The spatial module handles reading, reprojecting, repairing and writing. The contract between them is deliberately narrow: paths and small metadata values, never GeoDataFrames, which is what keeps the spatial code testable and portable between tools. the orchestrator owns when a run starts, and on what trigger retry counts, backoff, alerting concurrency pools against rate limits backfill windows and partition state run history and lineage contract uri in uri out counts, not frames the spatial module owns reading the source format CRS normalization and geometry repair schema mapping and validation writing GeoParquet, COG or PostGIS its own unit tests, no scheduler needed Cross this line and the pipeline becomes untestable: geometry logic buried in an operator can only be exercised by running the scheduler.

Airflow — Operator DAGs on a Fixed Schedule

Airflow remains the default for pipelines that run on a predictable cadence and lean on a broad operator ecosystem (S3, GCS, Postgres, HTTP). For spatial work the two recurring pitfalls are non-idempotent tasks that double-load tiles on retry, and the temptation to shuttle GeoDataFrames through XCom. Both are solved by treating object storage as the handoff medium: see writing idempotent spatial ETL tasks in Airflow and passing GeoDataFrames between Airflow tasks for the concrete patterns. The full DAG structure — scheduling, sensors, and pools sized for rate-limited APIs — is covered in building Airflow DAGs for spatial ETL.

Prefect — Python-Native Flows with Runtime Fan-Out

When the number of units of work is only known at runtime — the count of STAC items a query returns, the tiles intersecting a shifting area of interest — Prefect’s .map() fan-out expresses the pipeline more naturally than static Airflow tasks. Orchestrating spatial pipelines with Prefect shows the flow structure, while caching geospatial task results in Prefect and parallel tile downloads with Prefect task mapping cover the two features that matter most for spatial fan-out: content-addressed caching and bounded concurrent mapping.

Dagster — Partitioned, Testable Spatial Assets

Dagster models each output as a data asset with an explicit partitioning scheme, which maps almost perfectly onto spatial data indexed by tile or scene. Managing spatial assets with Dagster covers the asset graph; partitioning spatial assets by tile in Dagster and backfilling satellite scene partitions in Dagster show how partitioning turns a full-history reprocess into a targeted, parallelizable backfill instead of a monolithic re-run.

Cross-Cutting Concerns

Three concerns cut across every framework and must be designed at the pipeline level, not patched into individual tasks.

Idempotency and Exactly-Once Loads

Retries are a feature, which means every task will eventually run more than once. A pipeline is only safe if a second run produces the same state as the first. The universal pattern is to key each write on a stable identifier and use upsert semantics:

import geopandas as gpd
from sqlalchemy import create_engine, text

def upsert_tiles(gdf: gpd.GeoDataFrame, engine, table: str, key: str = "tile_id") -> int:
    """Atomically replace rows for the tiles present in this batch.

    Idempotent: re-running with the same batch leaves the table unchanged
    rather than appending duplicate geometries.
    """
    tile_ids = gdf[key].unique().tolist()
    with engine.begin() as conn:  # single transaction — delete + insert are atomic
        conn.execute(
            text(f"DELETE FROM {table} WHERE {key} = ANY(:ids)"),
            {"ids": tile_ids},
        )
        gdf.to_postgis(table, conn, if_exists="append", index=False)
    return len(gdf)

The same principle governs file outputs: write to a .part path and rename atomically only after validation, so an interrupted task never leaves a half-written GeoParquet that the next run mistakes for complete.

Scheduling, Backfills, and Incremental Windows

Spatial archives grow along a time axis — daily satellite passes, weekly portal refreshes — so the scheduler’s job is to track which windows have been processed and to make reprocessing a bounded operation. Scheduling and incremental spatial loads covers watermark tracking and catch-up runs; the key discipline is that a backfill of last month must cost exactly the compute of last month’s data, which only holds if the pipeline is partitioned rather than monolithic.

Load-Target Selection

The Load stage’s target shapes the whole pipeline: PostGIS gives you transactional upserts and a mature spatial index, while DuckDB’s spatial extension gives you columnar scan speed over GeoParquet with no server to operate. The PostGIS versus DuckDB comparison for analytical loads works through the decision, and loading GeoDataFrames into PostGIS with GeoPandas shows the write path in detail.

Observability: Knowing the Output Is Right

Task status answers whether the code ran, not whether it produced anything useful. Monitoring and observability for spatial pipelines covers the metrics that catch a green run serving a week-old table: volume ratios, output and source freshness, geometry health, coverage and cost.

Where the Data Lands

Object storage accepts any key, so the layout exists only in the code that writes it. Storing spatial data in cloud object storage covers partition keys that match how data is read, object sizes that avoid both extremes, replace-not-append writes, and the dataset metadata that makes a bucket usable by someone who did not build it.

Validation and Quality Gates in a DAG

Orchestrated validation differs from script-level validation in one way: a failing record must not fail the run. Route it to a dead-letter store and let the graph continue, then alert on dead-letter volume rather than on individual failures.

Gate before the load, not after it Two task chains. In the upper chain the validation task sits between transform and load, so a failed check aborts the run and the serving table keeps its previous good state. In the lower chain validation runs after the load, so by the time the alert fires the corrupt rows are already visible to consumers and a rollback is required. gate inside the DAG transform validate load fail → run stops, table untouched gate after the load transform load validate alert fires, but the bad rows are already being queried consumers never see an inconsistent state Same checks, same code, different position in the graph — and only one of the two lets you sleep through the alert. 
import logging
import geopandas as gpd

logger = logging.getLogger(__name__)

def gate_and_split(
    gdf: gpd.GeoDataFrame,
    required: list[str],
) -> tuple[gpd.GeoDataFrame, gpd.GeoDataFrame]:
    """Split a batch into (passed, dead_letter) instead of raising.

    Lets an orchestrated task load the good rows and quarantine the rest,
    so one bad scene never blocks a 500-scene backfill.
    """
    import shapely

    valid_geom = shapely.is_valid(gdf.geometry.values) & ~gdf.geometry.isna().values
    has_cols = gdf[required].notna().all(axis=1).values if required else True
    keep = valid_geom & has_cols

    passed = gdf.loc[keep].copy()
    dead = gdf.loc[~keep].copy()
    if len(dead):
        logger.warning("Quarantined %d/%d features to dead-letter", len(dead), len(gdf))
    return passed, dead

The gate_and_split contract composes with the geometry checks from geometry repair with Shapely and GeoPandas — run the repair inside Transform, then split on the residual invalids here in Validate.

Failure-Mode Reference

Failure Mode Root Cause Mitigation Strategy
Duplicate geometries after a retry Task appends instead of upserting; retry re-runs the load Key writes on tile_id/scene_id; delete-then-insert in one transaction
Worker OOM passing data between tasks GeoDataFrame serialized through XCom or result store Persist to GeoParquet in object storage; pass only the URI
Backfill of one month re-runs full history Monolithic DAG with no partitioning Partition by date/tile so backfills target only affected partitions
Silent gap in a time series Scheduler has no record of which windows ran Track a watermark timestamp; a sensor or catch-up run fills gaps
Whole run fails on one corrupt scene Validation raises instead of quarantining Dead-letter the bad record; alert on dead-letter volume, not per-record

Production Integration Notes

Wire the orchestrator to the same object-storage layout the ingestion and cleaning workflows produce: partition GeoParquet by temporal granularity and a spatial index (H3 resolution 5 or quadkey depth 8) so both backfills and downstream analytical scans touch only the partitions they need. Size concurrency pools to the slowest external dependency — a rate-limited Overpass or STAC endpoint, not the worker count — so the orchestrator throttles OSM extraction and STAC catalog syncs within their rate envelopes rather than tripping HTTP 429s across a whole fan-out. Finally, emit structured logs at every task boundary with the partition key attached, so an operator can answer “did tile 85283473 load last night?” without reading a stack trace.

Sizing the Unit of Work

The single most consequential decision in a spatial DAG is what one task does. Get it wrong in one direction and every failure costs an hour of recomputation; wrong in the other, and the scheduler spends more time tracking tasks than the tasks spend working.

A useful unit of work has four properties. It is independently retryable — rerunning it changes nothing that another unit depends on. It is independently addressable — there is a natural key, usually a tile, a scene or a date, that names it. It runs in minutes, not hours, so a failure late in the run is cheap to repeat. And it fits in one worker’s memory, which for spatial data usually means a bounded number of features or a single raster window rather than “whatever the query returns”.

Tiles and scenes satisfy all four naturally, which is why tiled pipelines are so much easier to operate than whole-country ones. When the natural unit is too large — a national parcel dataset arriving as one file — introduce an artificial one by partitioning on a spatial index or an administrative boundary, then process partitions independently. The extra bookkeeping pays for itself the first time a single partition fails.

The opposite failure is worth naming too. A DAG with one task per feature turns a scheduler into a queueing system it was never designed to be; thousands of task instances each doing milliseconds of work produce enormous metadata overhead and a UI nobody can read. Batch small units until the per-task overhead is a small fraction of the per-task work.

Configuration, Secrets and Environment Parity

Spatial pipelines accumulate connection details faster than most: object storage credentials, database URIs, API keys for imagery providers, PROJ data directories, GDAL driver flags. How they are supplied determines whether the pipeline can be run anywhere except the machine it was written on.

Keep three categories apart. Secrets — keys, passwords, tokens — belong in the orchestrator’s secret backend or the platform’s secret manager, never in the DAG file and never in a .env committed alongside it. Environment bindings — which bucket, which database, which STAC endpoint — belong in per-environment configuration that the same code reads. Behavioural parameters — target CRS, simplification tolerance, tile grid size — belong in version control, because a change to one of them changes the output and therefore deserves a review.

The GDAL and PROJ environment deserves specific attention. A container missing its PROJ grid files silently produces coordinates tens of metres off, exactly as described in CRS normalization across mixed datasets. Pin the geospatial stack in the image, assert the PROJ version and grid availability at task startup, and treat that assertion as part of the pipeline rather than as an operations concern.

Parity between local, CI and production is what makes a pipeline debuggable. If the only place the code runs is the production scheduler, every investigation becomes an experiment on live data.

What to Alert On, and What to Merely Record

Alert fatigue kills more pipelines than bugs do. The distinction that works in practice is between conditions that require a human tonight and conditions that require a human eventually.

Page a human when the pipeline cannot make progress or has produced something wrong that consumers can see: a DAG that has not run at all, a load that failed after the target was partially written, a quality gate that failed on the serving path, an authentication failure that will not resolve itself.

Record and review everything else: a tile that failed and will retry, a quarantine rate that rose within its threshold, a run that took 40% longer than usual, a source that returned fewer features than yesterday. These belong in a daily digest and a dashboard, not in a phone notification at 03:00.

Never alert on conditions the pipeline handles by design. A 429 that backoff absorbed, a cache miss, a skipped partition where no scene existed — surfacing these trains people to ignore the channel.

The one addition worth making for spatial work specifically is a freshness alert on the output rather than the job. A DAG can succeed every night while serving a table nobody has updated in a week, because the source stopped publishing and every conditional request returned 304. Alert on the age of the newest row in the serving table and that entire class of silent failure becomes visible.

Migrating a Cron Script into an Orchestrated Pipeline

Most spatial pipelines start as a shell script and a crontab entry, and the migration is easier if it is done in four passes rather than one rewrite.

Pass one: make it a function. Move the script body into Python functions that take and return paths, with no scheduling logic and no global state. This is the step that pays off regardless of which tool you eventually pick, and it is the boundary drawn at the top of this section.

Pass two: make it idempotent. Write to a staging location and rename or upsert, key on the logical date rather than the wall clock, and confirm that running twice produces the same result. Writing idempotent spatial ETL tasks in Airflow works through the two mechanisms this needs.

Pass three: split it. Break the function into the units of work identified above, and only now introduce the orchestrator. A DAG is a poor place to discover that a step was not separable.

Pass four: add the operational surface. Retries, pools, alerts, quality gates and backfill support — the things a cron entry cannot express and the reason for the migration in the first place.

Running the old cron job and the new pipeline in parallel for a week, then diffing their outputs, is the cheapest possible validation and it catches the assumptions nobody wrote down.

Backfills as a First-Class Operation

Backfills are not an exception to the pipeline; they are the pipeline run over a different window, and treating them as a separate mode is how organisations end up with two code paths that drift apart.

Three properties make a backfill safe. The task must read its window from the run context rather than from the clock, so the same code produces March data in March and in November. The write must be scoped to that window, so re-running one day cannot disturb another. And the concurrency must be capped independently of the daily schedule, because fourteen partitions launching at once is a burst the source never sees during normal operation.

Given those, a backfill needs no special code at all — it is a range of ordinary runs, as backfilling satellite scene partitions in Dagster shows for the partitioned case. Without them, a backfill becomes a hand-written script that nobody trusts and everybody runs at midnight.

One further habit is worth adopting: record, per partition, the code version that produced it. When a transform changes, that column is what tells you which historical partitions are stale and need reprocessing — and, just as usefully, which ones do not.

Documenting the Pipeline Where Its Operators Will Look

A spatial pipeline is operated by people who did not write it, often at an hour when nobody wants to read code. Three short documents, kept next to the DAG rather than in a wiki, carry almost all of the load.

A runbook listing the three or four failures that actually recur, each with the check to run and the fix — the credential that expires, the source that publishes late, the disk that fills during a backfill. A data contract naming the output’s grain, its CRS, its update cadence and who consumes it, so an operator can judge whether a delay matters. And a recovery note stating explicitly how to re-run a window safely, including whether re-running is safe at all, which is the question a tired operator most needs answered.

None of this survives if it lives apart from the code. Kept in the same directory and reviewed in the same pull request, it stays roughly true; kept elsewhere, it describes a pipeline that stopped existing two releases ago.