This guide sits under Choosing an Orchestrator for Spatial ETL, part of the broader Orchestrating Spatial ETL Pipelines reference. Where the parent guide weighs all three orchestrators, this one zooms in on the two most common finalists and settles them on one concrete workflow.

Airflow vs Prefect for Geospatial Workflows

The Problem: The Same Pipeline Feels Native in One Tool and Forced in the Other

A tile-download, reproject, and load pipeline is the “hello world” of spatial ETL — and it is exactly the workload where Airflow and Prefect diverge, because how many tiles exist is often not known until a discovery query runs. Pick the tool that matches that runtime property and the pipeline reads like three lines; pick the other and you spend the week working around its execution model.

The mismatch shows up in predictable ways:

• Unknown tile cardinality fights Airflow’s parse-time graph. Airflow builds its task graph when the DAG file is parsed. A tile count produced by a runtime query must be squeezed through dynamic task mapping, and the mapped values have to be small, serializable IDs — not tiles.
• Fixed schedules and mature operators are where Prefect asks more of you. If the real requirement is “run at 02:00, use the existing S3 and Postgres operators, and page someone on SLA miss”, Airflow already ships that; in Prefect you assemble more of it yourself.
• Rasters must never ride the orchestrator’s message bus. XCom and Prefect results are for metadata. Push a reprojected GeoTIFF through either and you get memory pressure and serialization errors, regardless of tool.
• Retries interact with idempotency. Both tools retry failed tasks, so a non-idempotent tile load double-writes on retry in either one — a correctness bug the orchestrator choice does not fix.

Version and Environment Compatibility

Where the Two Diverge on This Workflow

The diagram traces the identical three-stage task through each orchestrator, so the only difference visible is where the fan-out over tiles is decided — at graph-build time in Airflow, inside the running flow in Prefect.

The Same Tile Task in TaskFlow and Prefect

Below is one small spatial task — download a tile, reproject it with rasterio.warp, and load a footprint row keyed on tile_id — implemented conceptually in both orchestrators. The three inner functions are identical and pure; only the wrapper differs. Read the two blocks as the same pipeline, so the execution-model difference is the only variable.

# ---- Airflow: TaskFlow with dynamic task mapping (.expand) ----
from datetime import datetime
from pathlib import Path

import rasterio
from rasterio.warp import calculate_default_transform, reproject, Resampling
from airflow.decorators import dag, task

DST_CRS = "EPSG:3857"


def _reproject_tile(src_path: Path, dst_path: Path, dst_crs: str = DST_CRS) -> Path:
    """Pure reproject step shared by both orchestrators."""
    with rasterio.open(src_path) as src:
        transform, width, height = calculate_default_transform(
            src.crs, dst_crs, src.width, src.height, *src.bounds
        )
        profile = src.profile | {
            "crs": dst_crs, "transform": transform, "width": width, "height": height,
        }
        with rasterio.open(dst_path, "w", **profile) as dst:
            for band in range(1, src.count + 1):
                reproject(
                    source=rasterio.band(src, band),
                    destination=rasterio.band(dst, band),
                    resampling=Resampling.bilinear,
                )
    return dst_path


@dag(schedule="0 2 * * *", start_date=datetime(2026, 7, 1), catchup=False)
def tile_pipeline():
    @task
    def discover() -> list[str]:
        return list_tiles_for_aoi()  # runtime list, but resolved as a task

    @task(retries=3, retry_delay_seconds=30)
    def process(tile_id: str) -> str:
        raw = download_tile(tile_id, Path("/data/raw"))
        warped = _reproject_tile(raw, Path("/data/warp") / f"{tile_id}.tif")
        upsert_tile_footprint(tile_id, warped)  # idempotent load, keyed on tile_id
        return tile_id

    process.expand(tile_id=discover())  # fan-out width fixed when the graph is built


tile_pipeline()

# ---- Prefect: flow with runtime fan-out (.map) ----
from pathlib import Path

from prefect import flow, task

from etl.reproject import _reproject_tile, DST_CRS  # same pure function as above


@task(retries=3, retry_delay_seconds=30)
def process(tile_id: str) -> str:
    raw = download_tile(tile_id, Path("/data/raw"))
    warped = _reproject_tile(raw, Path("/data/warp") / f"{tile_id}.tif")
    upsert_tile_footprint(tile_id, warped)  # identical idempotent load
    return tile_id


@flow(name="tile-pipeline")
def tile_pipeline() -> list[str]:
    tile_ids: list[str] = list_tiles_for_aoi()  # ordinary Python value
    futures = process.map(tile_ids)  # fan-out width decided at run time
    return [f.result() for f in futures]


if __name__ == "__main__":
    tile_pipeline()  # runs locally with no scheduler or metadata database

Key Implementation Notes

• The pure task body is the portable core. _reproject_tile, download_tile, and upsert_tile_footprint never import Airflow or Prefect. That is what makes the two wrappers interchangeable and what turns a migration into a re-wrapping job rather than a rewrite.
• .expand() maps over IDs, .map() maps over a live list. Airflow’s mapped values are pulled from XCom and must stay small and serializable, so discover() returns tile_id strings, not tiles. Prefect’s .map() consumes the in-memory list directly, so an unknown or large tile count is an ordinary case.
• Retries are configured identically but do not grant idempotency. Both wrappers set retries=3; both will re-run process on failure. Only the upsert_tile_footprint keyed on tile_id makes that retry safe, which is why the load uses upsert rather than append semantics.
• Neither tool carries the raster. The reprojected GeoTIFF is written to disk or object storage inside the task; only the tile_id string crosses the task boundary. This keeps XCom and Prefect result storage within their intended size envelope.
• Local iteration cost differs sharply. The Prefect flow runs with a plain python tile_pipeline.py; the Airflow DAG needs a scheduler and metadata database (or airflow dags test) to exercise the mapping — a real factor when developing the spatial logic.

Troubleshooting Airflow vs Prefect Tile Workflows

Integration Note

This head-to-head is the practical companion to the decision framework in choosing an orchestrator for spatial ETL: once that guide narrows the field to Airflow or Prefect, the two blocks above are the concrete port. For the fixed-schedule branch, the full DAG structure — sensors, pools sized to rate-limited tile servers, and idempotent operators — is developed in building Airflow DAGs for spatial ETL, and the specific safety pattern for the load step appears in writing idempotent spatial ETL tasks in Airflow. For the runtime-fan-out branch, the bounded, throttled version of process.map() over thousands of tiles is covered in parallel tile downloads with Prefect task mapping.

Related

• Choosing an Orchestrator for Spatial ETL — the full decision framework that also weighs Dagster against these two
• Building Airflow DAGs for Spatial ETL — DAG structure, sensors, and pools for the fixed-schedule branch
• Orchestrating Spatial Pipelines with Prefect — flow structure and runtime fan-out for the dynamic branch
• Parallel Tile Downloads with Prefect Task Mapping — bounded, throttled .map() fan-out at tile scale