This guide is part of Managing Spatial Assets with Dagster, within the broader Orchestrating Spatial ETL Pipelines reference.

Backfilling Satellite Scene Partitions in Dagster

The Problem: A Full Re-Run Is the Wrong Tool for a Historical Fix

Satellite archives grow one acquisition date at a time, and reprocessing needs are almost always bounded: a cloud-mask algorithm changed, a two-week gap appeared after an outage, or a CRS-registration fix must be applied to last quarter’s scenes. Re-running the entire pipeline to repair a window is slow, expensive, and risks re-touching scenes that were already correct.

Treating history as monolithic breaks scene pipelines in predictable ways:

• Cost scales with the archive, not the fix. Recomputing three years of daily scenes to correct one month burns compute proportional to the whole history instead of the affected window.
• No parallelism across dates. A single sequential re-run processes dates one after another, when the dates are independent and could fan out across workers.
• One bad scene aborts everything. Without per-partition isolation, a corrupt acquisition halfway through kills the run and loses the progress on every prior date.
• No record of what actually reprocessed. A blanket re-run leaves no per-date status, so you cannot prove which scenes are now current.

Version and Environment Compatibility

pip install "dagster>=1.7,<1.9" dagster-webserver "geopandas>=1.0" "shapely>=2.0" "pyarrow>=14"

How a Range Backfill Recomputes Only the Gaps

Recipe: A DailyPartitionsDefinition Scene Asset and a Range Backfill

The asset below is date-partitioned by acquisition day and carries a RetryPolicy, so any single failed scene retries with backoff rather than sinking the backfill. Reading context.partition_key selects exactly one date; the backfill then drives the range.

import logging
from dagster import (
    asset,
    AssetExecutionContext,
    DailyPartitionsDefinition,
    RetryPolicy,
    Backoff,
)
import geopandas as gpd
import shapely

logger = logging.getLogger(__name__)

scene_partitions = DailyPartitionsDefinition(start_date="2022-01-01")


@asset(
    partitions_def=scene_partitions,
    io_manager_key="geoparquet_io",
    group_name="scenes",
    # A failed date retries up to 3 times with exponential backoff, so a
    # transient download 429 does not fail the whole backfill.
    retry_policy=RetryPolicy(max_retries=3, delay=10, backoff=Backoff.EXPONENTIAL),
)
def scene_footprints(context: AssetExecutionContext) -> gpd.GeoDataFrame:
    """Process the satellite scenes for exactly one acquisition date.

    context.partition_key is the ISO date string; the backfill fans out one
    run per date across the requested window, recomputing only missing ones.
    """
    acq_date: str = context.partition_key
    context.log.info("Processing scenes for %s", acq_date)

    scenes = gpd.read_parquet(f"s3://spatial-lake/raw/scenes/{acq_date}.parquet")
    if scenes.crs is None or scenes.crs.to_epsg() != 3857:
        scenes = scenes.to_crs("EPSG:3857")

    # Vectorized validity repair before load (Shapely 2.0 array API).
    invalid = ~shapely.is_valid(scenes.geometry.values)
    if invalid.any():
        logger.warning("Repairing %d invalid footprints on %s", int(invalid.sum()), acq_date)
        scenes.loc[invalid, scenes.geometry.name] = shapely.make_valid(
            scenes.geometry.values[invalid]
        )

    scenes["acq_date"] = acq_date
    context.add_output_metadata({"acq_date": acq_date, "scenes": len(scenes)})
    return scenes

Launch the historical window as a backfill from the command line. Dagster fans out one run per date across the range and, when you choose the missing-only option, skips dates that already materialized successfully:

# Recompute the March 2023 window; --from/--to bound the partition range.
dagster asset backfill --select scene_footprints \
  --partition-range 2023-03-01...2023-03-31

To target only failed and never-run dates programmatically, filter the partition keys against materialization status before submitting the backfill:

from dagster import DagsterInstance

def missing_scene_dates(from_date: str, to_date: str) -> list[str]:
    """Return partition keys in the window that are unmaterialized or failed."""
    instance = DagsterInstance.get()
    all_keys = scene_partitions.get_partition_keys_in_range(
        # PartitionKeyRange is accepted directly by the definition helper
        partition_key_range=(from_date, to_date)  # type: ignore[arg-type]
    )
    materialized = instance.get_materialized_partitions(
        asset_key=scene_footprints.key
    )
    return [key for key in all_keys if key not in materialized]

Key Implementation Notes

• The RetryPolicy is what makes a wide backfill survivable. Satellite source APIs throttle; a per-partition exponential backoff absorbs transient 429s so one flaky date does not require re-launching the whole window.
• Missing-only backfills keep cost proportional to the gap. Because Dagster records per-partition materialization, a range backfill over a quarter that has two missing weeks recomputes two weeks of compute — not the quarter.
• Concurrency should be sized to the slowest dependency. A month-long daily backfill is 30 independent runs; left unbounded they can open 30 simultaneous PostGIS connections or 30 concurrent API pulls. Cap it (below) to the download or write path, not the core count.
• Failed partitions stay visible and re-targetable. After retries, any still-failed date remains marked failed in the backfill view, so you fix the root cause and re-launch a narrow backfill over just those keys.
• Reprojection and repair live inside the partition. Doing the to_crs and vectorized make_valid per date keeps each partition self-contained and deterministic, which is the property that lets any single date be recomputed in isolation.

Bound the fan-out with a run-level concurrency cap so the backfill throttles to the write path:

from dagster import define_asset_job

# Benchmarked against one PostGIS instance: 6 concurrent scene dates
# sustained the highest steady throughput; beyond ~10 the write lock
# contended and total wall-clock time increased.
scene_backfill_job = define_asset_job(
    "scene_backfill_job",
    selection="scene_footprints",
    config={"execution": {"config": {"multiprocess": {"max_concurrent": 6}}}},
)

Troubleshooting Scene Backfills

Integration Note

Backfilling closes the loop on the asset graph in managing spatial assets with Dagster: the same date-partitioned scene_footprints asset that a schedule advances one day at a time is what a backfill sweeps across history when an algorithm changes. Where the tile-partitioned asset recipe defines the spatial dimension of the grid, this date dimension defines the temporal one — and a MultiPartitionsDefinition combining both lets a backfill target a specific tile across a specific window, recomputing one corner of the space-time grid without disturbing the rest.

Related

• Managing Spatial Assets with Dagster — the asset graph, IO manager, and schedules that this scene asset backfills against
• Partitioning Spatial Assets by Tile in Dagster — the spatial partition dimension that pairs with date partitions for tile-and-time backfills
• Orchestrating Spatial ETL Pipelines — the reference model for partitioned, restartable spatial pipelines