This guide is part of Managing Spatial Assets with Dagster, within the broader Orchestrating Spatial ETL Pipelines reference.

Partitioning Spatial Assets by Tile in Dagster

The Problem: An Unpartitioned Tile Asset Rebuilds the Whole Mosaic

A raster or vector product tiled across a coverage grid — MGRS squares, XYZ quadkeys, a national grid — is naturally a collection of independent cells. Model it as a single unpartitioned Dagster asset and you lose that independence: every materialization touches the entire grid, and Dagster has no way to tell you which tile is stale.

This breaks tiled spatial pipelines in specific ways:

  • One changed tile forces a full rebuild. Fixing the geometry in tile T31UDQ reprocesses all 400 tiles because the asset is atomic — there is no partition to target.
  • No per-tile freshness. The UI shows one materialization timestamp for the whole product, so you cannot see that half the mosaic is a week stale.
  • Concurrency is coarse. Without partitions, Dagster runs the asset as one unit; you cannot fan a backfill across tiles or cap per-tile concurrency against a rate-limited source.
  • Failures are all-or-nothing. A single corrupt tile aborts the run, and the run either succeeds wholly or fails wholly, with no record of which cells completed.

Version and Environment Compatibility

Dagster Tile partitioning API Notes
1.7.x – 1.8.x StaticPartitionsDefinition, MultiPartitionsDefinition, DynamicPartitionsDefinition Recommended; stable multi-dimensional keys and per-partition UI
1.6.x StaticPartitionsDefinition, MultiPartitionsDefinition Stable partitions; asset-check blocking still maturing
1.5.x StaticPartitionsDefinition, MultiPartitionsDefinition Works, but pin exactly — surrounding asset-check APIs were experimental 
pip install "dagster>=1.7,<1.9" "geopandas>=1.0" "shapely>=2.0" "pyarrow>=14"

How a Tile Partition Selects One Cell

Tile partition key routing to a single asset run A partition set of three tile keys. Selecting the key T31UEQ launches one asset run that reads context.partition_key, processes only that tile, and writes one GeoParquet output, leaving the other tiles untouched. partition set T31UDQ T31UEQ (selected) T32ULV launch key asset run tile = context.partition_key read → process → write T31UEQ.parquet only this tile written other tiles remain untouched — independent materialization & freshness per cell

Recipe: A Tile-Partitioned @asset That Reads context.partition_key

The function below is a complete, copy-pasteable tile-partitioned asset. It builds a tile grid, registers it as a StaticPartitionsDefinition, and inside the asset reads context.partition_key to process exactly one tile — clipping the source to that tile’s bounds and writing a single GeoParquet output.

import logging
from dagster import asset, AssetExecutionContext, StaticPartitionsDefinition
import geopandas as gpd
import shapely

logger = logging.getLogger(__name__)


def build_tile_grid(index_uri: str) -> list[str]:
    """Enumerate tile identifiers from a coverage index (fallback to a fixed set)."""
    try:
        grid = gpd.read_parquet(index_uri)
        return sorted(grid["tile_id"].astype(str).unique().tolist())
    except (FileNotFoundError, OSError):
        logger.warning("Tile index %s unavailable; using built-in coverage", index_uri)
        return ["T31UDQ", "T31UEQ", "T32ULV"]


# Tile grid resolved once at import time; add tiles here and history is preserved.
tile_partitions = StaticPartitionsDefinition(build_tile_grid("s3://spatial-lake/index/tiles.parquet"))


@asset(partitions_def=tile_partitions, io_manager_key="geoparquet_io", group_name="tiles")
def clipped_tiles(context: AssetExecutionContext) -> gpd.GeoDataFrame:
    """Materialize a single tile: read only that tile's key and clip source data to it.

    Reading context.partition_key guarantees this run processes exactly one cell,
    so a rebuild of one tile never touches the rest of the mosaic.
    """
    tile_id: str = context.partition_key  # single-dimension key -> tile string
    context.log.info("Materializing tile %s", tile_id)

    tile_bounds = gpd.read_parquet(
        "s3://spatial-lake/index/tiles.parquet"
    ).set_index("tile_id").loc[[tile_id]]

    source = gpd.read_parquet("s3://spatial-lake/raw/features.parquet")
    if source.crs != tile_bounds.crs:
        source = source.to_crs(tile_bounds.crs)

    # Vectorized spatial filter to this tile's geometry (Shapely 2.0 predicate).
    tile_geom = tile_bounds.geometry.iloc[0]
    within_mask = shapely.intersects(source.geometry.values, tile_geom)
    clipped = source.loc[within_mask].copy()
    clipped["tile_id"] = tile_id

    context.add_output_metadata({"tile_id": tile_id, "features": len(clipped)})
    return clipped

To track each tile and each acquisition date independently, promote the definition to a MultiPartitionsDefinition whose dimensions are the tile grid and a DailyPartitionsDefinition, then read context.partition_key.keys_by_dimension inside the asset:

from dagster import DailyPartitionsDefinition, MultiPartitionsDefinition

tile_date_grid = MultiPartitionsDefinition(
    {
        "tile": tile_partitions,
        "date": DailyPartitionsDefinition(start_date="2024-01-01"),
    }
)

# Inside the asset body:
#   keys = context.partition_key.keys_by_dimension
#   tile_id, date = keys["tile"], keys["date"]

Key Implementation Notes

  • context.partition_key is the whole contract. The single line tile_id = context.partition_key is what makes the asset process one cell. Everything downstream — the read path, the clip, the output key — derives from it, so the same asset body serves every tile without branching.
  • Generate the grid from a source of truth. Building tile_partitions from an index file rather than a hard-coded list means adding coverage is a data change, not a code change, and the fallback list keeps local tests runnable offline.
  • Use the vectorized Shapely predicate, not a per-row loop. shapely.intersects(source.geometry.values, tile_geom) filters the whole frame in one array operation; applying a predicate per geometry would dominate runtime on dense tiles.
  • Adding tiles preserves history; removing them does not delete data. A StaticPartitionsDefinition treats new keys as unmaterialized and drops removed keys from the active set only — the already-written GeoParquet for a removed tile stays in object storage until you prune it.
  • Reach for DynamicPartitionsDefinition only when tiles are unknown ahead of time. If a new area of interest introduces quadkeys you cannot enumerate at import, register them at runtime with instance.add_dynamic_partitions rather than editing the static list.

Troubleshooting Tile Partitioning

Failure Root Cause Fix
context.partition_key returns an unexpected object Asset uses a MultiPartitionsDefinition but code reads it as a string Read context.partition_key.keys_by_dimension and index by dimension name
New tiles never materialize Static list built at import from a stale index snapshot Regenerate the partition list from the current index; restart the code location
Backfill launches one run for all tiles Asset is unpartitioned or partitions_def omitted Attach partitions_def=tile_partitions so each tile is a separate run
Empty output for a valid tile Source CRS differs from tile-grid CRS, so the intersect mask is empty Reproject source to the tile-grid CRS before the vectorized intersects filter
DagsterUnknownPartitionError on materialize Requested key is not in the current partition set Confirm the tile ID exists in the grid, or register it as a dynamic partition

Integration Note

This tile-partitioned asset is the middle of the graph described in managing spatial assets with Dagster: the clipped tiles feed a validated then loaded asset, and the per-tile keys let the CRS and geometry asset checks report results per cell. Once the grid is partitioned by tile, extending the second dimension to date turns a coverage grid into a time series, which is exactly what the scene-partition backfill workflow reprocesses across a historical window — tile partitioning defines the where, and scene backfilling drives the when.