This guide is part of Orchestrating Spatial Pipelines with Prefect, within the broader Orchestrating Spatial ETL Pipelines reference.

Caching Geospatial Task Results in Prefect

The Problem: Every Re-Run Recomputes Reprojections That Never Changed

A spatial flow that reprojects and downloads the same tiles every night wastes most of its compute on work whose output is byte-for-byte identical to the previous run. Reprojecting a Sentinel-2 tile to a target CRS is a pure function of the source pixels and the transform; if neither changed, recomputing it is pure waste. Prefect can skip that recompute with a cache key — but the default key strategy misfires on spatial inputs in specific, costly ways:

• task_input_hash hashes unhashable or volatile arguments. Pass an open rasterio dataset, a pyproj.Transformer, or a GeoDataFrame and the hash either raises or changes every run, so the cache never hits.
• Identical bounds do not mean identical data. A government portal can republish a tile at the same bbox with corrected pixels; a bbox-only key would serve a stale cached result and silently propagate bad data downstream.
• No expiration means poisoned caches live forever. A cache entry written from a buggy reprojection persists across every future run until manually purged, with no time bound to force eventual recompute.
• Caching without persistence evaporates. An in-memory cache is gone when the scheduled process exits, so the next run recomputes everything and the cache appears not to work at all.

Version and Environment Compatibility

pip install "prefect>=3.0" "rasterio>=1.3" "pyproj>=3.6" requests

The cache key is the entire mechanism, so the diagram below shows what goes into it and when a hit versus a miss occurs.

The Recipe: A Cached Reproject Task with a Custom cache_key_fn

The key insight is that a cache_key_fn receives the flow-run context and the task’s keyword arguments, and returns a string. You control exactly which inputs contribute. The function below hashes only the tile bbox, the target CRS, and the source ETag — the three values that determine the output bytes — and ignores everything volatile.

import hashlib
import logging
from datetime import timedelta
from typing import Any

import rasterio
from rasterio.warp import calculate_default_transform, reproject, Resampling
from prefect import task
from prefect.context import TaskRunContext

logger = logging.getLogger(__name__)


def spatial_cache_key(context: TaskRunContext, arguments: dict[str, Any]) -> str:
    """Build a content-addressed cache key from only the semantically stable inputs.

    Ignores volatile arguments (open handles, worker id, retry count). The key
    changes only when the tile bounds, the target CRS, or the *source content*
    (via ETag) change — so a republished source invalidates the cache, but an
    unchanged nightly re-run hits it.
    """
    bbox: tuple[float, float, float, float] = arguments["bbox"]
    dst_crs: str = arguments["dst_crs"]
    source_etag: str = arguments["source_etag"]

    # Round bbox to a fixed precision so float noise does not fragment the key.
    bbox_str = ",".join(f"{c:.6f}" for c in bbox)
    material = f"{bbox_str}|{dst_crs}|{source_etag}".encode("utf-8")
    return hashlib.sha256(material).hexdigest()


@task(
    cache_key_fn=spatial_cache_key,
    cache_expiration=timedelta(days=7),   # force recompute weekly even on a hit
    persist_result=True,                   # required for cross-run cache hits
    retries=2,
    retry_delay_seconds=10,
)
def reproject_tile(
    src_path: str,
    dst_path: str,
    bbox: tuple[float, float, float, float],
    dst_crs: str,
    source_etag: str,
    resampling: Resampling = Resampling.bilinear,
) -> str:
    """Reproject a raster tile to dst_crs. Cached on (bbox, dst_crs, source_etag)."""
    with rasterio.open(src_path) as src:
        transform, width, height = calculate_default_transform(
            src.crs, dst_crs, src.width, src.height, *src.bounds
        )
        profile = src.profile.copy()
        profile.update(crs=dst_crs, transform=transform, width=width, height=height)

        with rasterio.open(dst_path, "w", **profile) as dst:
            for band in range(1, src.count + 1):
                reproject(
                    source=rasterio.band(src, band),
                    destination=rasterio.band(dst, band),
                    src_transform=src.transform,
                    src_crs=src.crs,
                    dst_transform=transform,
                    dst_crs=dst_crs,
                    resampling=resampling,
                )
    logger.info("Reprojected %s -> %s (cache key on bbox/CRS/etag)", src_path, dst_path)
    return dst_path

To supply the source_etag, cheaply resolve it with a conditional HEAD request before invoking the task, so a changed upstream file flows into the key without downloading the payload:

import requests
from prefect import task


@task(retries=3, retry_delay_seconds=5)
def source_etag(url: str) -> str:
    """Return the source object's ETag — the cheap content fingerprint for the cache key."""
    resp = requests.head(url, timeout=30)
    resp.raise_for_status()
    return resp.headers.get("ETag", resp.headers.get("Last-Modified", "no-validator")).strip('"')

Key Implementation Notes

• The bbox is rounded before hashing. Floating-point bounds derived from different projections can differ in the fifteenth decimal for the same tile. Rounding to six decimals (~0.1 m at the equator) stabilises the key so semantically identical tiles share a cache entry instead of fragmenting into near-duplicates.
• The ETag is the content fingerprint, not the timestamp. Keying on the download time would defeat caching entirely; keying on the ETag means the cache invalidates precisely when the bytes change. When a source exposes no ETag, fall back to Last-Modified, and only then to a sentinel that disables content-based invalidation.
• cache_expiration is a safety net, not the primary control. The custom key already invalidates on real change; the seven-day expiration bounds the blast radius of a poisoned cache written by a buggy deploy, forcing eventual recompute even if the inputs look unchanged.
• persist_result=True is mandatory for cross-run hits. The cache key names a stored result. Without a persisted result store, the mapping from key to output lives only in the current process and is lost on exit, so every scheduled run starts cold.
• The cache_key_fn must not open files or hit the network. It runs before the task and should be a pure function of already-resolved arguments. Resolve the ETag in its own upstream task (above) and pass the value in, rather than fetching inside the key function.
• Retries and caching compose cleanly. A cache hit short-circuits before the retry logic ever engages; retries only matter on a miss, when the reprojection actually executes.

Troubleshooting Cache Misses and Stale Hits

Integration Note

Caching plugs directly into the fan-out described in orchestrating spatial pipelines with Prefect: decorate the mapped reproject or download task with spatial_cache_key, and a nightly re-run over an unchanged tile set becomes almost free — every mapped run resolves to a cached state and skips recompute. It composes naturally with bounded mapping, too: because cache hits return before the task body executes, they never consume a slot against the concurrency limit used in parallel tile downloads with Prefect task mapping, so a mostly-cached fan-out clears in seconds while only the genuinely-new tiles contend for download bandwidth.

Related

• Orchestrating Spatial Pipelines with Prefect — the flow structure, runtime fan-out, and task-runner choices this caching pattern plugs into
• Parallel Tile Downloads with Prefect Task Mapping — bounded .map() concurrency and per-tile error isolation that composes with cache hits
• Orchestrating Spatial ETL Pipelines — the five-stage reference model and cross-cutting concerns like idempotency and result persistence