This guide is part of Syncing STAC Catalogs with pystac-client, which itself sits within the broader Mastering Geospatial Data Ingestion in Python reference.

Best Practices for STAC Catalog Pagination in Python

The Problem: Implicit Full-Collection Loads Crash Spatial Pipelines

STAC catalog pagination fails silently when Python code converts search results to a list before iterating. Calling .get_all_items() or wrapping .items() in list() forces pystac-client to resolve every next link synchronously and buffer every pystac.Item object in memory simultaneously — before your code processes a single record.

This pattern breaks spatial pipelines in predictable ways:

  • Out-of-memory crashes in containers: Kubernetes pods and CI runners with 2–4 GB RAM limits terminate the process mid-pagination, leaving downstream tables in a partial state with no rollback.
  • Silent data loss from incomplete traversal: Some STAC endpoints enforce a hard server-side maximum (often 10,000 items) without signalling it; a list-based approach silently truncates results, causing coverage gaps in raster mosaics or time-series joins.
  • Cascading backpressure into compute layers: Loading all items before processing means your GeoPandas or Arrow consumer waits for full network traversal before it can begin — multiplying total job time unnecessarily.
  • Undetectable pagination failures: When a next link returns a transient 503, converting to a list swallows the error mid-traversal. A generator surfaces it immediately at the item boundary where it occurred.

Version and Environment Compatibility

pystac-client pystac Python Pagination method Known caveats
>=0.8.0 >=1.8.0 3.11+ .items() generator Stable cursor-based traversal; max_items respected
0.7.x 1.7.x 3.9–3.11 .items() generator max_items sometimes requires manual guard; upgrade recommended
0.6.x 1.6.x 3.9–3.10 .get_all_items() only No lazy iteration; avoid for catalogs > 5,000 items
<0.6.0 <1.5.0 3.8 Manual next link traversal No built-in pagination abstraction; full custom implementation needed

Install the recommended stack:

pip install "pystac-client>=0.8.0" "pystac>=1.8.0" tenacity requests

How STAC Pagination Works Under the Hood

STAC cursor-based pagination sequence Diagram showing three rounds of HTTP request/response between a Python client and a STAC API. Each response carries a next link cursor. When no next link is returned, iteration stops. Python client STAC API GET /search?limit=500&collections=sentinel-2-l2a 200 OK — 500 items + link rel="next" token=abc GET /search?token=abc (cursor from next link) 200 OK — 500 items + link rel="next" token=def GET /search?token=def 200 OK — 312 items, no next link → iteration stops

STAC APIs expose paginated results through HTTP Link headers or inline links arrays where rel="next". Each response contains a cursor token — an opaque value the server uses to track position — not an offset integer. This is important: token-based cursors cannot be safely parallelised or seeked; they must be traversed sequentially. The pystac-client library abstracts this traversal inside .items(), but default configurations still allow aggressive fetching if you do not set explicit limit and max_items parameters.

Production-Ready Pagination Recipe

The following function is a single, copy-pasteable implementation covering conformance checking, generator-based streaming, exponential-backoff retry, and structured logging. Drop it into any ingestion module that sources data from a STAC API.

import logging
from typing import Iterator

import pystac
import pystac_client
from requests.exceptions import RequestException
from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential,
)

logger = logging.getLogger(__name__)

# STAC API Core conformance class that guarantees standard pagination behaviour
_STAC_CORE_CONFORMANCE = "https://api.stacspec.org/v1.0.0/core"


@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=1, min=2, max=30),
    retry=retry_if_exception_type(RequestException),
    reraise=True,
)
def _open_stac_client(catalog_url: str) -> pystac_client.Client:
    """Open the STAC client with retry logic for transient connection failures."""
    return pystac_client.Client.open(catalog_url)


def stream_stac_items(
    catalog_url: str,
    collections: list[str],
    bbox: tuple[float, float, float, float],
    datetime_range: str,
    page_limit: int = 500,
    max_items: int = 10_000,
) -> Iterator[pystac.Item]:
    """
    Yield STAC items one at a time using generator-based pagination.

    Parameters
    ----------
    catalog_url:     Root URL of the STAC API endpoint.
    collections:     List of collection IDs to search (e.g. ['sentinel-2-l2a']).
    bbox:            (west, south, east, north) in WGS84 decimal degrees.
    datetime_range:  RFC 3339 interval string, e.g. '2023-01-01/2023-12-31'.
    page_limit:      Items per page; match server's recommended max (usually 100-500).
    max_items:       Hard ceiling on total items yielded — prevents runaway queries.
    """
    stac_client = _open_stac_client(catalog_url)

    # Warn when the API does not guarantee standard pagination behaviour
    conforms_to: list[str] = stac_client.get_conforms_to() or []
    if _STAC_CORE_CONFORMANCE not in conforms_to:
        logger.warning(
            "Catalog %s does not advertise STAC API Core conformance. "
            "Pagination behaviour may differ from the specification.",
            catalog_url,
        )

    search = stac_client.search(
        collections=collections,
        bbox=bbox,
        datetime=datetime_range,
        limit=page_limit,
        max_items=max_items,
    )

    items_yielded = 0
    log_interval = max(1, max_items // 20)  # log roughly 20 times across the run

    # .items() traverses next links lazily — only one page is live in memory
    for item in search.items():
        items_yielded += 1

        if items_yielded % log_interval == 0:
            logger.info(
                "STAC pagination progress | catalog=%s | items_yielded=%d | ceiling=%d",
                catalog_url,
                items_yielded,
                max_items,
            )

        yield item

        # Redundant safety guard in case pystac-client < 0.8 ignores max_items
        if items_yielded >= max_items:
            logger.info(
                "Reached max_items ceiling (%d). Stopping pagination for %s.",
                max_items,
                catalog_url,
            )
            return

    logger.info(
        "Pagination complete | catalog=%s | total_items=%d",
        catalog_url,
        items_yielded,
    )

Key Implementation Notes

  • _open_stac_client is decorated separately from the search call. Retrying the connection open independently means transient DNS or TLS failures during client initialisation do not waste a search attempt. The search itself is not retried at this level — instead, pystac-client internally retries individual page fetches using its own session adapter.

  • get_conforms_to() can return None on older STAC 0.9 endpoints. The or [] guard prevents a TypeError when checking membership. Log the warning and proceed cautiously — test with a small max_items value before running a full sync.

  • page_limit and max_items serve different purposes. page_limit controls the HTTP request payload size (latency vs. bandwidth tradeoff); max_items caps total pipeline exposure. Setting max_items=None removes all ceilings — never do this in production without a separate circuit breaker.

  • The log_interval adapts to max_items. A fixed log-every-5-items interval floods structured log services when max_items=100_000. Computing the interval as max_items // 20 keeps log volume predictable regardless of query scale.

  • Yielding one item at a time enables back-pressure. When this generator feeds a downstream consumer that writes to PostGIS or S3, consumer slowness naturally throttles the iteration rate without additional coordination code. The network socket stays open, but no additional pages are fetched until the generator’s next() is called.

  • The redundant ceiling guard protects against pystac-client version regressions. In version 0.7.x, max_items was sometimes ignored when the server returned fewer items per page than expected. The explicit if items_yielded >= max_items: return costs nothing and prevents silent overruns.

Troubleshooting Common Pagination Failures

Failure Root Cause Fix
StopIteration immediately on first call Endpoint returns zero results — spatial or temporal filter too narrow Widen bbox or datetime_range; confirm collection ID is correct
Items repeat across pages Server uses offset pagination but limit changes between requests Use a consistent limit value; some servers encode it into the cursor
ConnectionError after page 3–4 Idle timeout on long-running page fetches Reduce page_limit so each response arrives faster; add keepalive headers
Item count stops at exactly 10,000 Server enforces a hard ceiling without a next link Log a warning; use spatial tiling to partition the query into sub-regions
KeyError: 'next' on manual cursor extraction API uses Link header pagination instead of inline links array Use pystac_client directly; it handles both transport variants automatically

Integration Note

Plug stream_stac_items into the asset-download step described in Syncing STAC Catalogs with pystac-client. The generator interface is directly consumable by the parallel asset download loop described there — pass each yielded pystac.Item to the download worker without batching, and the download pool’s queue depth acts as a natural flow-control mechanism. For pipelines that also fetch bulk raster scenes, the same pagination pattern transfers directly to Bulk Downloading Satellite Imagery, where chunked I/O and checksum verification extend the recipe shown here.