This guide is part of Ingesting WFS & OGC API — Features Services, within the broader Mastering Geospatial Data Ingestion in Python reference.

Paging Through OGC API — Features with Python

Reading a collection larger than one page means following the rel=next link the service returns, and getting the loop’s termination and resume conditions right — because both silent truncation and infinite looping are easy to write and hard to notice.

Why Paging Bugs Are Expensive Here

  • A truncated walk looks like a small dataset. Nothing raises. The frame is valid, the geometries are fine, and a third of the country is missing.
  • An offset-based walk over unsorted data duplicates and drops rows simultaneously. The totals can even look plausible while individual features are wrong.
  • A loop that re-requests the same page runs forever. Some services return a next link on the final page, pointing back at itself.
  • A restart without a cursor re-reads everything. On a half-million-feature layer that is an hour of transfer to recover from a thirty-second failure.

Version and Environment Compatibility

Component Version Note
Python 3.10+ `X
requests >=2.32 Session reuse keeps the TLS handshake off the per-page cost
GeoPandas >=1.0 from_features accepts the GeoJSON feature list directly
Shapely >=2.0 Vectorized validity check on the assembled frame

How the Cursor Walk Terminates

The three ways a paging loop should end A loop with one entry and three exits. The normal exit is the absence of a next link. The second exit fires when the next URL equals the URL just fetched, which indicates a service returning a self-referential link. The third fires when a page returns zero features while still advertising a next link, which indicates the result set changed underneath the cursor. fetch page append, persist cursor no rel=next normal completion — validate the total and return next == current URL self-referential link — stop, raise, do not spin empty page, next present result set shifted mid-walk — stop and report the gap Only the first exit is success. Treating the other two as success is how a partial layer reaches production looking complete.

The walk_items Recipe

from __future__ import annotations

import json
import logging
from pathlib import Path

import geopandas as gpd
import pandas as pd
import requests

logger = logging.getLogger(__name__)


class PagingError(RuntimeError):
    """Raised when the walk cannot be completed safely."""


def walk_items(
    items_url: str,
    limit: int = 1000,
    state_path: Path | None = None,
    max_pages: int = 10_000,
    session: requests.Session | None = None,
) -> gpd.GeoDataFrame:
    """Page an OGC API — Features collection, resumably and with completeness checks.

    The cursor is the service's own next link; it is persisted after every page so a
    failed run resumes where it stopped instead of re-reading the collection.
    """
    session = session or requests.Session()
    session.headers.setdefault("User-Agent", "spatial-etl/1.0 (data@example.org)")
    session.headers.setdefault("Accept-Encoding", "gzip, deflate")

    url: str | None = items_url
    params: dict | None = {"limit": limit}
    collected = 0
    expected: int | None = None
    frames: list[gpd.GeoDataFrame] = []

    if state_path and state_path.exists():
        saved = json.loads(state_path.read_text())
        url, collected, expected = saved["next_url"], saved["collected"], saved.get("expected")
        params = None  # the resumed link already carries every parameter
        logger.info("resuming walk at page %d (%d features so far)", saved["pages"], collected)

    for page_no in range(1, max_pages + 1):
        if url is None:
            break

        response = session.get(url, params=params, timeout=120)
        response.raise_for_status()
        payload = response.json()
        params = None  # only the first request carries params; next links are complete

        features = payload.get("features", [])
        if expected is None:
            expected = payload.get("numberMatched")

        # A silent server cap shows up on page one and never changes.
        if page_no == 1 and len(features) < limit and payload.get("links"):
            logger.info("effective page size is %d (requested %d)", len(features), limit)

        links = {link["rel"]: link["href"] for link in payload.get("links", [])}
        next_url = links.get("next")

        if next_url == url:
            raise PagingError(f"service returned a self-referential next link at page {page_no}")
        if not features and next_url:
            raise PagingError(f"empty page {page_no} with a next link — result set shifted mid-walk")

        if features:
            frames.append(gpd.GeoDataFrame.from_features(features, crs="OGC:CRS84"))
            collected += len(features)

        url = next_url
        if state_path:
            state_path.write_text(json.dumps({
                "next_url": url, "collected": collected,
                "expected": expected, "pages": page_no,
            }))
    else:
        raise PagingError(f"walk exceeded {max_pages} pages — check the service's next links")

    if expected is not None and collected != expected:
        raise PagingError(f"collected {collected} features, service reported {expected}")

    if not frames:
        return gpd.GeoDataFrame(geometry=[], crs="OGC:CRS84")

    result = gpd.GeoDataFrame(pd.concat(frames, ignore_index=True), crs="OGC:CRS84")
    logger.info("walk complete: %d features over %d pages", len(result), len(frames))
    if state_path:
        state_path.unlink(missing_ok=True)  # a completed walk has no cursor to keep
    return result

Key Implementation Notes

  • params is cleared after the first request. A next link already encodes the limit, filters and cursor; re-appending limit can reset the cursor on some implementations and silently restart the walk.
  • The cursor is persisted after the page is appended, not before. Writing it first means a crash between the write and the append loses that page’s features while recording it as read.
  • max_pages is a backstop, not a limit. It converts an infinite loop into a loud failure; set it well above any plausible page count for the collection.
  • Completeness is checked against numberMatched from the first page. Later pages may report a different figure if the source changed; the first value is the one the walk was sized against.
  • The state file is removed on success. A stale cursor left behind is the classic cause of a subsequent run silently resuming a walk that already finished.
  • Accept-Encoding is set explicitly because several implementations only compress when asked, and GeoJSON compresses by roughly 80%.
Why the cursor is written last Two orderings of the same three operations shown against a crash point. Writing the cursor before appending the features means a crash between the two loses that page permanently, because the resumed walk starts after it. Appending first means the crash costs at most a repeated page, which the deduplicating write absorbs. cursor first — a crash loses the page fetch page write cursor append — crashed resume skips it: features gone for good append first — a crash costs one repeat fetch page append features write cursor resume re-reads it: the upsert absorbs it

Troubleshooting Paging Failures

Symptom Likely cause Fix
Walk ends after one page next link stripped by a proxy rewriting absolute URLs Rebuild the next URL against the base, or bypass the proxy
PagingError: self-referential next link Implementation returns next on the final page Report to the publisher; the guard already prevents the spin
Collected count exceeds numberMatched Features added during the walk Re-run with a temporal filter, or accept and deduplicate on id
Every page returns exactly 500 features Undocumented server cap below the requested limit Size the loop to the observed cap and log it
Resume starts from page one State file written after the walk completed, or removed too early Delete on success only, as the recipe does
Parallelism comes from partitioning, not from the cursor On the left a single chain of pages runs strictly in order because each cursor is only known once the previous page returns. On the right the extent is divided into four bounding boxes, each of which runs its own independent sequential walk, so four walks proceed concurrently and any one of them can fail and retry alone. one cursor · strictly sequential p1 p2 p3 …p4 needs p3's cursor four bbox partitions · four walks at once bbox A · p1→pN bbox B · p1→pN bbox C · p1→pN bbox D · p1→pN Partition on a grid whose cells are smaller than the service's cap, and features on a shared edge get deduplicated on id afterwards.

Integration Note

walk_items returns a frame and takes no writing responsibility, so it drops directly into the ingestion stage described in ingesting WFS and OGC API — Features services. The state path should live on durable storage rather than a container’s local disk, so a rescheduled worker can resume; where the orchestrator provides a run directory, use it. For layers large enough to need partitioning, the per-partition walks map cleanly onto the mapped-task pattern in parallel tile downloads with Prefect task mapping.

Sizing the Page

The limit parameter trades round trips against per-response cost, and the useful range is narrower than it looks. Below about 200 features per page, latency dominates and the walk spends most of its time waiting. Above about 5 000, response assembly on the server slows and the risk of hitting a gateway timeout rises sharply — and a timeout mid-page costs the whole page rather than a fraction of it.

A limit of 1 000 is a reasonable default for feature collections with modest attribute counts. Layers with large geometries — coastlines, catchment boundaries, anything with tens of thousands of vertices per feature — should page smaller, because the byte size of a page, not its feature count, is what actually times out. Measure the first page’s payload size and adjust once rather than guessing per layer.