This guide is part of Parsing GeoJSON and Shapefile APIs, which itself sits within the broader Mastering Geospatial Data Ingestion in Python reference.

Streaming Large GeoJSON Responses With ijson

The Problem: Whole-Payload Parsers Fall Over on Large FeatureCollections

A spatial API that returns a GeoJSON FeatureCollection with millions of features hands you one enormous JSON document. The moment you call json.loads(response.text) or geopandas.read_file(url), Python builds the entire object graph in memory before you touch a single feature — and a multi-gigabyte response balloons to several times its size once geometries and attribute dicts are boxed into Python objects.

Whole-payload parsing breaks streaming ingestion in specific ways:

  • Out-of-memory kills in constrained runners: A container with a 4 GB limit dies parsing a 2 GB response, because the peak allocation is the raw text plus the fully materialized object tree at the same time.
  • No progress until the last byte lands: json.loads cannot yield anything until download and parse both finish, so a slow endpoint blocks the whole pipeline with zero throughput.
  • Doubled memory on the GeoDataFrame conversion: Even if the parse succeeds, handing the full feature list to GeoPandas duplicates the data during construction, pushing a marginal job over the edge.
  • Fragile all-or-nothing failure: A truncated response corrupts the entire parse; there is no partial result to checkpoint or resume from.

Incremental parsing with ijson reads the byte stream event by event, so peak memory tracks the chunk size you choose rather than the size of the response.

Version and Environment Compatibility

Library Version Role Notes
ijson >=3.2 Incremental JSON parsing The yajl2_c backend is fastest; falls back to pure Python if unavailable
geopandas >=0.14 Chunk to GeoDataFrame from_features builds a frame from a feature iterable
pyarrow >=14.0 GeoParquet append Backs to_parquet; needed for row-group appends
requests >=2.31.0 Streamed download Use stream=True and response.raw for a byte stream 
pip install "ijson>=3.2" "geopandas>=0.14" "pyarrow>=14.0" "requests>=2.31.0"

Constant-Memory Streaming From Response to GeoParquet

Constant-memory GeoJSON streaming pipeline Left to right: HTTP byte stream, ijson.items on features.item, a chunk buffer of N features, GeoDataFrame.from_features, and an append to GeoParquet. A caption notes peak memory equals one chunk, not the full payload. HTTP stream stream=True ijson.items 'features.item' Chunk buffer N features GeoDataFrame from_features GeoParquet append peak memory = one chunk, not the full FeatureCollection

The key insight is that ijson.items(stream, 'features.item') treats features as an array and item as each element, so it emits one fully-formed Feature dict per iteration and discards the parser state for the previous one. Nothing accumulates unless you choose to buffer it — and you only buffer one chunk at a time.

The stream_geojson_features Recipe

The generator below streams features straight from an HTTP response, batches them into fixed-size chunks, and a companion writer converts each chunk to a GeoDataFrame and appends it to a GeoParquet dataset. Together they hold at most one chunk in memory regardless of response size.

import logging
from pathlib import Path
from typing import Iterator

import geopandas as gpd
import ijson
import requests

logger = logging.getLogger(__name__)


def stream_geojson_features(
    url: str,
    chunk_size: int = 50_000,
    timeout: float = 120.0,
) -> Iterator[list[dict]]:
    """
    Stream a GeoJSON FeatureCollection and yield fixed-size feature batches.

    Uses ijson to parse the ``features`` array incrementally so peak memory is
    bounded by ``chunk_size`` rather than the size of the response body.
    """
    with requests.get(url, stream=True, timeout=timeout) as response:
        response.raise_for_status()
        response.raw.decode_content = True  # honour gzip/deflate transparently

        batch: list[dict] = []
        # 'features.item' matches each element of the top-level features array.
        for feature in ijson.items(response.raw, "features.item"):
            batch.append(feature)
            if len(batch) >= chunk_size:
                logger.info("Yielding chunk of %d features", len(batch))
                yield batch
                batch = []

        if batch:  # final short chunk
            logger.info("Yielding final chunk of %d features", len(batch))
            yield batch


def stream_geojson_to_geoparquet(
    url: str,
    out_path: str | Path,
    source_crs: str = "EPSG:4326",
    chunk_size: int = 50_000,
) -> int:
    """
    Stream a remote GeoJSON FeatureCollection into a GeoParquet dataset.

    Returns the total number of features written. The first chunk creates the
    dataset; later chunks append, so the whole payload is never resident at once.
    """
    out_path = Path(out_path)
    total = 0

    for i, batch in enumerate(stream_geojson_features(url, chunk_size=chunk_size)):
        gdf = gpd.GeoDataFrame.from_features(batch, crs=source_crs)
        # ijson decodes numbers as Decimal; cast object columns defensively.
        gdf = gdf.convert_dtypes()

        if i == 0:
            gdf.to_parquet(out_path, index=False)
        else:
            gdf.to_parquet(out_path, index=False, append=True)

        total += len(gdf)
        logger.info("Wrote chunk %d | cumulative_features=%d", i, total)

    logger.info("Streaming complete | total_features=%d -> %s", total, out_path)
    return total

Key Implementation Notes

  • Pass response.raw, not response.text. ijson needs a file-like byte stream. Reading .text or .content first defeats the whole purpose by buffering the payload; set response.raw.decode_content = True so compressed transfer encodings are still handled transparently.

  • ijson yields Decimal, not float. The parser decodes JSON numbers as decimal.Decimal for exactness. Coordinates flow into Shapely fine, but attribute columns can end up as object dtype — convert_dtypes() or an explicit cast keeps the GeoParquet schema clean and stable across chunks.

  • Choose chunk_size by geometry weight, not feature count alone. Fifty thousand points is light; fifty thousand dense multipolygons is not. Tune the batch so one chunk’s GeoDataFrame comfortably fits the runner, since that is the true memory ceiling.

  • Keep the CRS explicit and identical per chunk. GeoParquet stores CRS in its metadata, and appending chunks with mismatched CRS produces an inconsistent dataset. Resolve the source CRS once and pass it to every from_features call.

  • Use HTTPResponse as a context manager. The with requests.get(...) block guarantees the socket closes even if a downstream chunk write raises, preventing leaked connections during long streams.

  • A truncated stream raises inside the loop. If the connection drops mid-array, ijson raises an IncompleteJSONError on the next iteration — catch it to checkpoint the last successfully written chunk and resume rather than restarting from zero.

Troubleshooting Streaming GeoJSON

Failure Mode Root Cause Mitigation
MemoryError despite streaming Reading response.text/.content before ijson Pass response.raw and iterate features.item directly
Empty iterator, no features yielded Wrong prefix — array nested under another key Inspect the JSON shape; use data.features.item or the correct path
Attribute columns are object dtype ijson decodes numbers as Decimal Call convert_dtypes() or cast columns after from_features
GeoParquet append raises schema mismatch Chunks have differing columns or CRS Union columns to a fixed schema; pin one CRS across all chunks
IncompleteJSONError mid-stream Connection dropped before the array closed Catch it, checkpoint the last written chunk, and resume with a range request

Integration Note

stream_geojson_features slots into the ingestion stage described in Parsing GeoJSON and Shapefile APIs as the memory-safe alternative to a whole-file read for oversized endpoints. Before streaming, use the lightweight probe in extracting bounding boxes from GeoJSON APIs to confirm the response actually covers your area of interest and avoid streaming gigabytes you will discard. The chunked GeoParquet output follows the guidance in choosing a vector output format for spatial pipelines, whose columnar, append-friendly layout is exactly what this per-chunk writer needs.