This guide is part of Storing Spatial Data in Cloud Object Storage, within the broader Orchestrating Spatial ETL Pipelines reference.

Reading GeoParquet from S3 with pyarrow Filters

A read against object storage costs requests and bytes, and both are controllable. The controls are partition predicates, row-group predicates and column projection — applied together, they routinely turn a multi-gigabyte scan into a few tens of megabytes.

Why an Unfiltered Read Is So Expensive

  • Every column is fetched. A dataset with forty attributes transfers all of them even when a query needs three.
  • Every row group is opened. Without usable statistics, a bbox filter is applied after the data arrives, which saves nothing.
  • Every file is listed. A prefix scan over a badly partitioned dataset can take longer than the read.
  • The geometry column dominates. Geometry is usually the largest column, so fetching it when only attributes are needed doubles or triples the transfer.

Version and Environment Compatibility

Component Version Note
pyarrow >=14 dataset API, pc.field expressions, S3 filesystem
GeoPandas >=1.0 from_arrow preserves GeoParquet CRS metadata
s3fs >=2024.6 Alternative filesystem with familiar credentials handling
shapely >=2.0 Exact predicate applied to the pruned result 
pip install "pyarrow>=14" "geopandas>=1.0" "s3fs>=2024.6" "shapely>=2.0"

The Three Filters and What Each One Saves

Where each filter removes work A read pipeline with three reduction stages. Partition predicates eliminate directories using only their path values, costing nothing. Row-group statistics eliminate groups within surviving files after reading each file's footer. Column projection restricts which column chunks are fetched from the surviving groups. Each stage reduces both bytes and requests, and they compose multiplicatively. 2 400 files 2 TB partition filter → 3 files, 0 bytes read row-group filter → 11 groups, 3 footers column projection → 4 of 40 columns · ≈ 180 MB transferred The three compose: 2 TB → 180 MB.

The read_region Recipe

from __future__ import annotations

import logging

import geopandas as gpd
import pyarrow.compute as pc
import pyarrow.dataset as ds
import pyarrow.fs as fs
import shapely

logger = logging.getLogger(__name__)


def read_region(
    root: str,
    regions: list[str],
    periods: list[str],
    bbox: tuple[float, float, float, float],
    columns: list[str] | None = None,
    exact: bool = True,
) -> gpd.GeoDataFrame:
    """Read one spatial window out of a partitioned GeoParquet dataset.

    Assumes the writer stored bbox columns (minx, miny, maxx, maxy) alongside the
    geometry — Parquet statistics cannot prune on a binary geometry column.
    """
    filesystem = fs.S3FileSystem()  # credentials from the environment or instance role
    dataset = ds.dataset(root, filesystem=filesystem, format="parquet",
                         partitioning="hive")

    minx, miny, maxx, maxy = bbox
    predicate = (
        pc.field("region").isin(regions)
        & pc.field("period").isin(periods)
        # Bounding-box overlap, expressed so row-group statistics can prune it.
        & (pc.field("minx") <= maxx)
        & (pc.field("maxx") >= minx)
        & (pc.field("miny") <= maxy)
        & (pc.field("maxy") >= miny)
    )

    projection = None
    if columns:
        projection = sorted(set(columns) | {"geometry"})

    table = dataset.to_table(filter=predicate, columns=projection)
    logger.info("scanned %s: %d rows survived pruning", root, table.num_rows)

    if table.num_rows == 0:
        return gpd.GeoDataFrame(geometry=[], crs=4326)

    gdf = gpd.GeoDataFrame.from_arrow(table)
    if exact:
        window = shapely.box(minx, miny, maxx, maxy)
        gdf = gdf.loc[shapely.intersects(gdf.geometry.values, window)]
    return gdf

Key Implementation Notes

  • The bbox predicate is four scalar comparisons, not a spatial operator. Parquet statistics work on scalars; expressing overlap this way is what lets the reader skip row groups without opening them.
  • Bounds columns must exist in the file. They are written by the producer, and their absence turns this read into a full scan with an in-memory filter — see partitioning GeoParquet by region and date.
  • Geometry is always added to the projection. Dropping it produces a plain table and a confusing error two lines later; adding it explicitly avoids the surprise.
  • The exact predicate runs after pruning. Bounding-box overlap is a superset of true intersection, so the precise test is still needed — but it now runs over thousands of rows rather than millions.
  • Partition values are filtered as columns. With Hive partitioning, Arrow reconstructs region and period as real columns, so they read like any other predicate.
  • An empty result returns a typed empty frame. Returning None or a bare DataFrame pushes a branch into every caller.
Why the exact test still has to run A query window drawn over three features with their bounding boxes. All three bounding boxes overlap the window, so all three survive the pushdown filter. Only two of the geometries actually intersect it; the third is an L-shaped feature whose box overlaps while its geometry does not. The exact predicate removes it after the read. query window intersects ✓ intersects ✓ bbox overlaps the window geometry does not — dropped by the exact test after the read The pushdown is a cheap superset; the precise predicate is a cheap refinement once the set is small.

Requests, Not Only Bytes

Object-store reads are billed and rate-limited per request as well as per byte, and a naively configured reader can issue thousands of small ranged requests for a single scan.

Requests are billed and limited, not only bytes The same scan issued three ways. With very small row groups and no coalescing the reader makes about four thousand requests. Enabling range coalescing merges nearby reads and cuts it to roughly six hundred. Reusing a long-lived filesystem object removes the repeated TLS handshakes, leaving latency dominated by the data rather than the setup. requests issued for one scan tiny row groups, no coalescing ≈ 4 000 ranges coalesced ≈ 600 + reused filesystem object ≈ 600, without the handshakes A scan issuing far more requests than it opened files is fragmenting somewhere — usually in the writer's row-group size.

Three settings matter. Prefetch and coalescing let the reader merge nearby byte ranges into one request; Arrow’s default fragment readahead is usually fine, but a dataset with very small row groups defeats it. Concurrency should be bounded — dozens of parallel requests are productive, hundreds trigger throttling and produce retries that look like slowness. And connection reuse matters more than it seems: a fresh TLS handshake per object turns a fast read into a latency-bound one, so a long-lived filesystem object beats constructing one per call.

Measuring is straightforward: most stores can report request counts per prefix, and a read that issues far more requests than it opened files is fragmenting somewhere.

Troubleshooting Filtered Reads

Symptom Likely cause Fix
Filter appears ignored Predicate on a column absent from the files Confirm bounds columns exist; check partition names
Read transfers everything No column projection Pass columns explicitly, geometry included
Pruning saves nothing Rows not spatially sorted at write time Sort on a Morton key when writing
CRS lost after the read Table converted with a plain Arrow constructor Use GeoDataFrame.from_arrow
Thousands of tiny requests Row groups far too small Increase row-group size in the writer
Throttling and retries Unbounded read concurrency Cap parallel fragment reads

Integration Note

Reads of this shape belong behind a small function per dataset rather than being written inline by each consumer, so the bounds-column convention and the partition names live in one place. Where the same window is read repeatedly — a dashboard, a scheduled report — cache the result rather than the query, and key the cache on the window and the dataset version, following the reasoning in caching geospatial task results in Prefect.

Reading From a Compute Environment That Is Not Yours

A pipeline usually reads from the same account and region that wrote the data. Analysts, partners and downstream teams frequently do not, and three things change when they do.

Egress becomes visible. Cross-region reads are billed per byte and are often the largest single line on a lake’s bill. Where a consumer is consistently elsewhere, replicating the served subset to their region is usually cheaper than paying egress on every read.

Credentials become a design problem. Long-lived access keys handed to a partner are a liability; short-lived credentials issued through a role, or a requester-pays bucket that shifts the cost to the reader, are both better and both need the reader’s code to know about them.

Consistency expectations differ. A consumer reading during a partition rewrite may see an empty prefix. Publishing through a manifest or a staging-then-promote pattern removes the window, and is worth the extra step for any dataset read by someone who cannot be told to retry.

None of this is spatial-specific, but spatial datasets are unusually likely to be shared outside the team that produced them, which makes the questions arrive sooner.