This guide is part of Storing Spatial Data in Cloud Object Storage, within the broader Orchestrating Spatial ETL Pipelines reference.

Partitioning GeoParquet by Region and Date

Partitioning decides what a query can skip without reading, and spatial sorting decides what it can skip within the files it does read. The two together are the difference between a lake that answers a regional question in seconds and one that scans a terabyte to do it.

Why the Default Write Prunes Badly

• One file per batch fragments the dataset. Hourly writes across forty regions produce hundreds of thousands of small objects in a year.
• Unsorted rows defeat row-group statistics. Every group’s bounding box covers the whole extent, so a bbox filter cannot skip anything.
• Append-style writes duplicate on retry. A re-run adds a second copy of the partition, and both are read.
• Partitioning on a fine key explodes the object count. A partition per tile per day is correct pruning and unusable metadata.

Version and Environment Compatibility

pip install "geopandas>=1.0" "pyarrow>=14" "shapely>=2.0" "s3fs>=2024.6"

What Pruning Actually Skips

The write_geoparquet_partitioned Recipe

from __future__ import annotations

import logging

import geopandas as gpd
import numpy as np
import pyarrow as pa
import pyarrow.dataset as ds
import shapely

logger = logging.getLogger(__name__)


def morton_key(gdf: gpd.GeoDataFrame, bits: int = 16) -> np.ndarray:
    """Interleave scaled x and y so spatially close features sort together."""
    centroids = shapely.centroid(gdf.geometry.values)
    xs = shapely.get_x(centroids)
    ys = shapely.get_y(centroids)

    span = max(xs.ptp() or 1.0, ys.ptp() or 1.0)
    scale = (2 ** bits - 1) / span
    xi = ((xs - xs.min()) * scale).astype(np.uint64)
    yi = ((ys - ys.min()) * scale).astype(np.uint64)

    key = np.zeros_like(xi)
    for bit in range(bits):
        key |= ((xi >> bit) & 1) << (2 * bit)
        key |= ((yi >> bit) & 1) << (2 * bit + 1)
    return key


def write_geoparquet_partitioned(
    gdf: gpd.GeoDataFrame,
    root: str,
    region: str,
    period: str,
    filesystem=None,
    rows_per_group: int = 200_000,
    max_rows_per_file: int = 2_000_000,
) -> None:
    """Write one partition of a GeoParquet dataset, spatially sorted and replaceable."""
    if gdf.empty:
        logger.warning("nothing to write for region=%s period=%s", region, period)
        return
    if gdf.crs is None:
        raise ValueError("refusing to write a partition with no CRS")

    ordered = gdf.iloc[np.argsort(morton_key(gdf), kind="stable")].copy()
    ordered["region"] = region
    ordered["period"] = period

    table = ordered.to_arrow()   # carries the GeoParquet metadata through
    ds.write_dataset(
        table,
        base_dir=root,
        filesystem=filesystem,
        format="parquet",
        partitioning=["region", "period"],
        partitioning_flavor="hive",
        existing_data_behavior="delete_matching",
        max_rows_per_file=max_rows_per_file,
        min_rows_per_group=rows_per_group // 2,
        max_rows_per_group=rows_per_group,
        file_options=ds.ParquetFileFormat().make_write_options(
            compression="zstd", compression_level=3, write_statistics=True,
        ),
    )
    logger.info("wrote %d rows to region=%s/period=%s", len(ordered), region, period)

Key Implementation Notes

• The Morton sort is what makes row-group statistics useful. Without it, every group’s bounding box spans the partition and no group can be skipped; with it, a bbox query typically reads a small minority of groups.
• existing_data_behavior="delete_matching" replaces the partition on a re-run. This is the idempotency guarantee, and omitting it is the most common cause of duplicated rows in a lake.
• Row groups of ~200 000 rows keep statistics granular without inflating the footer. Very small groups make the metadata large; very large ones make pruning coarse.
• write_statistics=True is explicit because a file written without statistics cannot be pruned at all, and the flag has defaulted differently across Arrow versions.
• The CRS check comes before any work. A partition written without a CRS is unusable and will be discovered by a consumer rather than by the writer.
• ZSTD level 3 is a deliberate middle: level 1 is barely smaller than uncompressed for geometry, and levels above 6 cost write time out of proportion to the saving.

Choosing the Period Granularity

The date component of the partition key trades pruning against object count, and the right answer follows from how much data one period holds.

Aim for partitions in the hundreds of megabytes to low gigabytes. A dataset producing 50 MB a day partitions well by month; one producing 10 GB a day partitions by day. The mistake is picking the granularity from the update cadence rather than from the volume: a pipeline that runs hourly does not need hourly partitions, it needs hourly writes into a daily or monthly partition, with compaction closing the partition when the period ends.

Where a dataset spans both extremes — dense in cities, sparse elsewhere — a single global granularity will be wrong somewhere. Partitioning by region first means each region’s partition size follows its own density, which is usually enough to keep every partition in a sensible range without per-region configuration.

Troubleshooting Partitioned Writes

Integration Note

One task writes one partition: the same alignment used throughout storing spatial data in cloud object storage, and the reason a retry is safe. Where the orchestrator has its own partition concept, use the same keys so the storage layout and the run history describe the same units — a Dagster partition key of region|period maps directly onto this write with no translation, and a backfill of one cell rewrites exactly one prefix.

Evolving the Schema Without Rewriting History

A partitioned dataset outlives the schema it started with. Three changes come up repeatedly, and only one of them is expensive.

Adding a column is free. Parquet readers tolerate files that lack a column the schema declares, returning nulls for the older partitions. Write the new column going forward and backfill only if the historical values genuinely exist.

Widening a type — int32 to int64, float32 to float64 — is usually free on read and worth doing at a partition boundary so that no single partition mixes both. Narrowing is not safe and should be treated as a new column.

Renaming or removing a column breaks every consumer reading by name. Where it is unavoidable, write both names for a deprecation period, announce the removal with the dataset metadata, and drop the old one only after the consumers have moved.

The general principle is that partitions are immutable once written. A change that requires rewriting history is a new dataset version, published under a new prefix, rather than an edit in place — which keeps the old version readable while consumers migrate and makes the rollback trivial.

Related

• Storing Spatial Data in Cloud Object Storage
• Reading GeoParquet from S3 with pyarrow Filters
• Converting Shapefiles to GeoParquet with GeoPandas
• Partitioning Spatial Assets by Tile in Dagster