This guide is part of Choosing a Vector Output Format for Spatial Pipelines, within the broader Automated Vector & Raster Cleaning Workflows reference.

GeoParquet vs FlatGeobuf vs Shapefile for Pipeline Outputs

Three formats, one layer, three different jobs. The choice is decided by what reads the file next, and the differences are large enough that guessing costs real money at scale.

The Comparison That Matters

Where Each One Wins

Writing All Three From One Frame

from __future__ import annotations

import logging
from pathlib import Path

import geopandas as gpd

logger = logging.getLogger(__name__)


def write_outputs(
    gdf: gpd.GeoDataFrame,
    stem: Path,
    analytical: bool = True,
    streaming: bool = False,
    legacy: bool = False,
) -> dict[str, Path]:
    """Write the layer once per consumer that genuinely needs its own format."""
    if gdf.crs is None:
        raise ValueError("refusing to write a layer with no CRS")

    written: dict[str, Path] = {}

    if analytical:
        path = stem.with_suffix(".parquet")
        gdf.to_parquet(path, compression="zstd", write_covering_bbox=True, index=False)
        written["geoparquet"] = path

    if streaming:
        path = stem.with_suffix(".fgb")
        gdf.to_file(path, driver="FlatGeobuf", SPATIAL_INDEX="YES")
        written["flatgeobuf"] = path

    if legacy:
        path = stem.with_suffix(".shp")
        truncated = {c: c[:10] for c in gdf.columns if len(c) > 10 and c != gdf.geometry.name}
        if truncated:
            logger.warning("shapefile export truncates %d field name(s): %s",
                           len(truncated), truncated)
        gdf.to_file(path, driver="ESRI Shapefile", encoding="utf-8")
        written["shapefile"] = path

    logger.info("wrote %s", {k: v.name for k, v in written.items()})
    return written

Key Implementation Notes

• write_covering_bbox=True writes the bounds columns. Without them, a reader cannot prune row groups on a spatial predicate, which is most of GeoParquet’s advantage — the mechanism described in reading GeoParquet from S3 with pyarrow filters.
• SPATIAL_INDEX="YES" is what makes FlatGeobuf worth choosing. Without the packed R-tree it is a compact serialisation with no read advantage over anything else.
• Shapefile truncation is logged, not silent. The names that collide after truncation are the ones that will confuse whoever receives the file, and telling them is the minimum courtesy.
• The CRS check runs first. Two of the three formats record the CRS internally and one relies on a sidecar; a layer written without one is unusable in all three.
• Each format is optional. Writing all three by default triples the storage for consumers that may not exist; the flags make the decision explicit per dataset.
• UTF-8 is stated for the shapefile. GDAL writes a .cpg accordingly, which is the only thing standing between a receiving system and mojibake.

What Each Format Loses

GeoParquet loses nothing structural, and its one practical cost is tooling age: a consumer on an older GDAL or an old desktop GIS may not read it. That gap has closed substantially but is worth checking before making it the only export.

FlatGeobuf loses column pruning — it is row-oriented, so a query touching three columns reads all eighteen. It also has a smaller ecosystem than Parquet outside the geospatial world, which matters if a data team wants to read the layer with general-purpose tooling.

Shapefile loses the most and is the most widely readable. Ten-character field names collide, as described in standardizing column names across multiple shapefiles; nulls become empty strings or zeros; the 2 GB ceiling is per file and reached sooner than expected on dense geometry; and mixed geometry types cannot coexist in one file at all.

Troubleshooting Format Choices

Integration Note

Format is a publication decision, made per consumer rather than per pipeline. The curated copy the pipeline reads back should be GeoParquet, partitioned as described in partitioning GeoParquet by region and date; anything else is an export derived from it, regenerated rather than maintained. That separation is what keeps a mandated Shapefile handover from becoming the pipeline’s own storage format.

Where GeoPackage Fits

The three formats above cover machine consumers; a fourth covers the human one. GeoPackage is a single SQLite file holding one or many layers, with real nulls, unrestricted field names, an optional spatial index and universal support in desktop GIS. It sits between Shapefile and the modern formats: larger than GeoParquet, structurally lossless, and openable by double-clicking.

It is the right answer for a handover to someone who will open the file rather than query it, and for the case where several related layers should travel together — a parcels layer, its zoning lookup and a boundary, in one file that cannot be separated in transit.

It is the wrong answer as a pipeline’s internal storage. SQLite is a single-writer database, so concurrent partition writes contend, and there is no column pruning to reward a query that touches three fields of forty.

The practical rule across all four formats is that a pipeline should have exactly one lossless internal format and as many exports as it has kinds of consumer. Exports are regenerated from the internal copy and never edited, which keeps the number of things that can disagree at one.

Benchmark Your Own Layer

The figures on this page come from one 2.4 million polygon layer with eighteen attributes, and the ratios move with geometry complexity and attribute width. A layer of simple points with two columns compresses very differently from one of dense coastline polygons.

Benchmarking your own is half an hour of work: write the same frame in each format, record the file sizes, then time a column query, a bbox query and a full read against each. Those five numbers decide the format choice more reliably than any general recommendation, and they are worth repeating when the layer’s shape changes materially.

Migrating an Existing Archive

Most teams arrive at this decision with a directory of Shapefiles already in production, and the migration is more about bookkeeping than about conversion.

Convert in place under a new prefix rather than replacing files, so the old copies remain readable while consumers move. Record, per converted layer, the source path, the feature count, the CRS and a checksum — the manifest described in converting Shapefiles to GeoParquet with GeoPandas — which makes the conversion resumable and its completeness checkable.

Two things need a decision rather than a default. Truncated field names should be restored from the original header if one survives, and the mapping recorded either way; and layers whose CRS lived only in a .prj need that CRS asserted rather than assumed, because a missing sidecar produces a converted file that is silently unreferenced.

Announce a date after which the old copies stop being updated, and keep them read-only rather than deleting them. A migration where the old path simply stops changing is far easier for consumers to notice than one where it disappears.

Related

• Choosing a Vector Output Format for Spatial Pipelines
• Converting Shapefiles to GeoParquet with GeoPandas
• Partitioning GeoParquet by Region and Date
• Storing Spatial Data in Cloud Object Storage