This recipe belongs to Choosing a Vector Output Format for Spatial Pipelines, part of the broader Automated Vector & Raster Cleaning Workflows reference.

Converting Shapefiles to GeoParquet with GeoPandas

The Problem: Retiring a Shapefile Archive Without Losing Schema or CRS

Converting a Shapefile to GeoParquet looks like a one-liner — gpd.read_file(...).to_parquet(...) — and for a single clean file it nearly is. The failure surface appears when the input is a directory of dozens of Shapefiles produced by different tools over several years, where the naive loop quietly loses information no downstream stage can recover.

• Truncation is already baked in. The Shapefile DBF component caps attribute names at 10 bytes, so land_use_classification arrived on disk as land_use_c. Reading it into GeoPandas and writing GeoParquet preserves the truncated name — the conversion cannot un-mangle what the format already destroyed.
• A missing .prj yields a silent None CRS. If a source Shapefile lost its sidecar, gdf.crs is None, and to_parquet will happily write a GeoParquet with no coordinate reference system, poisoning every spatial join downstream.
• Mixed encodings corrupt attribute text. Legacy DBF files without a .cpg code-page sidecar are often Latin-1, and reading them as UTF-8 mojibakes accented place names before they ever reach the columnar file.
• One-file-at-a-time loops lose the failure record. A batch that aborts on the first unreadable Shapefile leaves the archive half-converted with no manifest of what succeeded, forcing a full restart.

Version and Environment Compatibility

pip install "geopandas>=1.0" "pyarrow>=14.0" "pyogrio>=0.7"

The conversion flow below shows where CRS recovery and truncation detection sit relative to the read and write — the two guard points that separate a safe batch from a lossy one.

The convert_shapefiles_to_geoparquet Recipe

The function below converts a list of Shapefile paths to GeoParquet in one call. It recovers a missing CRS from a caller-supplied fallback, records truncated column names without silently accepting them, writes ZSTD-compressed GeoParquet, and returns a per-file manifest so a batch never fails opaquely.

import logging
from pathlib import Path
from typing import Iterable, Optional

import geopandas as gpd
from pyproj import CRS

logger = logging.getLogger(__name__)

# DBF stores attribute names in a fixed 11-byte field: 10 usable characters.
_SHP_NAME_LIMIT = 10


def convert_shapefiles_to_geoparquet(
    paths: Iterable[str | Path],
    out_dir: str | Path,
    fallback_crs: Optional[int] = None,
    compression: str = "zstd",
    encoding: str = "utf-8",
) -> list[dict]:
    """Batch-convert Shapefiles to ZSTD-compressed GeoParquet.

    Parameters
    ----------
    paths:         Iterable of .shp file paths to convert.
    out_dir:       Directory that receives one .parquet per input.
    fallback_crs:  EPSG code applied only when a Shapefile has no .prj;
                   None means a missing CRS is a hard failure.
    compression:   Parquet codec ('zstd' recommended; 'snappy' if CPU-bound).
    encoding:      DBF text encoding; use 'latin-1' for legacy files lacking .cpg.

    Returns
    -------
    A manifest: one dict per input recording status, CRS, feature count,
    truncated column names, and output path.
    """
    out_root = Path(out_dir)
    out_root.mkdir(parents=True, exist_ok=True)
    manifest: list[dict] = []

    for raw_path in paths:
        src = Path(raw_path)
        record: dict = {"source": str(src), "status": "ok"}
        try:
            gdf: gpd.GeoDataFrame = gpd.read_file(src, encoding=encoding)

            # --- CRS guard: recover from fallback or fail loudly ---
            if gdf.crs is None:
                if fallback_crs is None:
                    raise ValueError(
                        f"{src.name} has no .prj and no fallback_crs was given"
                    )
                gdf = gdf.set_crs(epsg=fallback_crs)
                logger.warning(
                    "%s: missing CRS, applied fallback EPSG:%d",
                    src.name, fallback_crs,
                )
            record["crs"] = CRS(gdf.crs).to_epsg()

            # --- Detect names the Shapefile format already truncated ---
            truncated = [c for c in gdf.columns if len(c) >= _SHP_NAME_LIMIT]
            if truncated:
                logger.warning(
                    "%s: column names at/over %d chars (likely truncated): %s",
                    src.name, _SHP_NAME_LIMIT, truncated,
                )
            record["truncated_columns"] = truncated
            record["feature_count"] = len(gdf)

            # --- Write GeoParquet: CRS embedded as PROJJSON, no sidecar ---
            dest = out_root / f"{src.stem}.parquet"
            gdf.to_parquet(dest, compression=compression, index=False)

            # --- Verify the round-trip before trusting the output ---
            check = gpd.read_parquet(dest)
            if CRS(check.crs) != CRS(gdf.crs) or len(check) != len(gdf):
                raise RuntimeError(f"round-trip mismatch for {dest.name}")
            record["output"] = str(dest)

        except Exception as exc:  # noqa: BLE001 — collect, do not abort the batch
            logger.error("Failed to convert %s: %s", src.name, exc)
            record["status"] = "error"
            record["error"] = str(exc)

        manifest.append(record)

    ok = sum(1 for r in manifest if r["status"] == "ok")
    logger.info("Converted %d/%d Shapefiles to GeoParquet", ok, len(manifest))
    return manifest

To convert an entire directory, feed the recipe a glob:

from pathlib import Path

shp_paths = sorted(Path("data/legacy_shapefiles").rglob("*.shp"))
manifest = convert_shapefiles_to_geoparquet(
    shp_paths,
    out_dir="data/geoparquet",
    fallback_crs=4326,  # only used for files missing a .prj
)
failed = [r for r in manifest if r["status"] == "error"]

Key Implementation Notes

• Truncation is detected, not repaired. The truncated_columns field flags names that hit the 10-character wall so the loss is visible in the manifest, but the recipe deliberately does not guess the original names. Restoring them requires an explicit rename map applied before to_parquet — the same mapping used when standardizing column names across multiple Shapefiles, so the conversion and the harmonization share one source of truth.
• fallback_crs is opt-in on purpose. Defaulting a missing CRS to EPSG:4326 is a common way to silently mis-locate a projected dataset. Leaving fallback_crs=None makes an absent .prj a hard error the operator must resolve, and passing a code makes the assumption explicit and logged.
• ZSTD is the right default, Snappy the exception. ZSTD typically shrinks vector output two to four times versus the uncompressed Shapefile while decompressing fast enough that analytical scans remain I/O-bound. Switch to compression="snappy" only when write CPU, not storage, is the constraint.
• The round-trip check compares CRS objects, not strings. Two equivalent CRS definitions can differ as WKT strings, so the recipe compares pyproj.CRS instances; a raw string comparison would raise false mismatches on cosmetically different but equal projections.
• Legacy DBF encoding is a parameter, not a guess. Files without a .cpg sidecar are frequently Latin-1; passing encoding="latin-1" prevents mojibake in accented attribute text that would otherwise be frozen into the columnar output.
• The manifest makes the batch idempotent-friendly. Because each record names its output path and status, a re-run can skip files already marked ok and retry only the errors, without re-reading the whole archive.

Troubleshooting Conversion Failures

Integration Note

This recipe is the batch counterpart to the single-file write_geoparquet step in choosing a vector output format for spatial pipelines: the format-choice guide explains why GeoParquet is the analytical sink, and this page is how you migrate an existing Shapefile archive into it at scale. Run it after — never before — the naming reconciliation in standardizing column names across multiple Shapefiles, so the rename map fixes truncated attributes on the way into the columnar file rather than freezing them there permanently. The returned manifest slots directly into an orchestrated load: emit its error records to a dead-letter path and pass the ok outputs on to the analytical store.

Related

• Choosing a Vector Output Format for Spatial Pipelines — the format comparison that explains when GeoParquet is the right sink and when FlatGeobuf or GeoPackage fit better
• Standardizing Column Names Across Multiple Shapefiles — build the rename map that repairs 10-character truncation before conversion
• Attribute Mapping & Schema Harmonization — reconcile types and names across heterogeneous sources feeding one output
• Automated Vector & Raster Cleaning Workflows — the full reference for repairing and normalizing spatial data before load