This guide is part of Automating Government Portal Downloads, within the broader Mastering Geospatial Data Ingestion in Python reference.

Unzipping Shapefile Archives in Memory

Public portals distribute shapefiles as ZIP archives, and the reflex — extract to a temporary directory, read, delete — costs disk, time and a cleanup path that fails to run when a task is killed.

Why Extraction Is Worth Avoiding

  • Scratch space is finite and shared. A worker processing eight archives concurrently needs eight times the extracted size available at once.
  • Cleanup does not always happen. A killed task leaves a directory behind, and the next run finds a full disk rather than an error.
  • The archive is often much larger than the layer. A national release may contain forty layers when the pipeline needs one.
  • Archives are untrusted input. An archive with absolute paths, traversal entries or an enormous decompression ratio is a real risk when the source is a public portal.

Version and Environment Compatibility

Component Version Note
GDAL >=3.4 /vsizip/, /vsicurl/ and their composition
pyogrio >=0.9 Reads through GDAL paths without a temp copy
GeoPandas >=1.0 read_file with a vsizip path
Python 3.10+ zipfile.Path, union syntax 
pip install "geopandas>=1.0" "pyogrio>=0.9"

How the Virtual Filesystem Composes

Composing vsizip with vsicurl Three path forms. A local archive member is addressed with the vsizip prefix. A remote file is addressed with vsicurl. Composing them addresses a member inside a remote archive, and because the ZIP central directory is at the end of the file GDAL fetches only the ranges it needs rather than the whole archive. the same reader, three access paths /vsizip/data/parcels.zip/parcels.shp member of a local archive — no extraction, no temp files /vsicurl/https://portal.example/parcels.gpkg remote file read by range — only the pages the query touches /vsizip//vsicurl/https://portal.example/parcels.zip/parcels.shp member of a remote archive — the directory is read first, then the member The composed form is genuinely useful for one small layer inside a very large national release.

The read_zipped_shapefile Recipe

from __future__ import annotations

import logging
import zipfile
from pathlib import Path

import geopandas as gpd

logger = logging.getLogger(__name__)

REQUIRED_SIDECARS = (".dbf",)
MAX_UNCOMPRESSED_BYTES = 8 * 1024 ** 3
MAX_MEMBERS = 5_000
MAX_RATIO = 200


def list_shapefiles(archive: Path) -> list[str]:
    """Return the .shp members of an archive, after checking it is safe to read."""
    with zipfile.ZipFile(archive) as zf:
        infos = zf.infolist()

        if len(infos) > MAX_MEMBERS:
            raise ValueError(f"{archive.name}: {len(infos)} members exceeds the limit")

        uncompressed = sum(i.file_size for i in infos)
        compressed = sum(i.compress_size for i in infos) or 1
        if uncompressed > MAX_UNCOMPRESSED_BYTES:
            raise ValueError(f"{archive.name}: {uncompressed / 1e9:.1f} GB uncompressed")
        if uncompressed / compressed > MAX_RATIO:
            raise ValueError(f"{archive.name}: compression ratio {uncompressed / compressed:.0f}:1")

        for info in infos:
            name = info.filename
            if name.startswith("/") or ".." in Path(name).parts:
                raise ValueError(f"{archive.name}: unsafe member path {name!r}")

        names = [i.filename for i in infos]
        shapefiles = [n for n in names if n.lower().endswith(".shp")]

        for shp in shapefiles:
            stem = shp[:-4]
            for suffix in REQUIRED_SIDECARS:
                if not any(n.lower() == (stem + suffix).lower() for n in names):
                    raise ValueError(f"{shp}: required sidecar {suffix} missing from the archive")

        logger.info("%s: %d layer(s) — %s", archive.name, len(shapefiles), shapefiles[:5])
        return shapefiles


def read_zipped_shapefile(archive: Path, member: str | None = None) -> gpd.GeoDataFrame:
    """Read one shapefile out of a ZIP without extracting anything."""
    members = list_shapefiles(archive)
    if member is None:
        if len(members) != 1:
            raise ValueError(f"{archive.name} holds {len(members)} layers — name one explicitly")
        member = members[0]

    path = f"/vsizip/{archive.as_posix()}/{member}"
    gdf = gpd.read_file(path, engine="pyogrio")

    if gdf.crs is None:
        logger.warning("%s/%s: no .prj — CRS must be supplied downstream", archive.name, member)
    logger.info("read %d features from %s/%s", len(gdf), archive.name, member)
    return gdf

Key Implementation Notes

  • The guardrails run before any read. Member count, uncompressed size, compression ratio and path safety are all cheap to check from the central directory, and a public portal is an untrusted source however official it looks.
  • A missing .dbf is fatal, a missing .prj is a warning. Without attributes the layer is useless; without a CRS it is usable once the reference system is supplied, as covered in CRS normalization across mixed datasets.
  • Multiple layers require an explicit choice. Silently picking the first .shp in a national release is how the wrong dataset reaches production.
  • as_posix() builds the vsizip path. Backslashes in a Windows path break the virtual filesystem, and the failure message is unhelpful.
  • Case-insensitive sidecar matching. Archives produced on different systems mix .DBF and .dbf, and a case-sensitive check rejects valid archives.
  • The reader is pyogrio. It reads through GDAL without a temporary copy, which is the whole point of the exercise.
Extraction versus direct member access Two paths to the same GeoDataFrame. Extraction writes three gigabytes to scratch space, reads one forty megabyte layer, and requires a cleanup step that may not run. Direct access through the virtual filesystem reads the archive's directory and then only the member's compressed bytes, writing nothing. extract, then read archive 3 GB 3 GB on scratch read 40 MB + cleanup that may not run read the member directly archive 3 GB /vsizip/ member read 40 MB nothing written at all

Encoding, and Why the .cpg Matters

Shapefile attributes are bytes with a declared encoding that is frequently absent or wrong. A .cpg member naming UTF-8 is a promise; a missing one means GDAL falls back to a platform default, which on a Linux worker differs from the Windows machine that produced the file.

The encoding fallback ladder Four steps in order. The cpg member's declaration is tried first. If it is absent or fails, cp1252 is tried strictly, then latin-1, which accepts any byte sequence. The encoding that succeeded is recorded with the batch. Replacing undecodable bytes is never a step, because it produces text that looks valid and joins to nothing. decode strictly, in order, and record the winner 1 · the .cpg declaration — authoritative when it exists, absent surprisingly often 2 · cp1252, strict — the usual answer for European government exports 3 · latin-1, strict — accepts any byte, so it always "works"; note that in the log never · errors="replace" — silent corruption that passes every later check

The practical approach is to read the .cpg when present, decode strictly, and fall back through cp1252 and latin-1 while recording which encoding actually worked. A place-name column containing replacement characters is a failed read, not a cosmetic issue — the mechanism is the same one described in web scraping spatial metadata, and the consequence is the same: two spellings of one municipality that never join.

Where a portal reliably produces the same encoding, pin it in the connector configuration rather than detecting it per file. Detection is a fallback, not a strategy.

Troubleshooting Archive Reads

Symptom Likely cause Fix
No such file or directory on a valid path Backslashes in the vsizip path Build the path with as_posix()
Attributes are all null .dbf missing from the archive Reject the archive at the listing stage
Layer opens with no CRS .prj absent Warn and supply the CRS from configuration
Mojibake in name columns .cpg missing, wrong fallback encoding Decode strictly with an ordered fallback
Worker disk fills Extraction path still in use somewhere Read through /vsizip/ everywhere
Read is very slow over HTTP Composed vsizip//vsicurl on a huge archive Download once, then read the member locally

Integration Note

This read belongs at the ingestion boundary, immediately after the conditional download described in detecting dataset changes with ETag and Last-Modified: the archive is fetched only when it changed, and read without ever being unpacked. Record the member name alongside the batch, because a portal that adds a second layer to an existing archive will otherwise change what the pipeline ingests without changing anything you monitor.

Archives That Contain Other Archives

Portals occasionally publish a ZIP of ZIPs — one archive per municipality inside a national release. GDAL’s virtual filesystem composes here too, so a member of an inner archive is addressable in one path, but the nesting is worth handling deliberately rather than incidentally.

Enumerate the outer archive first and treat each inner archive as its own unit of work, with its own guardrail check, its own read and its own row in the ingestion manifest. That keeps a single corrupt municipality from failing the national batch, and it makes the per-unit counts meaningful.

Depth is the thing to bound. Two levels are common and legitimate; three suggests either an unusual publication process or a deliberately hostile file, and refusing to recurse past a fixed depth costs nothing while removing a class of resource exhaustion. The same applies to member counts, which should be checked at every level rather than only at the outer one.