This guide is part of Fetching OSM Data via Overpass API, within the broader Mastering Geospatial Data Ingestion in Python reference.

Converting Overpass JSON to GeoDataFrames

Overpass returns a flat list of elements — nodes, ways and relations — in which geometry is a set of references rather than coordinates. Turning that into usable frames is a resolution problem, and it has three traps.

Why the Naive Conversion Fails

• Ways carry node ids, not coordinates. Building geometry means resolving every reference against the node index in the same response.
• A closed way is not necessarily an area. OSM distinguishes a building from a ring road by tags, not by geometry.
• Tags are an open-ended dictionary. Promoting all of them to columns produces a frame with thousands of mostly-null columns.
• Mixed geometry in one frame breaks writers. GeoParquet and PostGIS both prefer homogeneous geometry per table.

Version and Environment Compatibility

pip install "geopandas>=1.0" "shapely>=2.0" "pandas>=2.2"

How Elements Resolve Into Geometry

The overpass_to_frames Recipe

from __future__ import annotations

import json
import logging

import geopandas as gpd
import pandas as pd
import shapely

logger = logging.getLogger(__name__)

# Tags whose presence means a closed way encloses an area rather than looping.
AREA_TAGS = {"building", "landuse", "natural", "leisure", "amenity", "area"}
PROMOTED_TAGS = ("name", "building", "highway", "landuse", "amenity", "surface")


def overpass_to_frames(payload: dict, crs: int = 4326) -> dict[str, gpd.GeoDataFrame]:
    """Split an Overpass JSON response into point, line and polygon frames."""
    elements = payload.get("elements", [])
    nodes = {e["id"]: (e["lon"], e["lat"]) for e in elements if e["type"] == "node"}
    logger.info("indexed %d nodes from %d elements", len(nodes), len(elements))

    points, lines, polygons = [], [], []

    for element in elements:
        tags = element.get("tags") or {}
        base = {"osm_id": element["id"], "osm_type": element["type"], **_tag_columns(tags)}

        if element["type"] == "node":
            if tags:  # untagged nodes are geometry carriers, not features
                points.append(base | {"geometry": shapely.Point(nodes[element["id"]])})
            continue

        if element["type"] == "way":
            refs = element.get("nodes") or []
            coords = [nodes[ref] for ref in refs if ref in nodes]
            if len(coords) < 2:
                logger.debug("way %s unresolvable — %d of %d nodes present",
                             element["id"], len(coords), len(refs))
                continue

            closed = len(coords) >= 4 and coords[0] == coords[-1]
            if closed and AREA_TAGS.intersection(tags):
                polygons.append(base | {"geometry": shapely.Polygon(coords)})
            else:
                lines.append(base | {"geometry": shapely.LineString(coords)})

    return {
        "points": _frame(points, crs),
        "lines": _frame(lines, crs),
        "polygons": _frame(polygons, crs),
    }


def _tag_columns(tags: dict) -> dict:
    """Promote the tags the pipeline uses; keep everything else as one JSON column."""
    promoted = {key: tags.get(key) for key in PROMOTED_TAGS}
    remainder = {k: v for k, v in tags.items() if k not in PROMOTED_TAGS}
    return promoted | {"tags_json": json.dumps(remainder, sort_keys=True) if remainder else None}


def _frame(rows: list[dict], crs: int) -> gpd.GeoDataFrame:
    if not rows:
        return gpd.GeoDataFrame(geometry=[], crs=crs)
    return gpd.GeoDataFrame(pd.DataFrame(rows), geometry="geometry", crs=crs)

Key Implementation Notes

• The node index is built in one pass before anything is resolved. Resolving during iteration fails whenever a way appears before its nodes, which Overpass does not guarantee.
• Untagged nodes are skipped as features. They exist to carry geometry for ways; emitting them as points produces millions of meaningless rows.
• Closure alone does not make a polygon. The AREA_TAGS intersection is what distinguishes a building from a ring road, and getting it wrong turns roundabouts into land parcels.
• Partially resolved ways are dropped with a debug line, not an exception. A bbox query legitimately clips ways at the boundary, and failing the batch for that would make every extraction fragile.
• Tags are split into promoted columns plus one JSON column. This keeps the schema stable across extractions while losing nothing, and the JSON column is queryable in both Parquet and PostGIS.
• Three frames are returned, not one. Homogeneous geometry per frame is what lets each be written to its own table or file without a geometry-type conflict.

Relations and When to Bother

Relations assemble multipolygons from member ways with outer and inner roles, and handling them properly is a meaningful amount of code — the ring-assembly problem covered in fetching OSM data via Overpass API.

Whether to implement it depends on the features being extracted. Buildings, small landuse parcels and points of interest are overwhelmingly ways; skipping relations loses a fraction of a percent. Administrative boundaries, large forests, lakes with islands and coastlines are predominantly relations; skipping them loses most of the data.

Where relations matter and the assembly logic is not the point of the project, osmnx and pyrosm both implement it and both are worth preferring over a hand-rolled version. The conversion above remains useful for the way-and-node majority, and for the cases where a dependency is unwelcome.

Troubleshooting the Conversion

Integration Note

This conversion is the parsing half of the ingestion split described in mastering geospatial data ingestion in Python: the transport function returns bytes and this returns frames, which makes the parser testable against saved responses with no network. Downstream, the polygon frame usually needs the validity pass from geometry repair with Shapely and GeoPandas, because community-edited rings self-intersect more often than surveyed ones.

Keeping OSM Identifiers Stable

The osm_id plus osm_type pair is the only stable key an extraction has, and treating it as the primary key changes what a re-run means.

An element keeps its id across edits: a building whose outline is corrected is the same id with different geometry, which is exactly what an upsert should treat as an update. Deleting and re-inserting the whole extract loses that distinction and makes every downstream change-detection query useless.

Two caveats. The id is only unique within a type — node 12345 and way 12345 are different elements — so the key must be the pair. And an element that is deleted upstream simply stops appearing in the extract; detecting deletions requires comparing the current id set against the previous one, which is another reason to store the extraction’s id set rather than only its rows.

Related

• Fetching OSM Data via Overpass API
• Writing Overpass QL Queries for Large Areas
• How to Handle Rate Limits When Downloading OSM Data
• Geometry Repair with Shapely & GeoPandas