This guide is part of Fetching OSM Data via Overpass API, within the broader Mastering Geospatial Data Ingestion in Python reference.

Writing Overpass QL Queries for Large Areas

A query that works over a neighbourhood frequently fails over a region, and the failure is nearly always a resource limit rather than a syntax problem. The query language gives you three levers over that, and using them is mostly a matter of ordering the statements so the expensive part sees fewer elements.

Why Large-Area Queries Fail

  • The server’s default timeout is short. A query that needs 300 seconds is killed at the default 180 with no partial result.
  • Recursion happens before filtering when the query is written that way. Pulling every node in a bbox and then filtering by tag makes the server materialise millions of elements it will discard.
  • Memory limits are hit by the intermediate set, not the output. A query returning 4 000 buildings can still exceed maxsize if it built a set of two million elements on the way.
  • Splitting by area is the wrong axis. A rural cell and an urban cell of the same size differ in element count by two orders of magnitude.

Version and Environment Compatibility

Component Version Note
Overpass API 0.7.57+ out geom, nwr shorthand, [maxsize] honoured
overpy >=0.7 Optional client; the recipe uses requests directly
requests >=2.32 Session reuse and explicit timeouts
shapely >=2.0 Cell geometry for the density split 
pip install "requests>=2.32" "shapely>=2.0" "geopandas>=1.0"

Where the Cost Goes

Where the intermediate set explodes Two query shapes over the same extent. The first selects all elements in the bounding box, recurses to their nodes, and only then filters by tag, producing an intermediate set of about two million elements. The second applies the tag filter in the selecting statement, so the intermediate set is about forty thousand and the recursion is proportionally cheaper. Both return the same four thousand buildings. filter last · intermediate set explodes select bbox ≈ 2 000 000 elements held filter → 4 000 filter first · the recursion sees far less select + filter ≈ 40 000 recurse → 4 000 same answer, 50× less memory The output is identical either way, which is why the expensive version survives review — the cost is entirely invisible in the result.

The Query Builder Recipe

from __future__ import annotations

import logging
import textwrap

import requests

logger = logging.getLogger(__name__)

ENDPOINT = "https://overpass-api.de/api/interpreter"


def build_query(
    bbox: tuple[float, float, float, float],
    filters: list[str],
    element_types: str = "nwr",
    timeout_s: int = 600,
    maxsize_bytes: int = 1_073_741_824,
    area_id: int | None = None,
) -> str:
    """Compose an Overpass QL query with explicit budgets and early filtering.

    `filters` are tag selectors such as ['"building"', '"building"="residential"'];
    they are applied in the selecting statement so the intermediate set stays small.
    """
    south, west, north, east = bbox
    scope = f"(area:{area_id})" if area_id else f"({south},{west},{north},{east})"

    selectors = "\n  ".join(f"{element_types}[{tag}]{scope};" for tag in filters)

    return textwrap.dedent(f"""
        [out:json][timeout:{timeout_s}][maxsize:{maxsize_bytes}];
        (
          {selectors}
        );
        out geom qt;
    """).strip()


def run_query(query: str, session: requests.Session | None = None,
              read_timeout: int = 900) -> dict:
    """POST the query, raising a clear error when the server reports a budget failure."""
    session = session or requests.Session()
    session.headers.setdefault("User-Agent", "spatial-etl/1.0 (data@example.org)")

    response = session.post(ENDPOINT, data={"data": query}, timeout=(30, read_timeout))
    if response.status_code == 400:
        raise ValueError(f"Overpass rejected the query: {response.text[:400]}")
    if response.status_code in (429, 504):
        raise TimeoutError(f"Overpass refused or timed out ({response.status_code})")
    response.raise_for_status()

    payload = response.json()
    remark = payload.get("remark")
    if remark:
        # A remark is how Overpass reports a partial result — never ignore it.
        raise RuntimeError(f"incomplete result: {remark}")

    logger.info("query returned %d elements", len(payload.get("elements", [])))
    return payload

Key Implementation Notes

  • timeout and maxsize are declared, never defaulted. The defaults are tuned for interactive use; a pipeline should state what it needs and fail clearly when the server will not grant it.
  • Filters go inside the selecting statement. nwr["building"](bbox) is a different query from nwr(bbox) followed by a tag filter, and the difference is the size of the intermediate set.
  • out geom returns way geometry inline. The alternative — recursing to nodes and resolving client-side — transfers far more and is only necessary when the node ids themselves are needed.
  • qt sorts by quadtile. Output arrives in spatial order, which makes the downstream write spatially coherent for free.
  • A remark in the payload means a truncated answer. It arrives with HTTP 200 and a plausible-looking element list, which is the single most dangerous failure mode of the API; raising on it is essential.
  • The read timeout exceeds the query timeout. A client that gives up before the server does turns a slow success into a retry storm.
Equal area versus equal element count The same extent divided two ways. Equal-area cells give a central urban cell containing 1.4 million elements while surrounding rural cells hold a few thousand each, so one request fails and the rest are wasteful. Density-aware cells subdivide the urban area further and merge the rural ones, giving every cell a similar element count and therefore a similar cost. equal area · one cell fails 1.4 M 5 cells return in seconds, 1 times out equal element count · all succeed rural, merged 9 cells, similar cost, all inside the budget

Estimating Density Before Splitting

Splitting by density requires knowing the density, and the cheapest estimate is a count-only query per candidate cell. Overpass supports out count;, which returns element counts without any geometry, at a fraction of the cost of the real query.

Subdividing only where the count demands it A four-cell grid where three cells hold few elements and remain whole, while the fourth exceeds the budget and is subdivided into four, one of which exceeds it again and is subdivided a second time. Only the count-only queries are issued during this pass, so planning the split costs a fraction of the extraction it plans. out count; per cell → subdivide only where needed 18 k 24 k 31 k budget: 60 k elements per request three cells pass on the first pass the city cell splits, then splits again the cell list is cached and reused — density changes far slower than data does

The practical procedure is a quadtree: run count queries over a coarse grid, subdivide any cell above the target count, and repeat until every leaf is under it. Two or three levels are usually enough, and the counting pass costs a small fraction of the extraction it plans.

Cache the resulting cell list next to the extraction configuration and refresh it occasionally rather than per run. Feature density changes slowly, and a stable cell list also makes the extraction’s per-cell outputs comparable between runs — which matters for the deduplication of edge-crossing features described in fetching OSM data via Overpass API.

Troubleshooting Large Queries

Symptom Likely cause Fix
504 after the full timeout Query genuinely too large for one request Split by density; raise timeout only if close
“runtime error: Query run out of memory” Intermediate set too large Move filters into the selecting statement; raise maxsize
200 with a remark field Partial result returned Raise on remark; re-run the cell smaller
Way geometry missing out; used instead of out geom; Request geometry, or add node recursion
Duplicate features between cells Features crossing cell boundaries Deduplicate on osm_id after concatenation
Rate limited despite few requests Query cost, not request count, is metered Reduce per-query cost; back off as usual

Integration Note

Cell enumeration, query construction and execution are three separable steps, and keeping them separate is what makes the extraction resumable: the cell list is state, each cell’s result is an artefact, and a failed cell retries alone. That maps directly onto the mapped-task pattern in parallel tile downloads with Prefect task mapping, with the concurrency cap set by the endpoint’s tolerance rather than by the worker count.

Public Instances, Mirrors and Self-Hosting

A large-area extraction eventually outgrows what a shared public endpoint should be asked to do, and there are three ways past that point.

Use a mirror. Several instances run the same API with independent quotas. Rotating between them spreads the load and gives the tiered fallback described in the rate-limit guide something to fall back to. Check each mirror’s stated policy — some ask that heavy users identify themselves.

Switch to an extract. For anything covering a country or larger, a regional .osm.pbf from a provider like Geofabrik plus a local reader is faster, kinder and entirely quota-free. The trade is freshness: extracts are cut daily rather than continuously.

Self-host the API. Running an Overpass instance against a local database removes every limit and adds an operational burden measured in hundreds of gigabytes and a multi-day initial import. It is worth it for teams whose work is predominantly OSM and rarely otherwise.

The decision usually resolves on freshness. If minute-level currency matters, stay on the API and split harder; if daily is enough, the extract path is better on every other axis.