This guide is part of Web Scraping Spatial Metadata, within the broader Mastering Geospatial Data Ingestion in Python reference.

Extracting Dataset URLs from ArcGIS Hub Catalogs

A great many public authorities publish through ArcGIS Hub, and the catalogue page that looks like a scraping target is a client-side rendering of a JSON API. Calling that API directly is faster, more stable and considerably kinder to the publisher.

Why the API Path Wins

• The page carries no data. The initial HTML is an application shell; every dataset row arrives by a later request.
• Records are typed and identified. Each result has a stable item id, a type and a service URL, none of which a parsed page gives you reliably.
• Filtering happens server-side. Restricting to feature layers updated in the last month is a query parameter rather than a post-filter.
• The layout changes and the API does not. A theme update breaks a scraper and leaves an API client untouched.

Version and Environment Compatibility

pip install "requests>=2.32" "pandas>=2.2" "geopandas>=1.0"

From Catalogue to Queryable Endpoint

The harvest_hub_catalog Recipe

from __future__ import annotations

import logging

import pandas as pd
import requests

logger = logging.getLogger(__name__)

SEARCH_URL = "https://hub.arcgis.com/api/v3/datasets"


def harvest_hub_catalog(
    site: str,
    page_size: int = 100,
    types: tuple[str, ...] = ("Feature Service",),
    max_pages: int = 200,
    session: requests.Session | None = None,
) -> pd.DataFrame:
    """Enumerate a Hub site's datasets through its search API rather than its pages."""
    session = session or requests.Session()
    session.headers.setdefault("User-Agent", "spatial-etl/1.0 (data@example.org)")

    records: list[dict] = []
    for page in range(1, max_pages + 1):
        response = session.get(
            SEARCH_URL,
            params={
                "filter[catalog][site]": site,
                "page[size]": page_size,
                "page[number]": page,
                "fields[datasets]": "name,type,url,modified,recordCount,owner",
            },
            timeout=60,
        )
        response.raise_for_status()
        payload = response.json()
        items = payload.get("data", [])
        if not items:
            break

        for item in items:
            attributes = item.get("attributes", {})
            if types and attributes.get("type") not in types:
                continue
            records.append({
                "item_id": item.get("id"),
                "title": attributes.get("name"),
                "type": attributes.get("type"),
                "service_url": attributes.get("url"),
                "modified": attributes.get("modified"),
                "record_count": attributes.get("recordCount"),
                "owner": attributes.get("owner"),
            })

        if len(items) < page_size:
            break
    else:
        raise RuntimeError(f"{site}: exceeded {max_pages} pages — check the paging parameters")

    frame = pd.DataFrame(records).drop_duplicates(subset="item_id")
    frame["query_url"] = frame["service_url"].fillna("") + "/0/query"
    logger.info("%s: %d datasets harvested", site, len(frame))
    return frame


def diff_registry(previous: pd.DataFrame, current: pd.DataFrame) -> dict[str, pd.DataFrame]:
    """Compare two harvests on item_id to find additions, removals and updates."""
    merged = previous.merge(current, on="item_id", how="outer",
                            suffixes=("_before", "_after"), indicator=True)
    return {
        "added": merged.loc[merged["_merge"] == "right_only"],
        "removed": merged.loc[merged["_merge"] == "left_only"],
        "updated": merged.loc[
            (merged["_merge"] == "both")
            & (merged["modified_before"] != merged["modified_after"])
        ],
    }

Key Implementation Notes

• The registry keys on item_id. Titles and URLs both change when a publisher reorganises; the identifier does not, and keying on anything else produces phantom additions every quarter.
• Layer index zero is an assumption. Many services publish several layers, and the right index has to be read from the service metadata rather than guessed — the recipe’s /0/ is a starting point, not a rule.
• The modified field drives the diff. Comparing it between harvests is what turns a catalogue listing into a change feed without downloading anything.
• Paging terminates on a short page. The API’s total count is not always present, and the short-page test works regardless.
• Duplicates are dropped on item_id. Hub catalogues legitimately list the same item under several categories, and a naive concatenation double-counts.
• Type filtering happens client-side here because the filter parameter’s spelling has changed across API versions; filtering after the fetch is stable and the pages are small.

Resolving the Right Layer

A feature service is a container, and assuming layer zero is the most common source of harvested endpoints that return the wrong data. The service root returns its layer list as JSON, with each layer’s name, geometry type and record count, and reading it is one extra request per dataset.

Match on layer name where the registry knows what it wants, and fall back to the single layer when a service has exactly one. Where a service has several and no rule applies, record all of them rather than choosing: a registry entry per layer is cheap and correct, and it surfaces the ambiguity to a human rather than resolving it invisibly.

Store the resolved layer index in the registry alongside the item id, so the resolution happens once rather than on every ingestion run — and so a service that gains a layer does not silently shift what index zero refers to.

Troubleshooting Catalogue Harvests

Integration Note

The harvest is a discovery job that runs on its own slow schedule — weekly is usually plenty — and writes the registry that the ingestion pipeline reads. Keeping the two separate means a catalogue outage delays discovery rather than stopping ingestion, and it makes the diff a reviewable artefact. Downstream, each resolved endpoint is read with the paging and error handling described in parsing GeoJSON and shapefile APIs, including the ArcGIS-specific error codes that arrive inside a 200 response.

What the Registry Should Carry Beyond the URL

A registry that holds only endpoints answers one question. A few more columns make it answer the questions that actually come up.

Licence and attribution as published, because the obligation travels with the data and is far easier to record at discovery than to reconstruct later.

Record count and last modified, which together decide whether an ingestion is worth running and give the freshness check something to compare against.

Owner and contact, so a broken endpoint has somebody to ask rather than a ticket that ages.

Our own classification: which pipeline consumes it, at what cadence, and whether anyone downstream depends on it. That last column is the one that makes a removal actionable — a dataset nobody consumes disappearing from a catalogue is a note, and one that feeds a published product is an incident.

None of this is expensive to collect at harvest time and all of it is expensive to reconstruct afterwards, which is the usual shape of metadata decisions.

Related

• Web Scraping Spatial Metadata
• Parsing GeoJSON & Shapefile APIs
• Handling Authentication Tokens for ArcGIS REST Services
• Parsing ISO 19115 Metadata with OWSLib