Handling authentication tokens for ArcGIS REST services

This guide is part of Parsing GeoJSON & Shapefile APIs, which covers the full workflow for fetching, validating, and persisting vector data from REST endpoints.

ArcGIS REST token authentication fails spatial ETL pipelines when scripts submit credentials on every request: the /generateToken endpoint imposes per-IP rate limits that trigger HTTP 429 responses under sustained load, and each credential POST adds 200–400 ms of latency per call. The correct pattern is to generate a short-lived token once, cache it in memory alongside its millisecond-epoch expiry timestamp, and proactively refresh it 120 seconds before it expires — before any in-flight spatial query hits an expired credential mid-stream.

Why expired or misconfigured tokens break spatial pipelines

Token failures are silent killers in automated ingestion. They do not raise immediate Python exceptions; instead, the ArcGIS REST endpoint returns a 200 OK response with a JSON body containing an error key. A pipeline that only checks HTTP status codes will log “success” while persisting zero features.

• Silent empty-result returns: A 498 Invalid Token or 499 Token Required error inside a 200 body means geopandas.read_file() receives an empty feature collection rather than a geometry error, producing a zero-row GeoDataFrame that silently propagates downstream.
• Cascading join failures: Downstream spatial joins against an empty or incomplete dataset produce silent attribute nulls. If the join feeds a rasterization step, entire grid cells are dropped with no traceable error.
• Broken pagination: Paginated feature service queries (using resultOffset) that span a token expiry boundary return the first page successfully but error silently on subsequent pages, delivering partial datasets.
• Audit trail gaps: Without explicit token-lifecycle logging, on-call engineers see only a downstream anomaly — an unusually small file, an empty PostGIS table — with no record of when or why authentication lapsed.

Token lifecycle diagram

Version and environment compatibility

Production-ready ArcGISTokenManager recipe

The function below is the single piece of code your pipeline needs. It encapsulates credential storage, epoch conversion, proactive refresh, and request dispatch inside a reusable manager class. Import it once per pipeline run and call request_service() instead of requests.get() directly.

import os
import time
import logging
from typing import Any, Dict, Optional

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

logger = logging.getLogger(__name__)


class ArcGISTokenManager:
    """
    Manages ArcGIS REST token generation, caching, and proactive refresh.

    Token expiry is returned by the server in milliseconds since Unix epoch.
    Tokens are refreshed automatically when fewer than REFRESH_BUFFER seconds remain.

    Usage:
        mgr = ArcGISTokenManager.from_env("https://services.arcgis.com/YOUR_ORG")
        data = mgr.request_service("https://services.arcgis.com/.../FeatureServer/0/query",
                                   params={"where": "1=1", "outFields": "*"})
    """

    REFRESH_BUFFER: int = 120  # seconds before expiry to trigger proactive renewal

    def __init__(
        self,
        base_url: str,
        username: str,
        password: str,
        referer: str = "https://geospatial-etl.com",
        client: str = "requestip",
    ) -> None:
        self.base_url = base_url.rstrip("/")
        self.username = username
        self.password = password
        self.referer = referer
        self.client = client

        self._token: Optional[str] = None
        self._expires_at: float = 0.0  # Unix seconds

        # Reuse a single session for connection pooling across all requests
        retry = Retry(
            total=3,
            backoff_factor=1.0,
            status_forcelist=[429, 500, 502, 503, 504],
            allowed_methods=["GET", "POST"],
        )
        self._session = requests.Session()
        self._session.mount("https://", HTTPAdapter(max_retries=retry))

    @classmethod
    def from_env(
        cls,
        base_url: str,
        referer: str = "https://geospatial-etl.com",
        client: str = "requestip",
    ) -> "ArcGISTokenManager":
        """Construct from ARCGIS_USERNAME / ARCGIS_PASSWORD environment variables."""
        username = os.environ["ARCGIS_USERNAME"]
        password = os.environ["ARCGIS_PASSWORD"]
        return cls(base_url, username, password, referer=referer, client=client)

    def _generate_token(self) -> str:
        """POST credentials to /sharing/rest/generateToken and cache the result."""
        endpoint = f"{self.base_url}/sharing/rest/generateToken"
        payload = {
            "username": self.username,
            "password": self.password,
            "referer": self.referer,
            "client": self.client,
            "expiration": 60,  # minutes; adjust to match your ArcGIS Server policy
            "f": "json",
        }
        resp = self._session.post(endpoint, data=payload, timeout=15)
        resp.raise_for_status()
        data = resp.json()

        # ArcGIS wraps auth errors in a 200 response body — always inspect
        if "error" in data:
            code = data["error"].get("code", "unknown")
            msg = data["error"].get("message", "")
            raise RuntimeError(
                f"ArcGIS token generation failed [{code}]: {msg}"
            )

        self._token = data["token"]
        # expires is milliseconds since Unix epoch — convert to seconds
        self._expires_at = data["expires"] / 1000.0

        logger.info(
            "ArcGIS token acquired; expires in %.0f seconds",
            self._expires_at - time.time(),
        )
        return self._token

    def get_valid_token(self) -> str:
        """Return a cached token, refreshing proactively if within REFRESH_BUFFER."""
        if not self._token or (self._expires_at - time.time()) < self.REFRESH_BUFFER:
            return self._generate_token()
        return self._token

    def request_service(
        self,
        service_url: str,
        params: Optional[Dict[str, Any]] = None,
        method: str = "GET",
    ) -> Dict[str, Any]:
        """
        Make an authenticated request to an ArcGIS REST endpoint.

        The token is appended as a query parameter (required by many legacy
        feature services) and as an Authorization header for modern endpoints.
        """
        token = self.get_valid_token()
        merged_params = {"f": "json", "token": token, **(params or {})}
        headers = {"Referer": self.referer}

        if method.upper() == "POST":
            resp = self._session.post(
                service_url, data=merged_params, headers=headers, timeout=30
            )
        else:
            resp = self._session.get(
                service_url, params=merged_params, headers=headers, timeout=30
            )

        resp.raise_for_status()
        data = resp.json()

        # Surface ArcGIS-level errors that arrive inside 200 responses
        if "error" in data:
            code = data["error"].get("code", "unknown")
            msg = data["error"].get("message", "")
            raise RuntimeError(f"ArcGIS service error [{code}]: {msg}")

        return data

Key implementation notes

• from_env class method: Credentials are read from ARCGIS_USERNAME and ARCGIS_PASSWORD environment variables at instantiation, never from hard-coded strings. CI/CD systems and container orchestrators inject secrets this way without exposing them in source control or log output.

• Millisecond-to-second epoch conversion: The ArcGIS generateToken response returns expires in milliseconds since the Unix epoch, not seconds. Dividing by 1000.0 is mandatory; treating the raw value as seconds makes every token appear to expire roughly 1,140 years in the future, disabling the refresh buffer entirely.

• Inspecting 200 OK bodies for error keys: ArcGIS REST wraps authentication failures, service errors, and quota violations inside HTTP 200 responses. Any code that only checks raise_for_status() will silently treat a 498 Invalid Token error body as a success. Both _generate_token() and request_service() explicitly inspect the response JSON for "error" before returning.

• client=requestip vs client=referer: requestip binds the token to the originating IP address — straightforward for a single-server ETL host. referer instead validates the HTTP Referer header on every subsequent request; the value must match exactly (including protocol and trailing slash) what was submitted during token generation. Mismatches produce 498 errors that are hard to diagnose without side-by-side comparison of the registered and transmitted referer strings.

• expiration parameter scoping: The expiration field (in minutes) requests a token lifetime from the server, but the server may cap it to the maximum allowed by the ArcGIS Server security policy. Always trust the expires value in the response, not the requested duration, when computing _expires_at.

• Single requests.Session with retry adapter: Reusing one session object across all token and service calls keeps the TCP connection pool alive, reducing per-request TLS handshake overhead by 100–300 ms on high-frequency pipelines. The Retry adapter with backoff_factor=1.0 handles transient 503 and 429 responses without manual time.sleep() calls.

Deployment checklist

Before scheduling this recipe in an automated pipeline, verify the following:

1. Confirm referer matches the allowed referrer registered under ArcGIS Server Manager → Security → Token Settings (or the equivalent ArcGIS Online app configuration).
2. Test that ARCGIS_USERNAME and ARCGIS_PASSWORD are injected as environment variables — not present in docker inspect, ps aux, or application logs.
3. Run a 10-minute soak test against a dev feature service to confirm the 120-second refresh buffer triggers correctly before expiry.
4. Add a log alert on "ArcGIS token generation failed" messages so on-call engineers are notified within minutes of credential rotation breaking the pipeline.
5. Rotate base credentials at least quarterly and invalidate existing tokens via the ArcGIS Administrator API immediately after rotation.

Troubleshooting: ArcGIS error codes

Integration note

This recipe slots in as the authentication layer immediately before the vector fetch step in Parsing GeoJSON & Shapefile APIs. Instantiate ArcGISTokenManager once at pipeline startup, then pass mgr.request_service(feature_service_url, params=query_params) results directly to geopandas.GeoDataFrame.from_features(). When the feature service returns paginated results, the resultOffset loop calls request_service() on each page — the proactive refresh buffer ensures no page boundary coincides with a token expiry. For spatial queries that require bounding-box pre-filtering before the authenticated call, see Extracting Bounding Boxes from GeoJSON APIs for coordinate-order-safe extent computation.

Related

• Parsing GeoJSON & Shapefile APIs in Python — full ingestion workflow: fetch, validate, normalize, and persist vector data from REST endpoints
• Extracting Bounding Boxes from GeoJSON APIs — RFC 7946 bbox extraction and spatial pre-filtering before authenticated queries
• Mastering Geospatial Data Ingestion in Python — top-level overview covering all ingestion sources: OSM, STAC, government portals, and commercial REST APIs
• Automating Government Portal Downloads — session-cookie and form-based authentication patterns for non-ArcGIS portals