This guide is part of Automating Government Portal Downloads, which itself sits within the broader Mastering Geospatial Data Ingestion in Python reference.

Detecting Dataset Changes With ETag and Last-Modified

The Problem: Blind Re-Downloads Waste Bandwidth and Corrupt Freshness Signals

Government portals publish shapefiles, GeoPackages, and CSV extracts that change on irregular cadences — a parcel layer might refresh quarterly, a flood-zone dataset yearly, and a permits feed nightly. A pipeline that unconditionally re-fetches every URL on every run downloads gigabytes it already has, and worse, it loses any reliable signal about when a dataset actually changed.

Naive scheduled downloads break spatial pipelines in specific ways:

• Bandwidth and rate-limit exhaustion: Re-pulling a 2 GB statewide parcel file nightly when it changes quarterly burns the portal’s quota and can trip IP-level throttling that blocks the datasets you do need.
• False change events downstream: Without a validator, every download looks new, so schema validation, reprojection, and PostGIS loads all re-run — inflating warehouse costs and masking genuine deltas behind constant churn.
• Race conditions on partial writes: Overwriting an existing extract mid-download leaves a truncated file that the next stage happily parses, silently dropping features.
• No audit trail of freshness: Compliance and reproducibility both require knowing the exact version ingested; a blind overwrite destroys the Last-Modified timestamp that proves it.

Conditional GET fixes all four by letting the server decide whether the bytes changed, using the HTTP validators it already emits.

Version and Environment Compatibility

pip install "requests>=2.31.0" "platformdirs>=4.0"

How Conditional GET Skips Unchanged Datasets

An ETag is an opaque token the server assigns to a specific version of a resource; a Last-Modified header is a timestamp of the last edit. On a follow-up request you echo the stored ETag back in an If-None-Match header and the stored timestamp in If-Modified-Since. If neither has changed, the server replies 304 Not Modified with an empty body — no payload transfer, no bandwidth. Only a 200 OK carries fresh bytes and fresh validators to persist.

The fetch_if_changed Recipe

The function below performs a conditional GET, handles 304, persists validators on 200, and falls back to content hashing when a portal emits neither header. It returns a (changed, payload) tuple so callers can branch cleanly.

import hashlib
import json
import logging
from pathlib import Path
from typing import Optional

import requests

logger = logging.getLogger(__name__)


class ValidatorStore:
    """Tiny JSON-backed store mapping a URL to its cached HTTP validators."""

    def __init__(self, path: Path) -> None:
        self.path = path
        self._data: dict[str, dict[str, str]] = {}
        if path.exists():
            self._data = json.loads(path.read_text(encoding="utf-8"))

    def get(self, url: str) -> dict[str, str]:
        return self._data.get(url, {})

    def set(self, url: str, validators: dict[str, str]) -> None:
        self._data[url] = validators
        tmp = self.path.with_suffix(".part")
        tmp.write_text(json.dumps(self._data, indent=2), encoding="utf-8")
        tmp.replace(self.path)  # atomic swap so a crash never truncates the store


def fetch_if_changed(
    url: str,
    state_store: ValidatorStore,
    timeout: float = 60.0,
) -> tuple[bool, Optional[bytes]]:
    """
    Fetch ``url`` only if it changed since the last successful download.

    Returns
    -------
    (changed, payload):
        ``(False, None)`` when the server reports 304 Not Modified or the
        content hash is unchanged; ``(True, bytes)`` when new bytes arrived.
    """
    cached = state_store.get(url)
    headers: dict[str, str] = {}
    if etag := cached.get("etag"):
        headers["If-None-Match"] = etag
    if last_modified := cached.get("last_modified"):
        headers["If-Modified-Since"] = last_modified

    response = requests.get(url, headers=headers, timeout=timeout)

    # Server confirms nothing changed — cheapest possible outcome.
    if response.status_code == 304:
        logger.info("Unchanged (304): %s", url)
        return False, None

    response.raise_for_status()
    body = response.content

    new_etag = response.headers.get("ETag")
    new_last_modified = response.headers.get("Last-Modified")

    # Fallback path: server sent no validators, so compare a content digest.
    if not new_etag and not new_last_modified:
        digest = hashlib.sha256(body).hexdigest()
        if digest == cached.get("sha256"):
            logger.info("Unchanged (content hash match): %s", url)
            return False, None
        state_store.set(url, {"sha256": digest})
        logger.info("Changed (content hash differs): %s", url)
        return True, body

    # Standard path: persist whichever validators the server provided.
    validators = {"sha256": hashlib.sha256(body).hexdigest()}
    if new_etag:
        validators["etag"] = new_etag
    if new_last_modified:
        validators["last_modified"] = new_last_modified
    state_store.set(url, validators)

    logger.info("Changed (200), validators refreshed: %s", url)
    return True, body

Key Implementation Notes

• Send the ETag verbatim, quotes and all. An ETag value like "a1b2-63f0" includes the surrounding double quotes as part of the token. Round-trip it exactly as received into If-None-Match; stripping the quotes makes the server treat it as a mismatch and reply 200.

• Always store a content hash, even on the validator path. Some portals rotate ETags on unrelated deploys without changing the file. Persisting a SHA-256 alongside the validators lets a downstream step confirm the bytes really differ before triggering an expensive reload.

• The state store writes atomically. Writing to a .part file and calling Path.replace guarantees that a process killed mid-write leaves the previous validator map intact rather than a half-written JSON that fails to parse on the next run.

• If-Modified-Since only helps when the format matches. The value must be an RFC 7231 HTTP-date (e.g. Wed, 21 Oct 2026 07:28:00 GMT). Echo back the exact Last-Modified string the server sent rather than reformatting a parsed datetime, which risks locale or timezone drift.

• raise_for_status runs after the 304 check, not before. A 304 is not an error; short-circuiting on it first keeps the empty body from being mistaken for a failed download.

• Content hashing still transfers bytes. The fallback saves downstream parsing and loading, not bandwidth. When a portal supports HEAD, issue a conditional HEAD first to probe validators before committing to a full GET.

Troubleshooting Conditional GET Failures

Integration Note

fetch_if_changed is the front gate of the download stage described in Automating Government Portal Downloads: call it before schema validation so unchanged datasets never enter the parse-and-load path at all. The (changed, payload) return value maps directly onto the watermark and delta logic in scheduling and incremental spatial loads, where a changed=False result lets the orchestrator mark the task successful without touching the target table. Persist the ValidatorStore JSON on the same durable volume your DAG or flow uses for run metadata so validators survive worker restarts.

Related

• Automating Government Portal Downloads — parent guide covering portal crawling, authentication, and download orchestration
• Scheduling and Incremental Spatial Loads — turn change-detection results into watermark-driven incremental table loads
• Mastering Geospatial Data Ingestion in Python — top-level reference for ingestion architecture and pipeline observability