This guide is part of Bulk Downloading Satellite Imagery, within the broader Mastering Geospatial Data Ingestion in Python reference.

Resuming Interrupted Scene Downloads with Range Requests

Scene files are large enough that a transfer failure at 80% is expensive and common. HTTP ranges make the retry proportional to what is missing rather than to the whole file — provided the client checks that the server actually honoured the request.

Why Restarting From Zero Is the Wrong Default

• The failure rate scales with duration. A 1.2 GB transfer over a shared link fails often enough that whole-file retries dominate a backfill’s runtime.
• Quota is consumed twice. Providers meter bytes transferred, not bytes kept, so a restart bills for the discarded prefix too.
• The retry window shrinks. Restarting a long transfer inside a task timeout can mean it never completes at all.
• Concurrency amplifies it. With eight parallel scenes, one restart per scene per hour is a permanent tax on throughput.

Version and Environment Compatibility

pip install "requests>=2.32" "rasterio>=1.3"

What the Server Is Telling You

The download_resumable Recipe

from __future__ import annotations

import hashlib
import logging
from pathlib import Path

import requests

logger = logging.getLogger(__name__)
CHUNK = 8 * 1024 * 1024


def download_resumable(
    url: str,
    destination: Path,
    session: requests.Session | None = None,
    validator: str | None = None,
    expected_sha256: str | None = None,
    read_timeout: int = 300,
) -> Path:
    """Download a large file, resuming from whatever is already on disk.

    `validator` is the ETag or Last-Modified captured on the first attempt; it is
    sent as If-Range so the server refuses a resume against a changed resource.
    """
    session = session or requests.Session()
    partial = destination.with_suffix(destination.suffix + ".part")
    offset = partial.stat().st_size if partial.exists() else 0

    headers = {}
    if offset:
        headers["Range"] = f"bytes={offset}-"
        if validator:
            headers["If-Range"] = validator

    with session.get(url, headers=headers, stream=True, timeout=(30, read_timeout)) as response:
        if response.status_code == 416:
            logger.warning("%s: offset %d not satisfiable — restarting", destination.name, offset)
            partial.unlink(missing_ok=True)
            return download_resumable(url, destination, session, None, expected_sha256, read_timeout)

        response.raise_for_status()

        if offset and response.status_code == 200:
            # The server ignored the range: everything from byte zero is coming.
            logger.warning("%s: server ignored Range — truncating and restarting", destination.name)
            partial.unlink(missing_ok=True)
            offset = 0

        if offset and response.status_code == 206:
            content_range = response.headers.get("Content-Range", "")
            if not content_range.startswith(f"bytes {offset}-"):
                raise ValueError(f"unexpected Content-Range {content_range!r} for offset {offset}")

        mode = "ab" if offset else "wb"
        with partial.open(mode) as handle:
            for chunk in response.iter_content(chunk_size=CHUNK):
                handle.write(chunk)

        total = response.headers.get("Content-Range", "").split("/")[-1]
        expected_size = int(total) if total.isdigit() else None

    actual = partial.stat().st_size
    if expected_size and actual != expected_size:
        raise ValueError(f"{destination.name}: got {actual} bytes, expected {expected_size}")
    if expected_sha256 and _sha256(partial) != expected_sha256:
        partial.unlink(missing_ok=True)
        raise ValueError(f"{destination.name}: checksum mismatch — partial discarded")

    partial.replace(destination)
    logger.info("%s complete (%d bytes, resumed from %d)", destination.name, actual, offset)
    return destination


def _sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for block in iter(lambda: handle.read(CHUNK), b""):
            digest.update(block)
    return digest.hexdigest()

Key Implementation Notes

• The 200-after-Range check is the critical one. Without it, appending a full response to a partial file produces a corrupt scene that opens, renders and contains a duplicated prefix.
• Content-Range is verified, not trusted. A proxy that rewrites ranges will return 206 with a different start, and the assertion catches it before any bytes are appended.
• If-Range guards against a changed resource. If the ETag no longer matches, the server sends 200 and the client restarts — which is correct, because resuming across a change would splice two different files.
• The .part suffix keeps incomplete files unusable. No downstream step can mistake a partial scene for a complete one, and the rename is the atomic publication.
• Checksum failure deletes the partial. Keeping a file that failed verification guarantees the next attempt resumes onto corruption.
• 416 restarts rather than raising. It usually means the partial is stale, and a restart is both correct and cheap relative to a failed backfill.

Verifying the Reassembled File

Byte count is necessary and not sufficient. Three further checks are worth the cost on scene downloads.

The published checksum, where the provider offers one, is definitive and cheap relative to the transfer. Compute it while writing rather than in a second pass over the file.

A structural open — rasterio.open on a GeoTIFF, a ZIP directory read on an archive — catches a file that is the right length and internally broken, which happens when a proxy substitutes an error page mid-stream.

A band statistics read on one overview level is the strongest cheap check for imagery: it touches real pixel data rather than only the header, and a file whose header parses but whose data is truncated fails here.

Where none of the three is available, at minimum record the byte count and the ETag with the file, so a later inconsistency can be traced to a specific transfer.

Troubleshooting Resumed Downloads

Integration Note

Resumption only helps if the partial survives the retry, which means writing it somewhere the next attempt can see — a mounted volume or object storage, not a container’s ephemeral filesystem. In an orchestrated pipeline the download task should therefore be given a stable working directory keyed by scene id, and the retry policy set to reuse it, following the idempotency reasoning in writing idempotent spatial ETL tasks in Airflow.

Parallel Ranges for a Single Large File

Resumption fetches one remaining range at a time, which is right for recovery and leaves throughput on the table for a first download. Where the provider allows it, splitting one file into several concurrent ranges can saturate a link that a single connection cannot.

The mechanics are straightforward: ask for the total length with a HEAD, divide it into equal parts, request each with its own Range header, and write each into its own offset of a pre-allocated file. Four to eight parts is usually where the gain flattens.

Two cautions keep it from backfiring. Providers frequently count each range as a separate request against a quota, so a scene split eight ways costs eight requests rather than one. And concurrent ranges against the same object can trip abuse heuristics on public endpoints, which is a reason to reserve the technique for providers whose terms explicitly permit it.

For most spatial pipelines the more valuable parallelism is across scenes rather than within one, because it needs no coordination and each unit remains independently retryable.

Related

• Bulk Downloading Satellite Imagery
• Automating USGS EarthExplorer Bulk Downloads with requests
• Detecting Dataset Changes with ETag and Last-Modified
• Lifecycle and Cost Controls for Raster Archives