This guide is part of Storing Spatial Data in Cloud Object Storage, within the broader Orchestrating Spatial ETL Pipelines reference.

Lifecycle and Cost Controls for Raster Archives

Imagery archives grow monotonically and are read decreasingly. That combination is what storage tiers are for, and applying them carelessly is how a cost-saving exercise makes a pipeline slower and more expensive at once.

Why Raster Archives Get Expensive

  • Raw scenes dominate the bytes. A year of daily acquisitions over a country is measured in terabytes, most of it never read after processing.
  • Derived products dominate the reads. Tiles and mosaics are small and constantly requested, which is the opposite profile.
  • Retrieval charges surprise. Archive tiers bill per retrieval as well as per byte, and a reprocessing job can retrieve everything at once.
  • Small objects defeat tiering. Minimum billable sizes mean archiving a million 200 KB files can cost more than leaving them in standard storage.

Version and Environment Compatibility

Component Version Note
boto3 >=1.34 Lifecycle configuration and storage-class inspection
pandas >=2.2 Aggregating inventory reports
Provider inventory — S3 Inventory, GCS storage insights or equivalent 
pip install "boto3>=1.34" "pandas>=2.2" "pyarrow>=14"

Tier by Role, Not by Age Alone

Bytes and reads are distributed inversely Two bars compare share of stored bytes against share of read requests for three categories. Raw scenes are eighty-four percent of bytes and three percent of reads. Analysis-ready mosaics are thirteen percent of bytes and twenty-two percent of reads. Map tiles are three percent of bytes and seventy-five percent of reads. The inversion is what makes role-based tiering effective. share of stored bytes raw scenes · 84% mosaics 13% share of read requests mosaics 22% map tiles · 75% raw scenes: 3% of reads · archive after the reprocessing window tiles and mosaics: 97% of reads · standard storage, indefinitely

The Lifecycle Configuration Recipe

from __future__ import annotations

import logging

import boto3

logger = logging.getLogger(__name__)

MIN_BILLABLE_BYTES = 128 * 1024   # objects below this cost more archived than stored


def raster_lifecycle_rules(
    raw_prefix: str = "landing/scenes/",
    derived_prefix: str = "served/tiles/",
    warm_days: int = 45,
    archive_days: int = 180,
    expire_landing_days: int = 30,
) -> list[dict]:
    """Rules that tier raw inputs aggressively and leave derived products alone."""
    return [
        {
            "ID": "raw-scenes-tiering",
            "Filter": {"And": {
                "Prefix": raw_prefix,
                "ObjectSizeGreaterThan": MIN_BILLABLE_BYTES,
            }},
            "Status": "Enabled",
            "Transitions": [
                {"Days": warm_days, "StorageClass": "STANDARD_IA"},
                {"Days": archive_days, "StorageClass": "GLACIER_IR"},
            ],
        },
        {
            # Intermediate staging is rebuildable; delete rather than tier it.
            "ID": "staging-expiry",
            "Filter": {"Prefix": "staging/"},
            "Status": "Enabled",
            "Expiration": {"Days": expire_landing_days},
        },
        {
            "ID": "abort-incomplete-uploads",
            "Filter": {"Prefix": ""},
            "Status": "Enabled",
            "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7},
        },
    ]


def apply_lifecycle(bucket: str, rules: list[dict], dry_run: bool = True) -> None:
    client = boto3.client("s3")
    if dry_run:
        logger.info("would apply %d lifecycle rules to %s: %s",
                    len(rules), bucket, [r["ID"] for r in rules])
        return
    client.put_bucket_lifecycle_configuration(
        Bucket=bucket, LifecycleConfiguration={"Rules": rules}
    )
    logger.warning("applied %d lifecycle rules to %s", len(rules), bucket)

Note what is absent: no rule touches the derived prefix. Tiles and mosaics are the read path, and moving them to a slower class trades a large latency cost for a small storage saving.

Key Implementation Notes

  • ObjectSizeGreaterThan guards the small-object trap. Below the minimum billable size, archiving increases cost — the object is billed as if it were larger, and a retrieval charge is added on top.
  • Incomplete multipart uploads are aborted. Failed large-file uploads leave parts that are billed indefinitely and invisible in a normal object listing; this rule is pure saving with no downside.
  • Staging is expired, not tiered. Rebuildable intermediates should be deleted; paying to archive something you would regenerate anyway is the most common wasted line.
  • dry_run defaults to true. A lifecycle rule applies to existing objects immediately and is not trivially reversible — a transition to archive cannot be undone without a retrieval charge.
  • Transitions are ordered by days ascending. Providers reject configurations where a later transition has a shorter age, and the error message is not always clear about which rule is at fault.
  • The raw prefix is separate from the derived one because the two need opposite policies, which only works if the layout separated them in the first place.
Sizing the warm reprocessing window A timeline of raw scene age across three storage classes. The first forty-five days are in standard storage where reprocessing starts immediately. From forty-five to a hundred and eighty days the scenes are in infrequent access, still immediately readable at a per-gigabyte retrieval fee. Beyond that they are archived, where reprocessing must wait hours for restoration. raw scene age → storage class → cost of reprocessing 0–45 d · standard 45–180 d · infrequent access 180 d+ · archive reprocess now — no retrieval, no wait reprocess now — per-GB retrieval fee restore first — hours, plus retrieval Set the first boundary from how far back a correction has ever actually needed to reach, not from how far back you imagine it might.

Measuring Before Tiering

Tiering decisions made from intuition are usually wrong in both directions: data assumed cold turns out to feed a weekly report, and data assumed hot has not been read in a year.

What to measure before writing a lifecycle rule Three questions a daily inventory report answers directly. Which prefixes hold the bytes identifies where tiering could matter. Which objects have not been read in ninety days identifies what is genuinely cold. How many objects fall below the minimum billable size identifies where archiving would increase rather than reduce cost. from one inventory report, before any rule is written where are the bytes? — group size by prefix; usually one prefix dominates what is genuinely cold? — last access, not last modified how many small objects? — below the minimum size, archiving costs more Run it again a month after the rule lands: a retrieval-charge spike is how a wrong rule announces itself.

Provider inventory reports make this measurable cheaply. A daily inventory listing every object with its size, storage class and last-access time, read into a dataframe, answers the only questions that matter: which prefixes hold the bytes, which have not been read in ninety days, and how many objects fall below the minimum billable size.

Run that analysis once before writing any lifecycle rule, and again a month after. The second run is what catches a rule that archived something the pipeline needed — visible as a spike in retrieval charges rather than as a failure, since the reads still succeed.

Troubleshooting Cost Controls

Symptom Likely cause Fix
Bill rose after archiving Many objects below the minimum billable size Add an ObjectSizeGreaterThan filter
Reprocessing takes hours Raw scenes archived inside the reprocessing window Extend the warm window to cover it
Retrieval charges every month A scheduled job reads archived data Move that data back, or change the job
Storage grows despite expiry rules Incomplete multipart uploads accumulating Add the abort rule
Cannot attribute cost Everything under one prefix Separate by zone and dataset; tag at write time
Deleted data still billed Minimum storage duration in the tier Do not archive short-lived data at all

Integration Note

Lifecycle rules belong in the same infrastructure definition as the buckets, reviewed like code rather than applied by hand — a rule set that lives only in a console is one nobody can audit. Where the pipeline itself needs archived data, request restoration explicitly as a task rather than letting a read block for hours, and record the retrieval in the run metrics described in monitoring and observability for spatial pipelines so that the cost is attributable to the job that caused it.

Deleting Raw Data, and When Not To

The largest saving available to any imagery archive is deleting raw scenes entirely, and it is the one decision worth making slowly.

Deletion is defensible when the source is durably republishable — a public satellite archive that will still serve the same scene in five years — and when the pipeline records enough provenance to fetch it again: the scene identifier, the source URI and the retrieval parameters. In that case the archive is a cache, and caches can be evicted.

It is not defensible when the source is a one-off delivery, a partner feed with no retention guarantee, or a portal whose publication history is overwritten in place. There the raw copy is the only evidence of what was received, and deleting it makes every later question about a discrepancy unanswerable.

The middle path, which suits most pipelines, is to keep a sample rather than everything: every scene for the current reprocessing window, and thereafter one scene per month per region, retained indefinitely. The sample costs almost nothing, and it is enough to demonstrate what the source looked like at a given time — which is what an audit actually asks.