This guide is part of Orchestrating Spatial ETL Pipelines.

Storing Spatial Data in Cloud Object Storage

The Problem: A Bucket Is Not a Dataset

Object storage is where spatial pipelines put almost everything, and it is unusually unforgiving of layout mistakes because nothing enforces a layout at all. A bucket accepts any key, so the structure exists only in the code that writes it and in the memory of whoever wrote that code.

The failures that follow are predictable. A dataset partitioned by a fine spatial index accumulates hundreds of thousands of small objects, and listing it becomes slower than reading it. A dataset partitioned by nothing forces every query to scan a terabyte to answer a question about one city. Files written per batch rather than per partition produce thousands of 4 MB objects whose request costs exceed their storage costs. And a layout that nobody wrote down means the first question every new consumer asks is “how is this organised”, which is a question that should have an answer in a catalogue rather than in a colleague.

Getting it right is mostly about matching three things to each other: how the data is written, how it is read, and how it expires.

Prerequisites and Environment

pip install "geopandas>=1.0" "pyarrow>=14" "s3fs>=2024.6" "boto3>=1.34"

pyarrow handles the partitioned dataset semantics; s3fs provides the filesystem abstraction that lets the same code address local paths and object storage. Confirm the Arrow build includes the dataset module and the S3 filesystem, since minimal wheels sometimes omit the latter:

import pyarrow.dataset as ds
import pyarrow.fs as fs

assert hasattr(ds, "write_dataset")
assert hasattr(fs, "S3FileSystem"), "Arrow built without S3 support — install pyarrow with S3 enabled"

Version and Compatibility Matrix

Choosing the Partition Keys

The partition keys are the only part of the layout that queries can prune on, and they should come from the read pattern rather than from the write pattern.

Almost always: a coarse region and a time period. Region can be a country, an administrative unit, a UTM zone or a large tile — anything with tens to hundreds of values, not tens of thousands. Time is usually month for analytical data and day for operational data.

Rarely: a category with low cardinality. Product type, sensor, or source agency, where a query genuinely filters on it and there are only a handful of values.

Never: a feature identifier, a fine tile index, or a timestamp. Each produces a partition per row or close to it, and the resulting object count makes every operation on the dataset slow.

The test is simple: a partition key should reduce a typical query’s scan by at least an order of magnitude, and should not multiply the object count by more than about a hundred. Keys that fail either test are better expressed as a sorted column with row-group statistics, which prunes within files rather than between them — the mechanism used in reading GeoParquet from S3 with pyarrow filters.

Object Size and the Small-File Problem

Writing one file per batch is the most common cause of a lake that becomes unusable. A pipeline running hourly against forty regions produces nearly 350 000 objects a year, most of them a few megabytes, and every query pays a request per object.

import pyarrow.dataset as ds


def write_partitioned(table, root: str, filesystem, partitioning: list[str]) -> None:
    """Write with a target file size so batches do not become the file granularity."""
    ds.write_dataset(
        table,
        base_dir=root,
        filesystem=filesystem,
        format="parquet",
        partitioning=partitioning,
        partitioning_flavor="hive",
        existing_data_behavior="delete_matching",  # replace the partition, never append
        max_rows_per_file=2_000_000,
        min_rows_per_group=100_000,
        max_rows_per_group=500_000,
    )

existing_data_behavior="delete_matching" is what makes the write idempotent: re-running a partition replaces it rather than adding a second copy, which is the object-storage equivalent of the transactional upsert described in writing idempotent spatial ETL tasks in Airflow.

Where the natural write granularity is genuinely small — hourly increments, per-tile outputs — accept small files in a landing area and run a periodic compaction job that rewrites them into partition-sized objects. Compaction is boring, it is easy to make idempotent, and it is the difference between a lake that gets faster as it grows and one that gets slower.

Storage Classes and Lifecycle

Spatial data has an unusually sharp access-decay curve: this week’s imagery is read constantly, last year’s is read by an audit once. That shape is exactly what storage classes exist for, and applying them is a lifecycle rule rather than a code change.

A workable default for a raster archive moves objects to infrequent-access after 30 days and to an archive tier after 180, with the derived products — the tiles and the analysis-ready mosaics — staying in standard storage indefinitely because they are what gets read. The raw scenes are the bulk of the bytes and the smallest share of the reads.

Two cautions apply. Archive tiers have retrieval latency measured in hours and a per-object retrieval charge, so a pipeline that might need to reprocess from raw should keep a reprocessing window in a warm tier. And lifecycle transitions have a minimum billable duration; moving objects that will be deleted in a week costs more than leaving them.

Failure-Mode Reference

Production Integration Notes

The layout should be a function, not a convention: one place that builds an object key from dataset, region, period and version, used by every writer and every reader. A convention documented in a wiki drifts within a quarter; a function cannot.

Partitioned writes map cleanly onto partitioned orchestration. A task that materialises one partition writes exactly one partition prefix, which makes the write idempotent, the retry cheap and the lineage obvious — the same alignment described in managing spatial assets with Dagster. Where the orchestrator has a notion of partitions, matching the storage partitions to it removes an entire class of bookkeeping.

Finally, publish a dataset-level metadata document alongside the data: schema, CRS, partitioning scheme, update cadence, owner and retention. It is a few dozen lines of JSON, it answers the questions every new consumer asks, and it is the difference between a bucket and a dataset.

Zones: Landing, Curated and Served

A single prefix holding everything is the layout every lake starts with and none survives. Three zones, each with different rules, keep the distinctions that matter visible in the path.

Landing holds data exactly as it arrived: the downloaded archive, the raw scene, the API response. Nothing here is cleaned, and nothing here is queried by a consumer. Its retention is short, its schema is whatever the source produced, and its value is entirely in reproducibility — a transform can be re-run against the original bytes rather than against someone’s recollection of them.

Curated holds the cleaned, validated, conformed output of the pipeline: one CRS, one schema, quality-gated. This is the zone that partitioning and compaction apply to, and it is the one that most deserves versioning, because a change in the cleaning rules produces genuinely different data.

Served holds whatever consumers read directly — aggregates, tiles, extracts shaped for a particular application. It is usually much smaller than curated and is regenerated rather than edited.

Keeping the three separate is what makes retention answerable. Landing expires in weeks, curated persists, served is disposable and rebuildable. Mixing them produces a bucket where nobody can safely delete anything, which is how a storage bill becomes a quarterly conversation.

Idempotent Writes Without Transactions

Object storage has no transactions, so an interrupted write leaves partial state that a reader cannot distinguish from complete state. Three patterns cover the cases that arise in spatial pipelines.

Write-then-promote. Write the partition under a staging prefix, verify it, then copy or move it into place. On most object stores a server-side copy is fast and atomic per object, and the reader never observes a half-written partition. This is the safest and costs one extra copy.

Partition replacement. Write directly into the partition prefix with delete_matching semantics, which removes the old objects and writes the new ones. There is a brief window where the partition is empty, which is acceptable for datasets read by scheduled jobs and not for those read continuously.

Manifest-based publication. Write new objects under unique names and publish a manifest listing the current set. Readers consult the manifest rather than listing the prefix, so the switchover is a single small write. This is what table formats do internally, and implementing a minimal version is reasonable when the alternative is adopting one.

The pattern to avoid is writing directly into a served prefix with append semantics, which is simultaneously non-atomic and non-idempotent — a failed run leaves both partial data and duplicates.

Access Patterns Shape the Layout

Two consumers of the same dataset can want opposite layouts, and the resolution is usually to serve both from one curated copy rather than to compromise.

Analytical scans filter on region and time, read a subset of columns and touch many rows. They want large files, columnar storage, coarse partitions and row-group statistics — the layout described above.

Point lookups ask for one feature by identifier. They want an index, which object storage does not provide. Serving these from a lake means either accepting a full scan or maintaining a lookup structure elsewhere, which is exactly the case for the relational serving layer discussed in PostGIS vs DuckDB spatial for analytical loads.

Tile reads want one object per tile, small and directly addressable, which is the opposite of the large-file guidance. That is not a contradiction: tiles are a served artefact with their own layout rules, derived from the curated data rather than competing with it.

Writing the access patterns down next to the dataset definition — who reads it, how, and how often — makes the layout decisions defensible and makes it obvious when a new consumer needs a new served artefact rather than a change to the shared one.

Cost Attribution That Survives Scrutiny

A storage bill for a spatial lake is dominated by three lines: bytes stored, requests issued and bytes transferred out. Attributing them to datasets rather than to a single bucket total is what turns a cost conversation into a decision.

Prefix-level reporting is usually available directly from the provider and is free. Tagging objects at write time with the dataset name and zone gives finer attribution and survives reorganisation. Either way, the useful figure is cost per dataset per month, next to the read volume for that dataset — the ratio between them identifies the archives nobody reads and the small datasets that generate enormous request counts.

The single most common surprise is request cost from small objects, which is invisible in a bytes-stored view and can exceed storage cost by an order of magnitude for a badly compacted dataset. The second is egress from cross-region reads that nobody intended, usually because a consumer’s compute moved and the data did not.

Encryption, Access and the Spatial Angle

Most access-control guidance for object storage applies unchanged to spatial data, with two additions worth naming.

The first is that geometry can be sensitive when attributes are not. A dataset of anonymised incident records is disclosive if each record carries a precise coordinate, because a location at metre resolution frequently identifies a dwelling. Where that applies, the served zone should carry a generalised geometry — snapped to a grid, aggregated to an area — while the precise version stays in curated with tighter access. Doing that generalisation at publication rather than at ingestion keeps the precise data available for legitimate internal use.

The second is that licence terms travel with spatial data more often than with other kinds. Many public sources permit derived products but restrict redistribution of the source itself. A lake that mixes them in one prefix makes an accidental redistribution easy; separating by licence, or at least tagging objects with it, makes the boundary checkable.

Both are metadata problems rather than technical ones, which is another argument for the dataset-level document: licence, sensitivity and access owner belong there, next to schema and CRS, where a reviewer will actually see them.

Related

• Partitioning GeoParquet by Region and Date
• Reading GeoParquet from S3 with pyarrow Filters
• Lifecycle and Cost Controls for Raster Archives
• Choosing a Vector Output Format for Spatial Pipelines
• PostGIS vs DuckDB Spatial for Analytical Loads