This guide is part of Orchestrating Spatial ETL Pipelines.

Monitoring and Observability for Spatial Pipelines

The Problem: Success Is Not the Same as Correct

Standard pipeline monitoring answers one question: did the tasks run. That question is necessary and nowhere near sufficient for spatial work, because the characteristic spatial failure is a run that succeeds and produces the wrong thing.

A source that stopped publishing keeps returning 304 Not Modified, so the conditional download skips, the transform has nothing to do and the load commits zero rows — three green tasks and a serving table that is a week stale. A bounding box filter written against a layer that changed CRS matches nothing, so the batch is empty and every downstream task handles the empty case gracefully. A reprojection introduced by a refactor moves every feature two metres and no assertion notices, because the geometry is still valid and the row count is unchanged.

None of these produce a red run. All of them produce a wrong product. The instrumentation that catches them measures the data, not the execution — and the two need different metrics, different thresholds and different responses.

The Spatial Metric Set

Five families cover almost everything worth measuring, and each maps to a failure that has bitten every long-running spatial pipeline.

Volume. Rows in, rows out, rows quarantined, per source and per run. The ratio between them is more informative than any absolute: a transform that normally emits 0.98 rows per input row and today emits 0.61 has dropped something.

Freshness. The age of the newest record in the serving table, and separately the age of the source data it came from. Two numbers, because a pipeline running perfectly against a stalled source is a different problem from a stalled pipeline.

Geometry health. Invalid count, empty count, geometry-type distribution, and the p99 vertex count. The last one is an early warning for memory problems: a source that starts emitting million-vertex polygons will exhaust a worker long before it fails a validity check.

Coverage. Total area or pixel count, and the bounding box of the output. A coverage that shrinks by 30% overnight is either a genuine change worth knowing about or a partition that failed silently.

Cost. Bytes transferred, requests issued, wall-clock per unit of work. These are the numbers that make a capacity conversation concrete rather than anecdotal, and they are almost never collected until someone asks why the bill moved.

Emitting Metrics Without Coupling to a Vendor

The instrumentation should not tie the pipeline to whichever metrics system is current. A thin emitter with a stable interface keeps the call sites unchanged when the backend changes.

from __future__ import annotations

import logging
from dataclasses import dataclass, field
from typing import Any

logger = logging.getLogger(__name__)


@dataclass
class RunMetrics:
    """Collects one run's measurements, then fans them out to every sink."""
    run_id: str
    dataset: str
    values: dict[str, float] = field(default_factory=dict)
    labels: dict[str, str] = field(default_factory=dict)

    def record(self, name: str, value: float, **labels: str) -> None:
        key = name if not labels else f"{name}[{','.join(f'{k}={v}' for k, v in sorted(labels.items()))}]"
        self.values[key] = float(value)

    def emit(self, sinks: list[Any]) -> None:
        for sink in sinks:
            try:
                sink.write(self.dataset, self.run_id, self.values, self.labels)
            except Exception:  # noqa: BLE001 — telemetry must never fail the run
                logger.exception("metric sink %s failed; continuing", type(sink).__name__)

The except clause is deliberate and worth defending: a monitoring system that can fail a data pipeline has inverted the dependency. Telemetry is best-effort by construction; the run’s correctness must not depend on whether a metrics endpoint answered.

Freshness as the Primary Signal

Of everything above, freshness earns its place first. It is one number per dataset, it is cheap to compute, and it catches the entire class of silent failures because every one of them eventually manifests as data that stopped arriving.

from datetime import datetime, timedelta, timezone

import sqlalchemy


def output_freshness(engine: sqlalchemy.Engine, table: str,
                     timestamp_column: str = "updated_at") -> timedelta:
    """Age of the newest record in a serving table, in wall-clock terms."""
    query = sqlalchemy.text(f"SELECT max({timestamp_column}) FROM {table}")
    with engine.connect() as connection:
        newest = connection.execute(query).scalar_one_or_none()

    if newest is None:
        raise ValueError(f"{table} is empty — freshness is undefined, which is itself an alert")
    if newest.tzinfo is None:
        newest = newest.replace(tzinfo=timezone.utc)
    return datetime.now(timezone.utc) - newest

Alert on this rather than on task failure and a whole category of incident disappears from the “how did nobody notice” category. The threshold comes from the update cadence with headroom: a daily pipeline alerts at 36 hours, not at 24, so that one late run is not an incident and two are.

Cross-Cutting Concerns

Correlation. Every metric, log line and quarantined row should carry the same run identifier. Without it, joining “the batch was small” to “the source returned a 304” is manual work; with it, it is a filter. The identifier costs nothing and is the single highest-leverage convention in pipeline observability.

Cardinality. Labels multiply. A metric labelled by dataset and stage is fine; one labelled by tile identifier produces hundreds of thousands of series and will be rejected or will bankrupt the metrics store. Tile-level detail belongs in the run record, not in the metrics system.

Sampling and cost. Reading every geometry to compute a vertex distribution on every run is affordable; reading every pixel of every raster to compute a histogram is not. Compute expensive metrics on a schedule rather than per run, and say so in the metric name so nobody reads a weekly number as a daily one.

Validation and Quality Gates

Observability and validation overlap but are not the same thing. A gate blocks; a metric describes. The productive relationship is that every gate emits a metric whether or not it fires, so the distribution of values leading up to a failure is available afterwards — which is the difference between “the quarantine gate failed” and “the quarantine rate had tripled over four nights”.

That pairing is why the checks in validating spatial data quality in pipelines return counts rather than booleans, and why the severity table lives beside them.

Failure-Mode Reference

Production Integration Notes

Every orchestrator has a place for per-run metadata — Airflow’s task instance fields, Prefect’s artifacts, Dagster’s asset metadata — and using it costs one call at the end of each task. That is where an operator investigating one run will look, so per-run counts belong there regardless of what else is collected.

The metrics store is the other half, and its job is history. Emit the same numbers to both from the same object, as the emitter above does, and the two views stay consistent without anyone maintaining a mapping. Where the pipeline writes to object storage, a small JSON metrics document beside each output is a third sink worth having, because it survives independently of any monitoring system’s retention policy.

Dashboards People Actually Read

A dashboard with forty panels is a dashboard nobody opens. The ones that get used share a structure: one screen that answers “is everything fine right now”, and drill-downs behind it.

The top screen holds one row per dataset with four columns — freshness, last run outcome, row count against its normal band, and quarantine rate against its threshold. Four numbers, colour-coded, no charts. Someone glancing at it before a meeting can tell in three seconds whether anything needs attention, which is the only interaction that matters for a summary view.

Everything else belongs one click deeper: the time series for each metric, the per-source breakdown, the run history. Those are investigation tools, and investigations start from a specific question rather than from a wall of charts.

Two panels are worth adding that most teams omit. A coverage map — the bounding box or footprint of the most recent output drawn over the expected extent — makes a partial run obvious in a way no number does, because the human eye finds a missing region instantly. And a run duration histogram rather than a mean, because spatial runtimes are heavily skewed and the mean hides the tile that took forty minutes.

Tracing a Single Feature Through the Pipeline

Aggregate metrics tell you something is wrong; tracing tells you where. The spatial version of a distributed trace does not need a tracing system — it needs three identifiers carried consistently.

The run id ties every artefact of one execution together. The source id — the feature’s identifier in the upstream system — survives every transform and is what a data owner will quote when they report a problem. The partition key — tile, date, region — names the unit of work that produced it.

With those three on every row, every log line and every quarantine record, the question “what happened to parcel A17 last Tuesday” becomes a query rather than an expedition: find the run for that partition on that date, read its metrics, look for the source id in the quarantine table, and read the log lines carrying that run id. Without them, the same question requires reading logs.

The discipline this requires is small and easily lost. A transform that regenerates identifiers, a join that drops the source id, a task that logs without the run id — each breaks the chain at one point, and the break is invisible until someone needs the chain. Asserting that the identifier columns survive each stage is a cheap test and worth having.

Alert Routing and Ownership

An alert with no owner is a notification. Three properties turn it into something that gets handled.

A named owner per dataset, recorded next to the pipeline configuration rather than in someone’s memory. Spatial pipelines cross organisational boundaries — the imagery team, the reference-data team, the platform team — and the right responder for a source outage is rarely the right responder for a schema change.

A severity that matches the response. Paging severity for anything that requires action within hours; ticket severity for anything that can wait for a working day; digest for everything else. The mapping should be stated in the alert definition, not decided by whoever receives it.

A runbook link in the alert body. Not a wiki homepage — the specific page for that alert, with the three checks to run first. The value is highest at 03:00, which is precisely when nobody wants to search for it.

Where an alert fires repeatedly without action, that is information about the alert rather than about the responder. Either the threshold is wrong, the condition is not actionable, or the work it implies is not being prioritised — and all three are worth surfacing in a periodic review of which alerts fired and what happened next.

Retention and What It Costs

Metrics are small and logs are not. A pipeline emitting fifty metrics per run at daily cadence produces a few thousand points a year, which is nothing. The same pipeline emitting a structured log line per feature produces hundreds of millions, which is a bill.

The resolution to that is not to log less but to log at the right grain. Per-run and per-partition records are proportionate; per-feature records belong in the quarantine table where only failures land, or behind a debug flag that is off in production. Where per-feature detail is genuinely needed for an investigation, enabling it for one partition for one run is far cheaper than carrying it always.

Set retention deliberately: metrics indefinitely, run records for a year, logs for thirty to ninety days, quarantined rows for the replay window described in quarantining invalid features to a dead-letter table. Those four numbers written down are also the answer to the audit question about how far back the pipeline can be reconstructed, which someone will eventually ask.

Measuring the Pipeline Against Its Consumers

The metrics above describe what the pipeline did. One more class describes whether that was enough, and it comes from the consumers rather than from the code.

Query patterns show which parts of the output are actually read. A tiled product where 90% of reads hit 4% of tiles is telling you where to spend effort on freshness and where a slower cadence would go unnoticed. Most object stores can emit access logs cheaply, and a weekly aggregate is enough.

Downstream failures attributable to the data are the most direct signal of all. A dashboard that broke, a model that retrained on a short batch, an export that a partner rejected — each is a quality escape, and counting them is the only honest measure of whether the gates are set correctly. A quarter with zero escapes and a rising quarantine rate suggests gates that are too tight; a quarter with several escapes and quiet gates suggests the opposite.

Time to detection is worth tracking explicitly: the gap between a problem entering the data and someone noticing. Every improvement to instrumentation should shorten it, and if it does not, the instrumentation was measuring something nobody needed.

Related

• Emitting Metrics for Spatial Record Counts
• Structured Logging for Geospatial ETL Jobs
• Alerting on Spatial Data Freshness SLAs
• Validating Spatial Data Quality in Pipelines
• Scheduling & Incremental Spatial Loads