This guide is part of Building Airflow DAGs for Spatial ETL, a chapter of the broader Orchestrating Spatial ETL Pipelines reference.

Passing GeoDataFrames Between Airflow Tasks

The Problem: XCom Is a Message Bus, Not a Data Channel

The instinct carried over from a notebook is to have one function return a GeoDataFrame and the next consume it. Translate that into Airflow’s TaskFlow API and every return gdf silently routes the entire frame through XCom — a mechanism designed for small control-plane values like counts and keys, backed by the scheduler’s metadata database.

• XCom serializes into the metadata database. Every returned GeoDataFrame is pickled or JSON-encoded and written as a row the scheduler reads on every parse. A few thousand geometries turn a lightweight coordination table into a multi-megabyte blob store and slow the whole scheduler.
• Tasks do not share memory. Airflow tasks are separate processes, often on separate workers. An in-memory GeoDataFrame exists only in the producing process; the consuming task on another host has no way to reach it.
• Worker restarts erase in-flight state. Autoscaling, spot-instance reclaim, and OOM kills recycle workers between tasks. Anything not persisted to a shared store vanishes, and the downstream task fails to deserialize a reference to memory that no longer exists.
• Size limits fail late and loudly. The frame that fit in testing overflows the XCom backend in production when a busy day returns ten times the geometries, and the DAG that passed for months starts erroring at the task boundary.

Version and Environment Compatibility

pip install "apache-airflow==2.9.3" "apache-airflow-providers-amazon>=8.0" \
  "geopandas>=1.0" "pyarrow>=15.0"

The pattern is a two-task pair: a producer serializes the GeoDataFrame to GeoParquet and returns the object key; a consumer receives the key and reads the frame back. XCom carries only the string.

Recipe: A Producer/Consumer Task Pair Over GeoParquet

The two tasks below implement the handoff end to end. write_features_to_s3 serializes a GeoDataFrame to a GeoParquet object and returns its key; read_features_from_s3 takes that key and reconstructs the frame. The S3 key is the only thing that crosses the task boundary, so the pair works identically whether the two tasks land on the same worker or on hosts an autoscaler spun up seconds apart.

from __future__ import annotations

import logging
from datetime import datetime

import geopandas as gpd
from airflow.decorators import task
from airflow.providers.amazon.aws.hooks.s3 import S3Hook

logger = logging.getLogger(__name__)

_BUCKET = "etl-spatial"


@task
def write_features_to_s3(
    gdf: gpd.GeoDataFrame,
    tile_id: str,
    logical_date: datetime,
    aws_conn_id: str = "aws_default",
) -> str:
    """Serialize a GeoDataFrame to GeoParquet in S3 and return only its key.

    The return value is a short string, so XCom stores a reference rather than
    the geometries themselves. Keying on ``logical_date`` + ``tile_id`` makes the
    write deterministic, so a retry overwrites the same object.
    """
    run_key = logical_date.strftime("%Y%m%d")
    object_key = f"handoff/{run_key}/{tile_id}.parquet"

    # Confirm the connection resolves before we commit to the write path.
    S3Hook(aws_conn_id=aws_conn_id).get_conn()
    gdf.to_parquet(f"s3://{_BUCKET}/{object_key}")  # GeoParquet preserves CRS + geometry
    logger.info("Wrote %d features to s3://%s/%s", len(gdf), _BUCKET, object_key)
    return object_key  # <-- the ONLY thing that travels through XCom


@task
def read_features_from_s3(
    object_key: str,
    aws_conn_id: str = "aws_default",
) -> gpd.GeoDataFrame:
    """Read a GeoParquet object back into a GeoDataFrame from its S3 key.

    Falls back with a clear error if the object is missing, which surfaces a
    lost or mistyped handoff key instead of a cryptic Arrow read failure.
    """
    hook = S3Hook(aws_conn_id=aws_conn_id)
    if not hook.check_for_key(object_key, bucket_name=_BUCKET):
        raise FileNotFoundError(
            f"Handoff object s3://{_BUCKET}/{object_key} is missing — "
            "the producer task did not persist it or the key was not passed through XCom."
        )
    gdf = gpd.read_parquet(f"s3://{_BUCKET}/{object_key}")
    logger.info("Read %d features from s3://%s/%s", len(gdf), _BUCKET, object_key)
    return gdf  # downstream: re-serialize and return the next key, never the frame

In a TaskFlow DAG the wiring is read_features_from_s3(write_features_to_s3(gdf, tile_id)) — Airflow records the producer’s returned key as an XCom reference and injects it as the consumer’s object_key. Note the consumer returns a GeoDataFrame here only to keep the recipe self-contained; in a real DAG it would itself write GeoParquet and return the next key so no frame ever reaches XCom.

Key Implementation Notes

• The producer returns a str, the consumer accepts a str. This is the whole discipline: the type crossing the task boundary is a key, never a frame. If a signature ever returns gpd.GeoDataFrame, the frame is going through XCom — treat that as a bug.
• GeoParquet is the right serialization, not pickle or CSV. It preserves the CRS and the geometry column natively, is columnar for fast partial reads, and is portable across the ingestion and cleaning references. A pickled GeoDataFrame is version-fragile and still too large for XCom.
• Keys are deterministic on logical_date and tile_id. A retried producer overwrites the same object rather than minting a new one, so the handoff stays consistent under the retry semantics detailed in writing idempotent spatial ETL tasks in Airflow.
• A custom XCom backend automates the pattern but does not replace it. Airflow lets you register an xcom_backend that transparently offloads large values to S3 and stores only a pointer. It is convenient, but the semantics are identical to this explicit pair — the data still lives in object storage and only a reference is in the database.
• The missing-object guard turns a silent gap into a loud failure. check_for_key converts a dropped or mistyped handoff key into an explicit FileNotFoundError at the consumer, instead of an opaque Arrow error three frames deep.

Troubleshooting Common Handoff Failures

Integration Note

This producer/consumer pair is the connective tissue between every stage in building Airflow DAGs for spatial ETL: Extract returns a raw key, Transform reads it and returns a clean key, and Load reads that clean key before its final write. Because each task hands the next a URI rather than a frame, the whole DAG survives worker churn and stays within XCom limits no matter how many geometries a busy day produces. The final Load stage consumes the last key and applies the transactional upsert from writing idempotent spatial ETL tasks in Airflow, closing the loop from durable handoff to exactly-once load.

Related

• Building Airflow DAGs for Spatial ETL — the DAG whose every task boundary uses this GeoParquet-URI handoff
• Writing Idempotent Spatial ETL Tasks in Airflow — the Load stage that consumes the final handoff key and upserts safely
• Orchestrating Spatial ETL Pipelines — the durable-handoff principle behind passing references instead of data