This guide is part of Spatial Deduplication & Topology Simplification, which sits within the Automated Vector & Raster Cleaning Workflows reference for building reliable spatial ETL pipelines.

Deduplicating Overlapping Polygons with Spatial Joins

The Problem: Duplicate Footprints Are Not Byte-Identical

When two pipelines ingest the same building footprints, parcels, or field boundaries from different vendors, the “duplicate” polygons almost never share identical coordinates — so a hash-based deduplication step passes them straight through as distinct features.

Vendor A digitized the parcel at one scale, vendor B at another; a reprojection nudged every vertex a few centimetres; one source has an extra vertex on a straight run the other collapsed. The polygons cover the same ground but differ byte-for-byte, so exact-match deduplication is blind to them. The reliable signal is area agreement: two polygons that overlap almost completely are the same feature. Intersection-over-union (IoU) turns that intuition into a single number you can threshold.

  • Double-counted areas: summing an attribute (population, yield, footprint area) over a layer with duplicate polygons overstates every total.
  • sjoin fan-out: a downstream join against a duplicated reference layer multiplies matched rows, silently inflating result cardinality.
  • Conflicting attributes: two copies of one parcel carry different owner or class values, and whichever is written last wins by accident rather than by rule.
  • Naive pairwise comparison is O(n²): comparing every polygon against every other is infeasible past a few thousand features, which is why the spatial index has to prune candidates first.

Version and Environment Compatibility

GeoPandas Shapely Recommended dedup path Caveat
≥ 1.0 ≥ 2.0 gpd.sjoin + vectorized shapely.intersection/union Stable predicate-based join; use this going forward
0.13–0.14 2.0.x gpd.sjoin + vectorized IoU sjoin returns index_right; confirm column name after reset_index
0.10–0.12 1.8.x sjoin + per-geometry area No vectorized array API; IoU loop is slow on large frames

Install the recommended stack:

pip install "geopandas>=1.0.0" "shapely>=2.0.0" numpy

How IoU Decides a Duplicate

Intersection-over-union threshold for polygon deduplication Two rectangles overlap. The overlapping region is shaded as the intersection; the combined outline is the union. IoU is the intersection area divided by the union area. When IoU is greater than or equal to the threshold, the two polygons are treated as one duplicate group and a single survivor is kept. polygon A polygon B ∩ IoU = area(A ∩ B) / area(A ∪ B) IoU ≥ threshold ↓ duplicate group ↓ keep 1 survivor

IoU ranges from 0 (disjoint) to 1 (identical). A threshold of 0.9 treats only near-perfect matches as duplicates and leaves partially overlapping neighbours intact; lowering it toward 0.5 merges looser matches at the risk of collapsing genuinely distinct-but-adjacent polygons. The spatial self-join does the pruning — it returns only pairs whose bounding boxes intersect — and IoU is computed just for those candidates, keeping the whole operation near-linear in practice.

Recipe: dedupe_overlapping_polygons

The function builds a GeoPandas spatial self-join, scores each candidate pair with the vectorized Shapely array API, groups duplicates into connected components, and keeps one survivor per group by a configurable rule.

import logging

import geopandas as gpd
import numpy as np
import shapely

logger = logging.getLogger(__name__)


def dedupe_overlapping_polygons(
    gdf: gpd.GeoDataFrame,
    iou_threshold: float = 0.9,
    priority_col: str | None = None,
    keep: str = "first",
) -> gpd.GeoDataFrame:
    """
    Collapse duplicate / overlapping polygons detected by a spatial self-join.

    Two polygons are treated as the same feature when their intersection-over-
    union (IoU) meets iou_threshold. Candidate pairs come from a GeoPandas
    spatial self-join (predicate='intersects'), and IoU is computed with the
    Shapely 2.0 vectorized array API — no per-row .apply().

    Parameters
    ----------
    gdf : GeoDataFrame
        Polygon layer in a projected (metric) CRS.
    iou_threshold : float
        Minimum IoU (0-1) for two polygons to merge into one group. 0.9 is a
        strict "near-identical" default.
    priority_col : str | None
        Column whose highest value wins the survivor slot within a group
        (e.g. a source-quality score or capture timestamp).
    keep : {'first', 'largest'}
        Fallback survivor rule when priority_col is None.

    Returns
    -------
    GeoDataFrame
        One representative polygon per duplicate group, in the original CRS.
    """
    if gdf.crs is None or gdf.crs.is_geographic:
        raise ValueError("Project to a metric CRS before area-based IoU dedup.")
    if len(gdf) < 2:
        return gdf.copy()

    work = gdf.reset_index(drop=True)
    geom = work.geometry.values  # GeometryArray for vectorized ops

    # 1. Candidate pairs: self-join returns every bounding-box-intersecting pair.
    joined = gpd.sjoin(
        work[["geometry"]], work[["geometry"]],
        predicate="intersects", how="inner",
    )
    left = joined.index.to_numpy()
    right = joined["index_right"].to_numpy()

    # 2. Keep ordered, non-self pairs only (i < j) to halve the work.
    ordered = left < right
    left, right = left[ordered], right[ordered]
    if len(left) == 0:
        return gdf.copy()

    # 3. Vectorized IoU across all candidate pairs in two GEOS calls.
    inter = shapely.area(shapely.intersection(geom[left], geom[right]))
    union = shapely.area(shapely.union(geom[left], geom[right]))
    with np.errstate(divide="ignore", invalid="ignore"):
        iou = np.where(union > 0, inter / union, 0.0)

    dup = iou >= iou_threshold
    dup_left, dup_right = left[dup], right[dup]
    logger.info(
        "sjoin found %d intersecting pairs; %d exceed IoU %.2f",
        len(left), int(dup.sum()), iou_threshold,
    )

    # 4. Union-find so transitive overlaps (A~B, B~C) collapse into one group.
    parent = np.arange(len(work))

    def find(i: int) -> int:
        while parent[i] != i:
            parent[i] = parent[parent[i]]  # path halving
            i = parent[i]
        return i

    for i, j in zip(dup_left.tolist(), dup_right.tolist()):
        ri, rj = find(i), find(j)
        if ri != rj:
            parent[ri] = rj

    roots = np.array([find(i) for i in range(len(work))])

    # 5. Choose one survivor index per group; higher order value wins.
    if priority_col is not None:
        order = work[priority_col].to_numpy()
    elif keep == "largest":
        order = shapely.area(geom)
    else:  # 'first' — lowest original position wins
        order = -np.arange(len(work))

    survivors = [
        members[np.argmax(order[members])]
        for members in (np.where(roots == r)[0] for r in np.unique(roots))
    ]

    result = work.iloc[sorted(survivors)].reset_index(drop=True)
    logger.info(
        "dedupe_overlapping_polygons: %d -> %d features (removed %d)",
        len(work), len(result), len(work) - len(result),
    )
    return result

Key Implementation Notes

  • The self-join prunes; IoU decides. gpd.sjoin uses the GEOS STRtree to return only pairs whose envelopes intersect, so the expensive area math runs on a short candidate list rather than the full n² grid. This is the same spatial-index discipline that makes large-layer deduplication tractable at all.
  • i < j removes identity and mirror pairs in one step. A self-join returns each polygon paired with itself and each real pair twice (A–B and B–A). Filtering to strictly increasing index pairs drops both classes without a separate de-mirroring pass.
  • IoU is fully vectorized. shapely.intersection(geom[left], geom[right]) and shapely.union(...) each dispatch a single GEOS call over the whole candidate array; dividing the two area arrays gives every IoU at once. There is no Python-level loop over geometry pairs.
  • Union-find captures transitive duplicates. Three near-identical copies where A overlaps B and B overlaps C but A and C are slightly offset still belong to one group. Connected-component merging guarantees they collapse to a single survivor, which a naive keep-one-of-each-pair rule would miss.
  • The survivor rule is explicit, not incidental. Passing priority_col (a quality flag, a source rank, an ingest timestamp) makes the kept feature a deliberate choice. The keep='first' default is only a fallback and depends on stable row order, so prefer an explicit priority in production.
  • Points need a different tool. This routine is area-based and only meaningful for polygons; near-duplicate point layers use a distance radius instead, covered in removing duplicate spatial points with tolerance thresholds.

Troubleshooting Overlap Deduplication

Symptom Root Cause Fix
Every polygon pairs with itself Self-join includes identity matches Filter left < right to drop identity and mirror pairs
Distinct-but-touching polygons wrongly merged predicate='intersects' matches edge touches Use predicate='overlaps' or raise iou_threshold
MemoryError on the self-join Dense overlaps produce O(n²) candidate pairs Tile by a spatial index and dedupe per tile with a boundary buffer pass
The wrong copy is kept keep='first' ignores data quality Pass priority_col with a quality score or timestamp
Partial overlaps survive dedup IoU threshold too high for offset digitizations Lower iou_threshold or switch predicate to overlaps

Integration Note

Run this step in the Transform stage after geometry repair and before any attribute aggregation, so downstream sums are computed over a deduplicated layer. It pairs naturally with the other routines in spatial deduplication and topology simplification: resolve overlapping source polygons here first, then hand the clean partition to simplifying polygons while preserving shared boundaries, because simplifying an unresolved overlap only hides it. Where duplicate copies carry conflicting attributes, reconcile the columns using the rules in attribute mapping and schema harmonisation before choosing a survivor, so the kept row also carries the authoritative values.