GeoParquet vs WKB Column Storage: Trade-offs
This guide settles a concrete storage decision: whether to persist geometry as a GeoParquet-native column carrying file-level geo metadata and a bbox coverin…
Read the guideA production-focused resource for implementing, optimizing, and maintaining spatial data in open table formats (Apache Iceberg, Delta Lake). The transition from monolithic spatial databases to a spatial data lakehouse is not a storage migration — it is a fundamental re-architecture of how geospatial data is serialized, versioned, indexed, and queried at scale.
This site documents the engineering contracts required to make that architecture deliver: deterministic geometry serialization (WKB / GeoParquet), partition strategies that align with real query patterns, predicate pushdown that actually pushes down, and Python orchestration that respects snapshot isolation.
Every guide is written for data engineers, platform architects, gis backend developers, and cloud/infrastructure teams. who own the pipeline end-to-end — from S3 buckets and catalog manifests to PySpark configurations and CI/CD validation gates. No vendor pitches, no toy examples; just patterns and trade-offs that survive contact with petabyte-scale data.
The catalogue now spans four sections and twenty-five topic areas, from geometry serialization and validation through partitioning, join tuning and streaming ingest to the query engines and the aggregation layer that serves maps and dashboards. Every page carries runnable code, the measurements that justify each recommendation, and the failure modes that make the recommendation necessary.
Pick a section below to explore deeper topics, troubleshooting matrices, and ready-to-paste configurations — or start with the deep-dives immediately below, which answer the questions teams ask first.
Nine deep-dives that answer the questions teams ask first — format and grid comparisons, the layout work that makes queries prune, streaming ingest, and the aggregation patterns behind fast dashboards.
This guide settles a concrete storage decision: whether to persist geometry as a GeoParquet-native column carrying file-level geo metadata and a bbox coverin…
Read the guideThis guide compares Apache Iceberg and Delta Lake specifically for spatial workloads — how each handles geometry, partitioning, clustering, engine access, an…
Read the guideThis guide runs H3, S2, and geohash head-to-head on the same points, measures the partition cardinality each produces at comparable cell sizes, and turns the…
Read the guideThis guide adds the four numeric bounding-box columns to an existing spatial table, places them where statistics will be collected, backfills history without…
Read the guideThis guide gives a measurement-driven decision procedure for the two distributed spatial join strategies, with the three numbers that decide it, the code to…
Read the guideThis guide is a complete, runnable recipe for loading DuckDB's spatial extension, reading a GeoParquet file straight from disk or S3, and executing an ST_Int…
Read the guideThis guide is a complete, runnable consumer that reads geospatial events from Kafka, validates and enriches them, and commits them to an Iceberg table with b…
Read the guideThis guide gives a complete, runnable recipe for compacting a small-file-ridden spatial Iceberg table using the rewrite_data_files procedure with a bounding-…
Read the guideThis guide computes counts and statistics per grid cell from a large point table using plain SQL, with the derived-column layout that makes it fast and the r…
Read the guideThe most recent additions to the catalogue — geometry quality, platform observability, join tuning, streaming ingest, and the aggregation and tiling layer that serves maps and dashboards.
An invalid geometry does not raise an error. It sits in the table, participates in joins, contributes to aggregates, and returns answers that are wrong in wa…
Most spatial lakehouse failures are silent. A partition grows until it becomes a straggler, a clustering order decays until pruning stops working, a feed sto…
A spatial join is the operation that decides whether a lakehouse feels fast or unusable. It is also the operation where the gap between a naive and a well-de…
A streaming spatial pipeline is where every constraint in this site meets at once: the write must be fast enough to keep up, the geometry must be validated b…
Most of what a spatial lakehouse serves is not raw geometry — it is summaries: counts per cell, densities per region, tiles for a map. Those outputs are comp…
Decouple storage, catalog, and compute. Master geometry serialization (WKB/GeoParquet), snapshot semantics, and the Iceberg/Delta trade-offs that govern production spatial stacks.
Hierarchical grids, Z-ordering, Hilbert curves, predicate pushdown, and raster/vector hybrid layouts engineered for sub-second queries at petabyte scale.
Arrow schemas, PyIceberg, delta-rs, async catalog orchestration, and CI/CD validation — the Python contract that keeps spatial pipelines reproducible and fast.
DuckDB, Trino, and Apache Sedona against lakehouse tables — spatial SQL functions, distributed joins, catalog federation, and the benchmarks that decide which engine owns each workload.