Selecting a Discrete Global Grid for Lakehouse Partitioning

Choosing the discrete global grid that becomes your partition key is the single highest-leverage decision in a spatial lakehouse layout: it fixes cell shape, neighbor semantics, partition cardinality, and how cleanly the key maps onto Iceberg and Delta partition transforms. This topic area compares the three grids that dominate production data engineering — H3 (hexagonal), S2 (quadrilateral cells ordered along a Hilbert curve), and geohash (base-32 rectangles) — and gives you a defensible way to pick one and size its resolution. It sits inside the broader Spatial Partitioning & Indexing Strategies section and assumes you have already ruled out naive range or hash partitioning on raw coordinates because of skew.

When to use a discrete global grid

A discrete global grid (DGG) earns its place when your access pattern is “give me everything near here” and your storage engine prunes on an equality or set predicate over a partition column. If your queries are pure bounding-box range scans, an in-file space-filling-curve layout via Z-ordering for geospatial queries may serve you better than a grid partition. Use the table below to decide.

The rest of this guide treats the grid id as a first-class partition column. For a wider comparison of grid schemes against space-filling curves and tree indexes, see spatial partitioning schemes.

How the three grids differ

The three systems disagree on four properties that matter for partitioning: cell shape, area distortion across latitude, the parent/child hierarchy, and neighbor traversal.

H3 tiles the sphere with hexagons (plus 12 pentagons at icosahedron vertices). Every hexagon has exactly six equidistant neighbors, which makes ring queries and flow modeling clean. Resolutions run 0–15; each finer resolution divides a cell into 7 children (aperture-7), so the hierarchy is not strictly containing — a child hexagon can straddle a parent boundary. Cell ids are 64-bit integers, conventionally handled as 15-character hex strings such as 872830828ffffff.

S2 projects the sphere onto the six faces of a cube and recursively subdivides each face into four quadrilateral children, ordering the cells along a Hilbert space-filling curve. Levels run 0–30. Because subdivision is aperture-4 and strictly containing, an S2 cell id encodes its full ancestry: truncating the 64-bit id gives you the parent, and a contiguous id range covers a contiguous region. That range property is what makes S2 attractive for covering an arbitrary polygon with a compact set of cell ranges.

Geohash interleaves latitude and longitude bits and base-32 encodes them into strings where each added character refines the cell and every prefix is a valid coarser cell. Cells are lat/lon rectangles in EPSG:4326, so they distort badly toward the poles and are non-square at most latitudes. Geohash has no native hexagonal neighborhood; adjacency requires computing the eight bordering rectangles, and neighbors across a base-32 “seam” share no common prefix — the classic geohash edge problem.

Prerequisites and environment setup

Install the three encoders plus a Spark/Iceberg or Delta runtime. The h3 package is h3-py 4.x (module-level functions), s2sphere is a pure-Python S2 implementation, and python-geohash provides geohash.encode.

python -m pip install "h3>=4.1,<5" "s2sphere>=0.2.5" "python-geohash>=0.8.5" \
    "pyspark==3.5.1" "delta-spark==3.2.0"

For Iceberg, use the Spark 3.5 runtime with the Iceberg 1.9.0 jars and format-version 2 tables. The grid encoders run on the driver during UDF registration but execute per-row on executors, so every worker image must contain them.

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("grid-system-selection")
    .config("spark.jars.packages",
            "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0")
    .config("spark.sql.extensions",
            "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.lake.type", "hadoop")
    .config("spark.sql.catalog.lake.warehouse", "s3://lakehouse/warehouse")
    .getOrCreate()
)

Step-by-step implementation

1. Encode the point to all three cell keys

Register one UDF per grid so you can compare cardinality on real data before committing. All three take EPSG:4326 lat/lon and return a string key.

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
import h3
import s2sphere
import geohash  # python-geohash

@udf(StringType())
def h3_key(lat: float, lon: float, res: int = 7) -> str:
    # h3-py 4.x: module-level latlng_to_cell returns a 15-char hex id
    return h3.latlng_to_cell(lat, lon, res)

@udf(StringType())
def s2_key(lat: float, lon: float, level: int = 13) -> str:
    ll = s2sphere.LatLng.from_degrees(lat, lon)
    cell = s2sphere.CellId.from_lat_lng(ll).parent(level)
    return cell.to_token()  # compact hex token, e.g. '89c25a'

@udf(StringType())
def geohash_key(lat: float, lon: float, precision: int = 7) -> str:
    return geohash.encode(lat, lon, precision=precision)

The resolution defaults above are deliberately chosen to yield roughly comparable cell footprints: H3 res 7 (~5 km² average), S2 level 13 (~1.3 km² average), geohash precision 7 (~150 m × 150 m). Resolution sizing, not the grid choice alone, controls partition cardinality — see choosing an H3 resolution for point data for the sizing method.

2. Materialize the partition column

Add the chosen grid id as a real column so the engine can prune on it. Do this once at ingest; never compute the key inside a query predicate, because a UDF in the WHERE clause defeats partition pruning.

from pyspark.sql.functions import col

raw = (
    spark.read.schema("device_id STRING, lat DOUBLE, lon DOUBLE, ts TIMESTAMP")
    .json("s3://raw-telemetry/2024-10/")
    .filter((col("lat").between(-90, 90)) & (col("lon").between(-180, 180)))
)

enriched = raw.withColumn("h3_res7", h3_key(col("lat"), col("lon")))

3. Map the key onto a partition transform

Both formats accept the string key directly. Wrapping it in a bucket transform caps directory/manifest count when cell cardinality is high — the recommended default for a global dataset.

-- Apache Iceberg (Spark SQL), format-version 2, Iceberg 1.9.0
CREATE TABLE lake.geo.events (
    device_id STRING,
    lat DOUBLE,
    lon DOUBLE,
    ts TIMESTAMP,
    h3_res7 STRING
) USING iceberg
PARTITIONED BY (bucket(256, h3_res7), days(ts))
TBLPROPERTIES (
    'format-version' = '2',
    'write.parquet.compression-codec' = 'zstd'
);

For Delta Lake, partitionBy("h3_res7") writes physical directories, so prefer a coarser resolution there; Iceberg’s bucket transform is the safer high-cardinality choice. The concrete Delta directory-based flow is covered in implementing H3 hexagon partitioning in Delta Lake.

Verification and testing

Confirm the key prunes partitions and check its cardinality before you scale ingest. Cardinality that climbs past ~50k live partitions per day is the leading cause of manifest bloat.

enriched.createOrReplaceTempView("enriched")
spark.sql("""
    SELECT COUNT(DISTINCT h3_res7) AS cells,
           COUNT(*)                AS rows,
           ROUND(COUNT(*) / COUNT(DISTINCT h3_res7), 1) AS rows_per_cell
    FROM enriched
""").show()

Then validate that an equality predicate on the key produces PartitionFilters rather than a full scan:

EXPLAIN
SELECT * FROM lake.geo.events
WHERE h3_res7 = '872830828ffffff'
  AND ts >= DATE '2024-10-01';

A healthy plan lists the grid column under partition filters and shows a bucket/partition count far below the table total. If it does not, the predicate is being applied post-scan — see predicate pushdown optimization for the pushdown checklist.

Performance and tuning

• Resolution is the dominant knob. Each H3 level multiplies cell count by ~7, each S2 level by 4, each geohash character by 32. Moving H3 res 7 → 8 roughly septuples partition cardinality. Size for 100 MB–1 GB of data per cell after compaction.
• Bucket count. With bucket(N, grid_id), set N so that rows_per_bucket × avg_row_bytes lands in the 128 MB–1 GB target. For a 500 GB daily table, N between 256 and 1024 is typical.
• Neighbor queries. H3 grid_disk(cell, k) expands to a k-ring in one call; geohash needs geohash.neighbors() iterated per level and must handle base-32 seams; S2 uses S2CellUnion range covers. If your workload is ring-heavy, H3’s constant 6-neighbor topology cuts query fan-out.
• Hot partitions. Dense urban cells hold orders of magnitude more rows than rural ones. The bucket transform spreads a hot cell across files by hashing; without it, a single H3 cell over a city center becomes a multi-gigabyte partition that stalls compaction. Pair coarse grid partitions with intra-file Z-ordering on lon, lat to keep row groups locally clustered.
• Interop cost. Geohash strings are self-describing and need no library to decode a bounding box, which lowers cross-team friction; H3 and S2 ids are opaque without their libraries.

Common errors and fixes

Once you have chosen a grid, the two guides in this topic area go deeper: a full head-to-head H3 vs S2 vs geohash comparison with runnable cardinality benchmarks, and a data-driven method for choosing an H3 resolution for point data. Authoritative references: the H3 documentation and the S2 Geometry library.