Bucket Mapping for Raster Data

Raster datasets—satellite imagery, digital elevation models (DEMs), LiDAR derivatives, and climate reanalysis grids—introduce distinct storage and query challenges in modern data lakehouses. Unlike vector geometries, rasters are inherently grid-aligned, frequently exceed multi-terabyte scales, and exhibit highly localized, bounding-box-driven access patterns. Bucket mapping translates continuous spatial coordinates into discrete, query-optimized partition directories, enabling efficient predicate pushdown, metadata pruning, and predictable I/O behavior. This technique operates as a specialized implementation layer within broader Spatial Partitioning & Indexing Strategies, where physical storage layout must align tightly with downstream analytical and GIS processing workloads.

Deterministic Coordinate Transformation & CRS Alignment

Effective bucket mapping begins with deterministic coordinate transformation. Raw geographic coordinates (WGS84, EPSG:4326) introduce severe distortion and uneven bucket sizes at higher latitudes, making them unsuitable for direct partitioning. Production pipelines must project raster extents into a metric-aligned coordinate reference system (CRS) before computing bucket identifiers.

For continental-scale ingestion, Universal Transverse Mercator (UTM) zones establish a natural, meter-based grid. The ingestion pipeline derives the UTM zone identifier, truncates easting/northing values to a fixed tile size, and hashes them into partition columns. See Mapping UTM zones to Iceberg partition columns for detailed schema evolution patterns.

Explicit Production Parameters:

• Target CRS: EPSG:32633 (UTM Zone 33N)
• Tile Size: 2000m × 2000m (aligns with typical 1024×1024 GeoTIFF block boundaries)
• Bucket Formula: bucket_id = CONCAT(FLOOR(easting / 2000), '_', FLOOR(northing / 2000))
• Partition Bounds: easting ∈ [100000, 900000], northing ∈ [1100000, 9200000]

This deterministic mapping ensures that adjacent spatial tiles map to predictable directory paths, preventing coordinate drift from propagating into consumer queries.

Partition Hierarchy & Directory Layout

Architecting the partition hierarchy requires balancing directory depth against query selectivity. Spatial Partitioning Schemes outlines the trade-offs between coarse administrative boundaries, hierarchical quadtrees, and flat spatial hashing. For raster workloads, a two-tier partition strategy consistently delivers optimal query planner performance:

1. Coarse Partition: utm_zone, acquisition_year, sensor_type
2. Bucket Partition: spatial_bucket (string-encoded grid cell)

This structure prevents metadata explosion while maintaining high pruning efficiency for regional or temporal queries. Target file sizes should align with cloud storage block limits (typically 128MB–512MB per Parquet/GeoParquet file) to avoid excessive LIST API calls during manifest reads. Over-partitioning below 64MB/file triggers metadata bloat, while under-partitioning above 2GB/file degrades parallel read throughput.

Production Implementation Patterns

PySpark Ingestion Pipeline

The following snippet demonstrates coordinate projection, bucket derivation, and Iceberg table writes with explicit partition specs. It assumes easting_utm33n and northing_utm33n columns were computed upstream (e.g., via pyproj or Sedona ST_Transform):

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, floor, concat_ws
import pyspark.sql.types as T

spark = SparkSession.builder \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.lakehouse", "org.apache.iceberg.spark.SparkCatalog") \
    .getOrCreate()

# Load raw raster catalog (GeoTIFF paths + bounding boxes pre-projected to EPSG:32633)
raw_df = spark.read.parquet("s3://raw-catalog/landsat_metadata/")

# Compute 2000m bucket IDs from UTM 33N coordinates
bucket_df = raw_df.withColumn(
    "spatial_bucket",
    concat_ws(
        "_",
        floor(col("easting_utm33n") / 2000).cast(T.IntegerType()),
        floor(col("northing_utm33n") / 2000).cast(T.IntegerType())
    )
)

# Write with Iceberg partitioning
bucket_df.writeTo("lakehouse.raster.landsat_bucketed") \
    .partitionedBy("utm_zone", "acquisition_year", "spatial_bucket") \
    .append()

SQL DDL & Query Pruning

Define the table schema and verify partition pruning via EXPLAIN:

CREATE TABLE lakehouse.raster.landsat_bucketed (
    raster_path      STRING,
    sensor_type      STRING,
    acquisition_date DATE,
    utm_zone         INT,
    easting          DOUBLE,
    northing         DOUBLE,
    spatial_bucket   STRING
) USING iceberg
PARTITIONED BY (utm_zone, year(acquisition_date), spatial_bucket);

-- Query engine will prune partitions matching the bucket range
EXPLAIN SELECT COUNT(*) FROM lakehouse.raster.landsat_bucketed
WHERE spatial_bucket BETWEEN '450_5200' AND '455_5205'
  AND acquisition_date >= '2023-01-01';

CI/CD Validation Step

Validate partition structure before merging pipeline changes:

# .github/workflows/validate-partitions.yml
name: Validate Raster Partition Schema
on: [pull_request]
jobs:
  check-partitions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run partition validator
        run: |
          python scripts/validate_bucket_schema.py \
            --catalog-path s3://lakehouse/raster/landsat/ \
            --expected-tile-size 2000 \
            --crs EPSG:32633 \
            --max-partitions-per-year 15000

Query Execution & Pruning Mechanics

Bucket mapping enables the query planner to translate spatial predicates directly into directory scans. When a bounding box intersects multiple tiles, the engine computes the overlapping spatial_bucket range, reads only the relevant manifest entries, and skips non-matching partitions entirely. This reduces I/O by 60–85% compared to unpartitioned lakehouse scans.

For multi-dimensional workloads combining spatial, temporal, and spectral filters, bucket mapping pairs effectively with Z-Ordering for Geospatial Queries. While bucket mapping handles coarse directory pruning, Z-ordering optimizes file-level data layout within those directories, minimizing the number of Parquet row groups scanned per query.

Operational Guardrails & Troubleshooting

Common Failure Modes

Retention & Lifecycle Policies

Raster archives require strict lifecycle management to control storage costs. Implement time-based retention aligned with data utility:

• Operational Satellite Imagery: retention_days = 1095 (3-year rolling)
• Climate Reanalysis Grids: retention_days = 3650 (10-year archival)
• LiDAR Point Clouds: retention_days = 7300 (20-year compliance)

Automate cleanup using scheduled Iceberg maintenance:

CALL lakehouse.system.expire_snapshots(
    table => 'lakehouse.raster.landsat_bucketed',
    older_than => TIMESTAMPADD(YEAR, -3, CURRENT_TIMESTAMP),
    retain_last => 5
);

Debugging Workflow

1. Verify Bucket Alignment: Cross-check spatial_bucket values against known tile boundaries using the GDAL Raster Data Model reference.
2. Inspect Manifests: Query lakehouse.raster.landsat_bucketed.partitions to confirm partition distribution matches expected spatial density.
3. Profile Query Plans: Run EXPLAIN FORMATTED to verify PartitionFilters predicates are applied before FileScan.
4. Validate CRS Consistency: Ensure all upstream producers reference the official EPSG Geodetic Parameter Dataset to prevent silent coordinate drift.

Bucket mapping transforms unstructured raster sprawl into deterministic, query-ready storage layouts. By enforcing strict CRS alignment, calibrated tile sizes, and automated lifecycle policies, platform teams can deliver sub-second spatial pruning at petabyte scale while maintaining full compatibility with downstream GIS and analytical engines.

Why Raster Does Not Fit a Row-Oriented Table

Open table formats model a table as rows with columns. A raster is a regular grid of values with an affine transform mapping grid indices to coordinates. Forcing one into the other is possible in three ways, and the choice determines everything downstream.

The middle option is the one this section is mostly about, because it is the only one that keeps pixel values inside the lakehouse at a scale that stays operable. The arithmetic is straightforward: a 10,980-pixel-square scene at 10-metre resolution holds around 120 million pixels per band, so a row-per-pixel table for a year of global coverage reaches a scale no engine will enjoy. The same scene split into 512-pixel tiles is about 460 rows per band — a number that partitions, prunes and joins comfortably.

The right-hand option is not a failure to commit; it is frequently the correct architecture. Keeping pixels in Cloud-Optimised GeoTIFF or Zarr, and keeping footprints, acquisition times, cloud cover and quality flags in a lakehouse table, gives fast discovery over a huge archive with a table that stays small. Analysis then reads only the assets the discovery query selected. The cost is that the pixel access path is not SQL, which matters only if pixel-level SQL is actually required.

Most mature platforms run the right-hand and middle patterns together: a catalogue table over the full archive, and tiled tables for the specific products that analysts query repeatedly. That combination keeps the expensive representation scoped to the data that earns it.

Aligning Raster Tiles With Vector Partitions

The reason to think about raster bucketing at all, rather than treating it as a separate storage problem, is that raster and vector data are usually joined — extracting values at points, summarising a band within polygons, masking one by the other. The join is efficient only when the two layouts agree.

The four-to-one difference shown here is the best case for misalignment; with an offset in both axes and a tile smaller than the cell it is worse. On a join over millions of cells, that factor multiplies directly into bytes read and into shuffle volume, and it is entirely avoidable by choosing the tile origin and size so that tile boundaries fall on partition-cell boundaries.

Achieving alignment usually means accepting a tile size that is not a round number of pixels, or a partition resolution that is not the one a purely vector analysis would choose. That trade is worth making when raster-vector joins are a routine workload, and not worth making when they are occasional — in which case a reprojection-and-resample step at join time is cheaper than distorting both layouts permanently.

One caution: alignment holds only within a single projection. A raster tiled in a projected coordinate system and a vector table partitioned by a geographic grid cannot align, because the cell boundaries are curves in the raster’s space. Where both matter, pick the coordinate system for the join, and store a partition key derived in that system on both sides.

Choosing a Tile Size

Tile size is the raster equivalent of partition resolution, and the same arithmetic decides it: the tile should be large enough that the per-row overhead is negligible and small enough that a typical query does not read much it does not need.

The 256-to-512 pixel band is where most production systems land, and the reason is not arbitrary. At 512 pixels square with a 16-bit band, a tile holds about 512 KB uncompressed — large enough that the row overhead and the per-tile metadata are irrelevant, small enough that a point query decodes half a megabyte rather than fifty. It also matches the internal tiling of most Cloud-Optimised GeoTIFF products, so a conversion is a copy rather than a resample.

Two workload characteristics push the choice within the band. Point sampling — extracting a value at scattered locations — favours smaller tiles, because every sample decodes a whole tile to read one pixel. Window statistics over polygons favour larger tiles, because the decode cost amortises across many pixels that are all needed. Where both workloads matter and the archive is large enough to justify it, storing two tilings is defensible; where it is not, size for the more frequent one.

The decision is effectively permanent, because re-tiling is a full rewrite of the pixel data rather than a metadata change. That asymmetry with vector partitioning is worth stating plainly: a vector partition key can often evolve, a raster tiling cannot, so the measurement effort belongs up front.

Metadata Every Tile Row Needs

A tile row is only useful if the reader can place it, decode it and trust it without consulting anything else.

The spatial placement needs both the tile’s bounding box in the table’s declared coordinate system and the affine transform mapping pixel indices to coordinates. The bounding box drives pruning; the transform is what lets a consumer compute which pixel corresponds to a given location without re-deriving it from the tiling scheme.

The decode contract needs the data type, the band count, the compression codec and the nodata value. Nodata in particular has to be explicit, because the alternative is that every consumer invents its own convention and aggregate statistics silently include fill values. A mean computed over a tile whose nodata is 0 and whose consumer assumed -9999 is wrong by an amount nobody will notice.

The provenance needs the source scene identifier, the acquisition timestamp and the processing version. These are what make a result reproducible and what make a targeted reprocessing possible — when a provider reissues a scene, the rows to replace are exactly those carrying its identifier, and finding them should be a predicate rather than a search.

Finally, a quality summary per tile pays for itself immediately: the fraction of valid pixels, and the minimum and maximum of each band. The valid fraction lets a query skip tiles that are entirely cloud or entirely outside the swath before decoding anything, and the per-band range serves the same pruning role for value-based filters that the bounding box serves for spatial ones.

Operational Notes for Tiled Raster Tables

Tiled raster tables behave differently from vector tables in maintenance, and three habits keep them healthy.

Compaction has a lower ceiling. A tile row already holds a large binary payload, so target file sizes are reached with far fewer rows than a vector table needs. Compacting to 128 MB files means roughly 250 tiles per file at half a megabyte each, which is a perfectly reasonable target — but a compaction configured by row count rather than by byte size will produce multi-gigabyte files and long tail-latency reads.

Sorting matters less, tiling matters more. Because the tile identifier already encodes position, and because tiles are large, within-file ordering has much less effect than it does for point data. Effort that would go into sort tuning on a vector table is better spent on tile alignment and on the per-tile quality summary.

Reprocessing is routine. Providers reissue scenes, and a raster table that cannot cleanly replace one scene’s tiles will accumulate duplicates or require full rewrites. Partitioning by scene identifier — or at minimum keeping it as a column with statistics — makes the replacement a scoped delete-and-append rather than a table-wide operation.

Together these keep the table’s maintenance cost proportional to the data that changes rather than to the archive’s total size, which is the only property that lets a raster table grow for years without the operations budget growing with it.

A Short Decision Summary

Store pixels as tiles inside the lakehouse when analysts run repeated value extractions and the product set is bounded. Keep pixels in their native cloud-optimised format with a lakehouse catalogue over the footprints when the archive is large, heterogeneous and mostly searched rather than computed over. Size tiles in the 256-to-512 pixel band unless a measurement says otherwise, align tile boundaries to the vector partition grid when raster-vector joins are routine, and give every tile row a bounding box, an affine transform, an explicit nodata value, a provenance identifier and a valid-pixel fraction. Those five fields are what make the table self-describing enough that a query planner can prune it and a future reader can use it, which is the same standard the vector tables on this site are held to.

The recurring theme is that raster in a lakehouse succeeds when the table describes the pixels rather than trying to be the pixels. Every field above exists so that a planner can decide what to read and a consumer can decode what it gets, and the tables that disappoint are almost always the ones that stored the bytes and left the description to the pipeline that wrote them. Design the description first, and the storage decision usually follows from it.

A last practical suggestion: keep a single worked example of the full path — one scene, tiled, catalogued, joined to a vector layer and sampled at points — in the repository as an executable notebook. Raster layouts have enough moving parts that a written specification is regularly interpreted two ways, and a runnable example settles the interpretation in seconds. A notebook that runs is worth more than a page that describes. Keep it current, and keep it small.