Using delta-rs to write spatial parquet files

Unbounded memory consumption and transaction contention are the dominant failure modes when ingesting vector geometries into Delta Lake. The root cause is a structural mismatch between in-memory spatial representations, Parquet row group boundaries, and Delta’s optimistic concurrency control. Resolving this requires a deterministic write pipeline that isolates geometry serialization, enforces strict partition pruning, and leverages the async Rust execution engine. This guide details the exact configuration, failure resolution, and parameter tuning required for production-grade spatial ingestion.

Serialization Contract & Schema Enforcement

Spatial columns stored as raw WKB bytes lack native Delta type hints. Relying on automatic inference defaults to generic binary without spatial metadata, which breaks downstream spatial indexing and forces full table scans. Pre-serialize geometries to Well-Known Binary (WKB) and attach GeoParquet-compliant metadata before invoking the write engine. This aligns with the Delta-rs Geometry Processing validation pipeline, ensuring CRS consistency and bounding-box constraints are enforced prior to heap allocation. Stripping GeoJSON overhead reduces write-phase memory pressure by 40–60%.

Explicit schema enforcement prevents drift during schema evolution. Always construct a pyarrow.Schema object with pa.binary() for geometry columns and pass it directly to the writer. Omitting this step triggers naive binary inference, causing row group fragmentation and amplifying compaction overhead during OPTIMIZE cycles.

Async Execution & Chunked Write Pipeline

The synchronous write_deltalake API blocks the Python GIL and serializes commit attempts, which causes transaction retries under concurrent workloads. Production pipelines must stream spatial data through chunked iterators that respect a 256MB–512MB per-partition threshold. Chunking must occur before the Rust write engine is invoked, as delta-rs does not perform automatic spill-to-disk during serialization.

import asyncio
import pyarrow as pa
from deltalake import write_deltalake
from concurrent.futures import ThreadPoolExecutor

# Explicit schema to prevent drift and enforce WKB binary layout
SPATIAL_SCHEMA = pa.schema([
    ("geometry",  pa.binary()),             # WKB bytes, little-endian
    ("h3_res8",   pa.string()),             # H3 partition key
    ("event_ts",  pa.timestamp("us")),
    ("bbox_min_x", pa.float64()),
    ("bbox_min_y", pa.float64()),
    ("bbox_max_x", pa.float64()),
    ("bbox_max_y", pa.float64()),
])

async def stream_write_spatial(table_uri: str, chunk_iterator, max_workers: int = 4):
    """
    Writes spatial chunks to Delta Lake using the Rust engine.
    Each chunk is offloaded to a thread pool to avoid blocking the event loop.
    """
    loop = asyncio.get_running_loop()
    with ThreadPoolExecutor(max_workers=max_workers) as pool:
        tasks = []
        for chunk in chunk_iterator:
            # Enforce schema at Arrow conversion boundary
            arrow_tbl = pa.Table.from_pandas(chunk, schema=SPATIAL_SCHEMA)

            # Offload to delta-rs Rust runtime in thread pool
            task = loop.run_in_executor(
                pool,
                lambda tbl=arrow_tbl: write_deltalake(
                    table_or_uri=table_uri,
                    data=tbl,
                    mode="append",
                    partition_by=["h3_res8"],
                    schema_mode="merge"
                )
            )
            tasks.append(task)

        # Await all chunks; propagate first exception
        results = await asyncio.gather(*tasks, return_exceptions=True)
        for res in results:
            if isinstance(res, Exception):
                raise res

This pattern integrates cleanly into broader Python Ecosystem & Integration Workflows by decoupling DataFrame materialization from the write boundary. Aligning DataFrame partitions with Delta’s target file size prevents write amplification and ensures each Parquet file contains a single, contiguous spatial extent.

Failure Modes & Deterministic Resolution

Production Parameter Matrix

Post-write, compact fragmented files:

from deltalake import DeltaTable

dt = DeltaTable(table_uri)
# Bin-pack into 512MB target files
dt.optimize.compact(target_size=512 * 1024 * 1024)

Monitor _delta_log commit latency; sustained values >2s indicate partition skew or insufficient backoff configuration. Enforce deterministic serialization contracts at the ingestion boundary to eliminate schema drift and guarantee reproducible spatial lakehouse performance.

The Schema Is the Whole Design

Everything that determines whether the resulting table is fast lives in the schema declaration, and it is worth laying out explicitly before writing any code.

Putting the geometry column last is deliberate on two counts. Statistics on a WKB column are meaningless — the minimum and maximum of a byte array carry no spatial information — and collecting them wastes space in the log while inflating the statistics payload on every commit. Placing wide columns beyond the window also leaves room for the columns that genuinely benefit.

The four bounding-box columns should be adjacent and immediately after the partition column. Adjacency is not required by anything, but it makes the schema self-documenting and makes the “are they inside the window” check a matter of counting to eight rather than auditing a fifty-column list.

Getting the Write Options Right

The schema-mode default deserves particular attention on spatial tables. Automatic merging is convenient during development and dangerous in production: a batch with a mistyped column name silently adds a column rather than failing, and the mistake is only visible later as a mostly-null field nobody can account for. Enable merging deliberately, for the write that is meant to evolve the schema, and leave it off otherwise.

Verifying the Result Immediately

This check costs one JSON read and takes a second. Wire it into the write job as a post-condition rather than running it manually, because the property it guards is the one that silently disappears when a schema grows — and a schema that grows is the normal course of events rather than an exception.

Common Mistakes

A short list of what goes wrong in practice, all of which the checks above catch:

• Geometry stored as WKT. Convenient during debugging and roughly twice the size on disk, with a slower decode on every read. Convert to WKB before writing and keep WKT for log lines only.
• Bounding boxes derived before reprojection. The values describe the geometry’s old position, so skipping excludes the files a query needs. Derive after every transform, and assert coverage.
• Mixed dimensionality. A source containing both 2D and 3D geometries produces a column that behaves inconsistently across engines. Force to two dimensions at ingest unless elevation is genuinely required, and record the decision in table properties.
• Partitioning on a fine grid resolution. Delta writes each partition as a directory, and a fine cell produces a directory explosion that no compaction repairs. Partition on day and a coarse cell; cluster on the fine one.
• Relying on schema inference per batch. A batch whose optional column is entirely null infers a different schema and fails the write. Declare the schema once and cast every batch to it.
• Committing per row group. Each commit adds a log entry, and a job that commits thousands of times leaves a table whose reads spend longer replaying the log than reading data. Accumulate and commit once per unit of work.

None of these produces an error at write time except the last two, which is precisely why the post-write verification matters more here than it would for a scalar table.

The habit that prevents all six is to treat the write function as a contract rather than as a script: it declares its schema, it derives what it needs, it asserts its post-conditions, and it fails rather than producing a table that merely resembles the intended one. Everything else in this guide follows from that stance, and pipelines that adopt it tend to need very little maintenance afterwards.

For the wider context — when this path is the right one at all, and what to do when the data outgrows it — see delta-rs geometry processing.

That page also covers the maintenance operations available from the same Python process, so a table created with this recipe can be compacted and vacuumed without introducing a second toolchain.

Keeping the whole lifecycle in one language and one process is much of the appeal, and it holds until the largest single operation stops fitting on the node — at which point the same table, unchanged, can be handed to a cluster. Nothing about the data or its layout changes in that transition, which is exactly the property that makes starting small a low-risk decision rather than a bet. The write path stays the same in either regime, which is what keeps the eventual change cheap. The executor changes; the contract does not. Verify that once on a copy and the eventual migration is a scheduling decision.