Mapping GeoPandas DataFrames to Arrow Schemas

This recipe converts a GeoPandas GeoDataFrame into a PyArrow table under an explicit geometry-as-WKB schema that carries CRS metadata and tolerates null geometries, producing bytes ready for PyIceberg or Parquet writes, and verifies a lossless round-trip back to GeoPandas.

Context and prerequisites

A GeoDataFrame’s geometry column holds Shapely objects, which are not an Arrow-native type; to hand geometry to Iceberg, Delta, or Parquet you must serialize it to a stable binary encoding and attach the CRS as sidecar metadata, because Arrow itself has no geometry type or coordinate-reference concept. This page sits in the DataFrame mapping strategies topic area and is the practical counterpart to the GeoParquet encoding standards that formalize how that metadata is written. You need GeoPandas 1.0+, Shapely 2.x, and PyArrow 15+ (pip install "geopandas>=1.0" "pyarrow>=15"). Two encodings matter: plain WKB in an Arrow binary field (maximally portable, the choice for PyIceberg), and GeoArrow (native coordinate arrays, faster but newer). This recipe uses WKB with an explicit schema and shows the GeoArrow shortcut at the end.

Complete working solution

The mapping builds an Arrow schema by hand, serializes each geometry with Shapely’s vectorized to_wkb (passing nulls through as Arrow nulls), and stamps the CRS and encoding into the schema-level metadata so downstream readers can reconstruct the geometry faithfully.

# geodf_to_arrow.py
# GeoPandas 1.0+, Shapely 2.x, PyArrow 15+
import json

import geopandas as gpd
import numpy as np
import pyarrow as pa
from shapely import to_wkb, from_wkb
from shapely.geometry import Point, Polygon

# --- sample GeoDataFrame with a null geometry and non-geometry columns ---
gdf = gpd.GeoDataFrame(
    {
        "station_id": [101, 102, 103],
        "name": ["alpha", "beta", "gamma"],
        "geometry": [
            Point(-122.42, 37.77),
            Polygon([(-121, 38), (-120, 38), (-120, 39), (-121, 38)]),
            None,                               # null geometry must survive
        ],
    },
    crs="EPSG:4326",
)


def geodf_to_arrow(gdf: gpd.GeoDataFrame, geom_col: str = "geometry") -> pa.Table:
    """Map a GeoDataFrame to an Arrow table with geometry as WKB + CRS metadata."""
    geom = gdf[geom_col]
    epsg = geom.crs.to_epsg() if geom.crs else None
    if epsg is None:
        raise ValueError("GeoDataFrame has no EPSG-resolvable CRS; set one first")

    # Vectorized WKB; None geometries become None -> Arrow null.
    mask = geom.isna().to_numpy()
    wkb_vals = to_wkb(geom.to_numpy(), output_dimension=2)
    wkb_vals = np.where(mask, None, wkb_vals)
    geom_arr = pa.array(list(wkb_vals), type=pa.binary())

    # Non-geometry columns straight from pandas.
    fields = [pa.field("geom_wkb", pa.binary(), nullable=True)]
    arrays = [geom_arr]
    for col in gdf.columns:
        if col == geom_col:
            continue
        arrays.append(pa.array(gdf[col].to_numpy()))
        fields.append(pa.field(col, arrays[-1].type))

    # GeoParquet-style geo metadata on the geom field and the schema.
    geo_meta = {
        "encoding": "WKB",
        "crs_epsg": epsg,
        "geometry_type": "mixed",
    }
    field_meta = {b"ARROW:extension:name": b"geoarrow.wkb",
                  b"geo": json.dumps(geo_meta).encode()}
    fields[0] = fields[0].with_metadata(field_meta)

    schema = pa.schema(fields, metadata={
        b"geo": json.dumps({"version": "1.1.0",
                            "primary_column": "geom_wkb",
                            "columns": {"geom_wkb": geo_meta}}).encode()
    })
    return pa.Table.from_arrays(arrays, schema=schema)


table = geodf_to_arrow(gdf)
print(table.schema)
print("geo metadata:", table.schema.metadata[b"geo"].decode())

Step-by-step walkthrough

1. Resolve the CRS to an EPSG integer. geom.crs.to_epsg() collapses a full PROJ CRS to the numeric authority code (here 4326). Arrow metadata is bytes-only, so a compact integer is far more portable than an embedded WKT2 string, and it fails loudly with ValueError when the frame has no CRS — an unset CRS is the single most common source of silent reprojection bugs downstream.

2. Vectorize WKB serialization. Shapely 2’s to_wkb operates on the whole NumPy array of geometries at once rather than per-row Python calls, which is roughly an order of magnitude faster on large frames. output_dimension=2 forces XY output, dropping stray Z values that would otherwise bloat the bytes.

3. Preserve nulls explicitly. to_wkb returns None for a None geometry, but mixing None into a NumPy object array is fragile, so np.where(mask, None, wkb_vals) normalizes it and pa.array(..., type=pa.binary()) maps those to true Arrow nulls. A null geometry stays null rather than becoming empty bytes, which matters because empty WKB and null are distinct states.

4. Copy the attribute columns. Non-geometry columns convert through pa.array(gdf[col].to_numpy()), letting Arrow infer int64, string, etc. Each field’s inferred type is recorded so the schema is fully explicit rather than left to Table.from_pandas guesswork.

5. Attach GeoParquet-compatible metadata. The geometry field carries ARROW:extension:name = geoarrow.wkb so GeoArrow-aware readers recognize it, and a geo JSON blob records the encoding, EPSG, and geometry type. The schema-level geo key mirrors the GeoParquet 1.1 metadata layout with version, primary_column, and per-column entries, so a pyarrow.parquet.write_table of this object is a valid GeoParquet file.

6. Build the table from explicit arrays. pa.Table.from_arrays(arrays, schema=schema) binds columns to the hand-built schema, guaranteeing column order and the WKB binary type instead of relying on inference. This is the object you pass to a PyIceberg table.append(...) or pq.write_table(...).

If you want native GeoArrow coordinate storage instead of WKB, GeoPandas 1.0 exposes gdf.to_arrow(geometry_encoding="geoarrow") directly, which yields interleaved coordinate arrays and richer metadata; WKB remains the safer default for PyIceberg today because the geometry lands as an ordinary binary column that any engine can store. The trade-offs between the two are laid out in GeoParquet vs WKB column storage trade-offs.

Common errors and fixes

Verification

Round-trip the Arrow table back into a GeoDataFrame and assert geometry, CRS, and null all survive. Equality on the WKB bytes is the strict test.

# verify_roundtrip.py
import json
import geopandas as gpd
import pyarrow as pa
from shapely import from_wkb

meta = json.loads(table.schema.metadata[b"geo"].decode())
epsg = meta["columns"]["geom_wkb"]["crs_epsg"]
assert epsg == 4326

wkb_back = table.column("geom_wkb").to_pylist()
geoms = [from_wkb(b) if b is not None else None for b in wkb_back]

gdf_back = gpd.GeoDataFrame(
    {"station_id": table.column("station_id").to_pylist(),
     "name": table.column("name").to_pylist(),
     "geometry": geoms},
    crs=epsg,
)

assert gdf_back.geometry.iloc[2] is None          # null preserved
assert gdf_back.geometry.iloc[0].equals(gpd.GeoSeries([__import__("shapely").geometry.Point(-122.42, 37.77)]).iloc[0])
assert gdf_back.crs.to_epsg() == 4326
print("round-trip OK:", gdf_back.geometry.tolist())

When all three assertions pass — the null stays None, the point geometry compares equal, and the CRS resolves back to 4326 — the mapping is proven lossless and safe to persist. From here the same Arrow table feeds a concurrent load via async catalog writes with PyIceberg and asyncio, or an efficient bulk ingest as described in reading shapefiles into PyIceberg dataframes efficiently.

The load-bearing decisions are resolving the CRS to a numeric EPSG code before serialization and keeping nulls distinct from empty geometries — get those right and the WKB Arrow table interoperates cleanly with every lakehouse engine. For the encoding standard this mapping conforms to, see GeoParquet encoding standards, and the GeoPandas Arrow documentation for the native GeoArrow path.

The Metadata That Makes the Output Self-Describing

Converting the geometry to a binary column is only half the job; the other half is attaching the metadata that lets a reader interpret those bytes without asking anyone.

Attach it to the schema, not to the file, so it survives every operation that preserves the schema — a concatenation, a chunked write, a re-partition. Attaching it at write time only means a chunked write can produce files where the first has metadata and the rest do not, which is a genuinely confusing state to debug.

Write the CRS as PROJJSON rather than as an EPSG code. The code alone depends on the reader having the same registry version and does not pin the datum transformation path, which is the mechanism behind the metre-scale offsets described in the CRS guidance. The full definition is a few hundred bytes and removes the dependency entirely.

Validating the Schema Before the Write

The third check catches a specific and nasty bug: a pipeline that derives bounding boxes and then reprojects, leaving boxes that describe the geometry’s previous position. Every value is present, every type is right, and the table’s data skipping now excludes exactly the files a query should read. Asserting coverage takes one vectorised comparison and rules it out permanently.

Run all three assertions inside the conversion function itself rather than as a separate step. A conversion that can only produce valid output is a much stronger guarantee than a conversion followed by a validation somebody may forget to call, and the cost is a few milliseconds per batch. The result is a conversion that either produces a correct, self-describing table or raises — with no third outcome in which it produces something that merely looks right. That property is what makes the conversion safe to call from a dozen pipelines without reviewing each of them. Callers get correctness by default rather than by discipline. The schema definition then becomes the single artefact worth reviewing carefully, and everything downstream inherits its guarantees. Review it once, thoroughly, and the review holds for every table written through it.