Validating GeoParquet Metadata in CI

This guide gives you a complete Python plus GitHub Actions recipe that inspects a GeoParquet file’s geo metadata — version, primary_column, CRS presence, and the bbox covering — and fails the build before the file is promoted, so encoding drift never reaches a production table.

Context and prerequisites

A GeoParquet file is only interoperable if its footer geo block is well-formed: the wrong version, a missing primary_column, a dropped CRS, or an absent covering bbox each silently breaks a downstream reader or defeats file skipping. Rather than discover that at query time, assert the contract in continuous integration. This page builds on the field definitions in the GeoParquet encoding standards topic area and the layout trade-offs in GeoParquet vs WKB column storage. You need pyarrow 17+ and pytest; no cloud credentials are required because validation runs on the file’s footer alone. The CRS assertions here complement the runtime enforcement in detecting CRS drift in ingestion pipelines.

Complete working solution

Two files: a reusable validator module and a pytest wrapper. The validator reads only the Parquet footer, so it is fast and safe to run on large files. It returns a list of human-readable failures; an empty list means the file conforms.

# geoparquet_gate.py
import json
import sys
import pyarrow.parquet as pq

REQUIRED_VERSION_PREFIX = "1.1"
EXPECTED_EPSG = 4326  # numeric EPSG literal for this project's canonical CRS


def read_geo_metadata(path):
    """Return the parsed 'geo' block from a Parquet file footer, or None."""
    kv = pq.ParquetFile(path).metadata.metadata or {}
    if b"geo" not in kv:
        return None
    return json.loads(kv[b"geo"].decode("utf-8"))


def validate_geoparquet(path):
    """Return a list of validation failures; empty list means the file is valid."""
    failures = []
    geo = read_geo_metadata(path)
    if geo is None:
        return [f"{path}: no 'geo' metadata block in footer"]

    version = geo.get("version", "")
    if not version.startswith(REQUIRED_VERSION_PREFIX):
        failures.append(f"{path}: version {version!r} is not {REQUIRED_VERSION_PREFIX}.x")

    primary = geo.get("primary_column")
    if not primary:
        failures.append(f"{path}: missing 'primary_column'")
        return failures  # nothing else is checkable without it

    col = geo.get("columns", {}).get(primary)
    if col is None:
        failures.append(f"{path}: primary_column {primary!r} absent from 'columns'")
        return failures

    crs = col.get("crs")
    if not crs:
        failures.append(f"{path}: CRS missing for column {primary!r}")
    else:
        code = crs.get("id", {}).get("code")
        if code != EXPECTED_EPSG:
            failures.append(f"{path}: CRS EPSG code {code} != {EXPECTED_EPSG}")

    covering = col.get("covering")
    if not covering or "bbox" not in covering:
        failures.append(f"{path}: no bbox 'covering' for column {primary!r}")

    if col.get("encoding") not in {"WKB", "point", "linestring", "polygon",
                                   "multipoint", "multilinestring", "multipolygon"}:
        failures.append(f"{path}: unexpected encoding {col.get('encoding')!r}")

    return failures


def main(paths):
    all_failures = []
    for p in paths:
        all_failures.extend(validate_geoparquet(p))
    if all_failures:
        print("GeoParquet validation FAILED:")
        for f in all_failures:
            print(f"  - {f}")
        return 1
    print(f"GeoParquet validation passed for {len(paths)} file(s).")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

# .github/workflows/geoparquet-gate.yml
name: GeoParquet metadata gate
on:
  pull_request:
    paths:
      - "data/**.parquet"
      - "geoparquet_gate.py"
      - "tests/test_geoparquet_gate.py"
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: python -m pip install "pyarrow>=17.0.0" "pytest>=8.0" "geopandas>=1.0.1" "shapely>=2.0.4"
      - name: Validate GeoParquet metadata
        run: pytest tests/test_geoparquet_gate.py -v

Step-by-step walkthrough

Footer-only read. pq.ParquetFile(path).metadata.metadata returns the file’s key-value metadata without scanning any row groups, so validation cost is independent of file size.
Presence gate. A missing b"geo" key means the file is not GeoParquet at all — the first and most common drift, caught immediately.
Version gate. version.startswith("1.1") rejects files written against an older spec whose covering semantics differ. Tighten or loosen the prefix to match your promotion policy.
Primary column gate. Everything else keys off primary_column; if it is absent or points at a column not in columns, the function returns early with a precise message rather than raising.
CRS gate. It asserts both that a crs object exists and that its EPSG code equals the numeric literal 4326. Dropping the code check but keeping the presence check is a valid looser policy for multi-CRS datasets.
Covering gate. A missing covering.bbox means engines cannot use per-row bounds for file skipping — a performance regression that this catches before merge.
Exit code. main returns 1 on any failure so the CI step fails; the pytest wrapper below turns each file into an assertion for readable reports.

# tests/test_geoparquet_gate.py
import glob
import pytest
from geoparquet_gate import validate_geoparquet

PARQUET_FILES = sorted(glob.glob("data/**/*.parquet", recursive=True))


@pytest.mark.parametrize("path", PARQUET_FILES or [pytest.param(None, marks=pytest.mark.skip)])
def test_geoparquet_metadata_is_valid(path):
    failures = validate_geoparquet(path)
    assert not failures, "\n".join(failures)

Common errors and fixes

Error	Cause	Fix
`no 'geo' metadata block in footer`	File written with a plain Parquet writer, not GeoParquet	Rewrite via GeoPandas `to_parquet(schema_version="1.1.0")`
`CRS EPSG code None != 4326`	GeoDataFrame written without a CRS set	Set `crs=4326` before writing; reject undefined-CRS inputs upstream
`no bbox 'covering'`	`write_covering_bbox=True` was omitted at write time	Re-export with the covering flag enabled
`version '1.0.0' is not 1.1.x`	File produced by an older GeoParquet writer	Upgrade the writer or relax `REQUIRED_VERSION_PREFIX` deliberately
Job passes but skips every file	`data/**.parquet` glob matched nothing	Confirm the path filter and that artifacts are checked out

Verification

Prove the gate before trusting it: build one conforming file and one broken file, then confirm the validator disagrees about them.

import geopandas
import pyarrow as pa
import pyarrow.parquet as pq
from shapely.geometry import Point
from geoparquet_gate import validate_geoparquet

good = geopandas.GeoDataFrame(
    {"id": [1]}, geometry=[Point(-122.4, 37.8)], crs=4326)
good.to_parquet("good.parquet", schema_version="1.1.0", write_covering_bbox=True)

pq.write_table(pa.table({"id": [1]}), "bad.parquet")  # no geo metadata at all

assert validate_geoparquet("good.parquet") == []
assert validate_geoparquet("bad.parquet")  # non-empty -> fails the build
print("gate distinguishes conforming from non-conforming files")

CI gate flow

Wiring this gate into promotion closes the loop opened by the GeoParquet encoding standards topic area within the Spatial Lakehouse Fundamentals & Architecture section, and it pairs naturally with the schema checks around Iceberg spatial type support. Keep the assertions aligned with the field names in the GeoParquet specification and the CRS model in the OGC Simple Features Access standard so the gate stays correct as the spec evolves.

Validating GeoParquet Metadata in CI

Related in this section