Validating GeoParquet Metadata in CI

This guide gives you a complete Python plus GitHub Actions recipe that inspects a GeoParquet file’s geo metadata — version, primary_column, CRS presence, and the bbox covering — and fails the build before the file is promoted, so encoding drift never reaches a production table.

Context and prerequisites

A GeoParquet file is only interoperable if its footer geo block is well-formed: the wrong version, a missing primary_column, a dropped CRS, or an absent covering bbox each silently breaks a downstream reader or defeats file skipping. Rather than discover that at query time, assert the contract in continuous integration. This page builds on the field definitions in the GeoParquet encoding standards topic area and the layout trade-offs in GeoParquet vs WKB column storage. You need pyarrow 17+ and pytest; no cloud credentials are required because validation runs on the file’s footer alone. The CRS assertions here complement the runtime enforcement in detecting CRS drift in ingestion pipelines.

Complete working solution

Two files: a reusable validator module and a pytest wrapper. The validator reads only the Parquet footer, so it is fast and safe to run on large files. It returns a list of human-readable failures; an empty list means the file conforms.

# geoparquet_gate.py
import json
import sys
import pyarrow.parquet as pq

REQUIRED_VERSION_PREFIX = "1.1"
EXPECTED_EPSG = 4326  # numeric EPSG literal for this project's canonical CRS


def read_geo_metadata(path):
    """Return the parsed 'geo' block from a Parquet file footer, or None."""
    kv = pq.ParquetFile(path).metadata.metadata or {}
    if b"geo" not in kv:
        return None
    return json.loads(kv[b"geo"].decode("utf-8"))


def validate_geoparquet(path):
    """Return a list of validation failures; empty list means the file is valid."""
    failures = []
    geo = read_geo_metadata(path)
    if geo is None:
        return [f"{path}: no 'geo' metadata block in footer"]

    version = geo.get("version", "")
    if not version.startswith(REQUIRED_VERSION_PREFIX):
        failures.append(f"{path}: version {version!r} is not {REQUIRED_VERSION_PREFIX}.x")

    primary = geo.get("primary_column")
    if not primary:
        failures.append(f"{path}: missing 'primary_column'")
        return failures  # nothing else is checkable without it

    col = geo.get("columns", {}).get(primary)
    if col is None:
        failures.append(f"{path}: primary_column {primary!r} absent from 'columns'")
        return failures

    crs = col.get("crs")
    if not crs:
        failures.append(f"{path}: CRS missing for column {primary!r}")
    else:
        code = crs.get("id", {}).get("code")
        if code != EXPECTED_EPSG:
            failures.append(f"{path}: CRS EPSG code {code} != {EXPECTED_EPSG}")

    covering = col.get("covering")
    if not covering or "bbox" not in covering:
        failures.append(f"{path}: no bbox 'covering' for column {primary!r}")

    if col.get("encoding") not in {"WKB", "point", "linestring", "polygon",
                                   "multipoint", "multilinestring", "multipolygon"}:
        failures.append(f"{path}: unexpected encoding {col.get('encoding')!r}")

    return failures


def main(paths):
    all_failures = []
    for p in paths:
        all_failures.extend(validate_geoparquet(p))
    if all_failures:
        print("GeoParquet validation FAILED:")
        for f in all_failures:
            print(f"  - {f}")
        return 1
    print(f"GeoParquet validation passed for {len(paths)} file(s).")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

# .github/workflows/geoparquet-gate.yml
name: GeoParquet metadata gate
on:
  pull_request:
    paths:
      - "data/**.parquet"
      - "geoparquet_gate.py"
      - "tests/test_geoparquet_gate.py"
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: python -m pip install "pyarrow>=17.0.0" "pytest>=8.0" "geopandas>=1.0.1" "shapely>=2.0.4"
      - name: Validate GeoParquet metadata
        run: pytest tests/test_geoparquet_gate.py -v

Step-by-step walkthrough

1. Footer-only read. pq.ParquetFile(path).metadata.metadata returns the file’s key-value metadata without scanning any row groups, so validation cost is independent of file size.
2. Presence gate. A missing b"geo" key means the file is not GeoParquet at all — the first and most common drift, caught immediately.
3. Version gate. version.startswith("1.1") rejects files written against an older spec whose covering semantics differ. Tighten or loosen the prefix to match your promotion policy.
4. Primary column gate. Everything else keys off primary_column; if it is absent or points at a column not in columns, the function returns early with a precise message rather than raising.
5. CRS gate. It asserts both that a crs object exists and that its EPSG code equals the numeric literal 4326. Dropping the code check but keeping the presence check is a valid looser policy for multi-CRS datasets.
6. Covering gate. A missing covering.bbox means engines cannot use per-row bounds for file skipping — a performance regression that this catches before merge.
7. Exit code. main returns 1 on any failure so the CI step fails; the pytest wrapper below turns each file into an assertion for readable reports.

# tests/test_geoparquet_gate.py
import glob
import pytest
from geoparquet_gate import validate_geoparquet

PARQUET_FILES = sorted(glob.glob("data/**/*.parquet", recursive=True))


@pytest.mark.parametrize("path", PARQUET_FILES or [pytest.param(None, marks=pytest.mark.skip)])
def test_geoparquet_metadata_is_valid(path):
    failures = validate_geoparquet(path)
    assert not failures, "\n".join(failures)

Which Checks Belong in CI and Which Belong in the Pipeline

Not every validation belongs in a pull request. Splitting them by what they can actually see keeps the build fast and keeps the production check meaningful.

The left-hand checks answer “did we configure the writer correctly”, and a tiny fixture answers that as well as a terabyte would. The right-hand checks answer “did this particular write behave”, and they cannot be moved left because a fixture cannot exhibit a real extent or a real statistics limit. Teams that put everything in CI end up with a slow build that still misses production defects; teams that put everything in the pipeline discover configuration errors after the data is written.

Failing Loudly Without Blocking Delivery

A metadata gate that halts an hourly load because one file is imperfect will be switched off. Grade the responses instead.

The blocking set should be small enough that a failure is unambiguous and always worth a human’s attention. Everything else accumulates as a metric, and the metric is what tells you whether a warning is a one-off or a trend — which is the only basis on which a warning should ever be promoted to a block.

Common errors and fixes

Verification

Prove the gate before trusting it: build one conforming file and one broken file, then confirm the validator disagrees about them.

import geopandas
import pyarrow as pa
import pyarrow.parquet as pq
from shapely.geometry import Point
from geoparquet_gate import validate_geoparquet

good = geopandas.GeoDataFrame(
    {"id": [1]}, geometry=[Point(-122.4, 37.8)], crs=4326)
good.to_parquet("good.parquet", schema_version="1.1.0", write_covering_bbox=True)

pq.write_table(pa.table({"id": [1]}), "bad.parquet")  # no geo metadata at all

assert validate_geoparquet("good.parquet") == []
assert validate_geoparquet("bad.parquet")  # non-empty -> fails the build
print("gate distinguishes conforming from non-conforming files")

CI gate flow

Wiring this gate into promotion closes the loop opened by the GeoParquet encoding standards topic area within the Spatial Lakehouse Fundamentals & Architecture section, and it pairs naturally with the schema checks around Iceberg spatial type support. Keep the assertions aligned with the field names in the GeoParquet specification and the CRS model in the OGC Simple Features Access standard so the gate stays correct as the spec evolves.

Extending the Gate to a Whole Catalogue

The same validation function that guards one file is more valuable pointed at everything. Run it as a weekly sweep over every spatial table’s most recent files, and publish the result as a short table of contract violations ordered by data volume behind each. The output tends to be dominated by tables nobody has touched in a year, written by a pipeline that predates the convention — which is exactly the set that will surprise someone during an incident. Fixing them is usually a rewrite of the most recent partition plus a table-property change, and doing it proactively converts an unknown risk into a scheduled afternoon. Pair the sweep with the table-level audit described in Iceberg spatial type support so file-level and catalog-level contracts are checked by the same job and reported together.

Treat the gate as permanent infrastructure rather than as a migration aid. Metadata regressions do not arrive once and get fixed; they arrive whenever a dependency is upgraded, a pipeline is rewritten by someone new, or a table is recreated during an incident under time pressure. A check that runs on every write costs nothing on the days it passes, and on the day it fires it saves a rewrite of everything downstream.

The measure of a good gate is that nobody talks about it. It should be fast enough that nobody proposes skipping it, precise enough that a failure is never argued with, and quiet enough that a green build carries no information. Everything else — the trend metrics, the catalogue sweep, the warnings — belongs on a dashboard that somebody reviews deliberately rather than in the path of a deployment.