Skip to content

[GH-3369] Preserve WKB dimensions in file and adapter readers - #3378

Merged
jiayuasu merged 34 commits into
apache:masterfrom
jiayuasu:fix/wkb-ingress-dimensions
Sep 21, 2026
Merged

jiayuasu merged 34 commits into
apache:masterfrom
jiayuasu:fix/wkb-ingress-dimensions

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Sep 16, 2026

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Yes.

Is this PR related to a ticket?

Follow-up to #3369. Builds on merged #3374 and #3377.

org.datasyslab:jts-io-patch:1.21.0-datasyslab-2 is published on Maven Central.

WKB output is reviewed separately in #3381.

What changes were proposed in this PR?

Use IO2's WKBReader.forDeclaredDimensions() in both GeoParquet conversion paths, GeoPackage, the Python/JVM adapter, and lazy JTS geography materialization. Empty and all-NaN layouts then survive Sedona serialization consistently with SQL WKB constructors.

Preserve metadata/header SRIDs, Python framing and userData, and geography caching. The separate S2 geography reader is unchanged. This changes ingestion; GeoParquet output is a separate format-aware concern.

How was this patch tested?

Reader regressions cover each integration. The GeoParquet primitive reader also alternates populated NaN-Z and XY points, checking coordinates and dimensions after GeometryUDT serialization.

At 0f26a29afa9583897d59856256631e5dd8087d32, 36 WKBGeography tests and 10 Scala reader/adapter tests passed on Spark 3.5 / Scala 2.12 against the published IO2 artifact. The new NaN-Z test fails with the master GeoParquet reader because dimension 3 becomes 2. Formatting and commit hooks passed.

After synchronizing with master and merged #3377, the combined stack at 6881691d278e27bb160ee6be3f4fbd671e9b0192 passed 390 focused common geometry/WKB tests against the published org.datasyslab:jts-io-patch:1.21.0-datasyslab-2.

Validated the combined Sedona stack at 76f59c947971e06657d01229a4f940863d3e5253 against org.datasyslab:jts-io-patch:1.21.0-datasyslab-2-SNAPSHOT, built from merged JTS commit 1a382cc402f495adf1ea39c4905cafbac491e40f:

  • Full common suite: 1,396 passed.
  • Spark 3.5.0/Scala 2.12 and Spark 4.1.1/Scala 2.13: 324 selected Java and Scala tests passed per profile, covering SQL functions, UDT, collection and reader suites.
  • Stock PySpark 3.5.0 and 4.1.1: 96 native/fallback constructor tests, 144 primitive WKB/EWKB checks, and 18 multipart SQL round-trip checks passed.
  • Spark 4.1 checks exercised two separate executor JVMs. Its bundled JTS 1.20 jar remained unchanged, and ST_GeneratePoints stayed XY.

The published 1.21.0-datasyslab-2 artifact passes 32 isolated-artifact tests against stock JTS 1.20. All nine implementation class files are byte-for-byte identical to the tested snapshot; public downloads and signatures were verified. The full CI matrix and MySQL/Docker constructor suite were not part of these local checks.

Did this PR include necessary documentation updates?

Added short notes to the GeoParquet and GeoPackage reader pages explaining dimension preservation and linking to the ST_Collect guidance for mixed layouts.

@jiayuasu
jiayuasu marked this pull request as ready for review September 21, 2026 06:10
@jiayuasu jiayuasu linked an issue Sep 21, 2026 that may be closed by this pull request
1 task
@jiayuasu jiayuasu added this to the sedona-2.0.0 milestone Sep 21, 2026
@jiayuasu
jiayuasu merged commit dc89c83 into apache:master Sep 21, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GeoPandas: leading-null GeoSeries input loses Z dimensions

1 participant