Skip to content

[GH-3369] Preserve declared coordinate layouts in WKB output - #3381

Open
jiayuasu wants to merge 40 commits into
apache:masterfrom
jiayuasu:fix/wkb-output-dimensions
Open

jiayuasu wants to merge 40 commits into
apache:masterfrom
jiayuasu:fix/wkb-output-dimensions

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Sep 18, 2026

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Yes.

Is this PR related to a ticket?

Follow-up to #3369. Builds on merged #3374, #3377, and #3378. The diff against master contains only the WKB output changes.

org.datasyslab:jts-io-patch:1.21.0-datasyslab-2 is published on Maven Central.

What changes were proposed in this PR?

ST_AsBinary, ST_AsEWKB, and ST_AsHEXEWKB now preserve M for measured geometries from any source, including WKT, and retain WKB-declared Z/M layouts for empty points, lines, and polygons and NaN ordinates. This applies to Spark, Flink, and Snowflake, which use the same common output helpers. They use the isolated WKB writer's opt-in preservation mode instead of choosing dimensions from the first coordinate. A finite Z later in a line is no longer dropped, and mixed GeometryCollection members keep their individual layouts.

Ordinary XY output, byte order, and SRID behavior remain intact. The public explicit-dimension writer helper keeps its stock JTS return type and behavior. GeoParquet output and explicit 2D adapter writers are unchanged. The GeoParquet follow-up is tracked in #3383. Collections with no members lack sequence metadata and are written as XY.

How was this patch tested?

At 05bbe736951d81b4b57ff1a2859677db4ded5d62, all eight WkbOutputTest tests and the Spark 3.5 / Scala 2.12 function tests passed against the published IO2 artifact (245 Scala test executions, including WKB output after shuffle/cache and null inputs). Formatting and commit hooks passed.

Six of eight common tests fail against master's old output helpers. The measured-WKT regression specifically detects the missing M flag. Added coverage checks measured WKT before and after serialization, and MultiPoint, MultiLineString, and MultiPolygon output with an empty member across WKB, EWKB, and both HEXEWKB byte orders.

After synchronizing with master and merged #3377, the combined stack at 6881691d278e27bb160ee6be3f4fbd671e9b0192 passed 390 focused common geometry/WKB tests against the published org.datasyslab:jts-io-patch:1.21.0-datasyslab-2.

Validated the combined Sedona stack at 76f59c947971e06657d01229a4f940863d3e5253 against org.datasyslab:jts-io-patch:1.21.0-datasyslab-2-SNAPSHOT, built from merged JTS commit 1a382cc402f495adf1ea39c4905cafbac491e40f:

  • Full common suite: 1,396 passed.
  • Spark 3.5.0/Scala 2.12 and Spark 4.1.1/Scala 2.13: 324 selected Java and Scala tests passed per profile, covering SQL functions, UDT, collection and reader suites.
  • Stock PySpark 3.5.0 and 4.1.1: 96 native/fallback constructor tests, 144 primitive WKB/EWKB checks, and 18 multipart SQL round-trip checks passed.
  • Spark 4.1 checks exercised two separate executor JVMs. Its bundled JTS 1.20 jar remained unchanged, and ST_GeneratePoints stayed XY.

The published 1.21.0-datasyslab-2 artifact passes 32 isolated-artifact tests against stock JTS 1.20. All nine implementation class files are byte-for-byte identical to the tested snapshot; public downloads and signatures were verified. The full CI matrix and MySQL/Docker constructor suite were not part of these local checks.

Did this PR include necessary documentation updates?

Updated all nine Spark, Flink, and Snowflake output function pages to describe M preservation and the zero-member collection limit. Removed the stale Flink claim that M is ignored and added the measured-output change to the 2.0.0 release notes.

Signed-off-by: Jia Yu <jiayu@apache.org>
Signed-off-by: Jia Yu <jiayu@apache.org>
@jiayuasu
jiayuasu marked this pull request as ready for review September 21, 2026 07:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant