[SPARK-58949][CORE] Use unsigned UTF-8 byte ordering for Variant object fields

[peterxcli]

What changes were proposed in this pull request?

This PR fixes SPARK-58949 by:

• sorting newly written Variant object fields by unsigned lexicographic UTF-8 bytes, as required by the Variant encoding specification;
• using binary search for object lookup at every object size and comparing UTF-8-encoded keys using unsigned byte ordering;
• retrying lookup with Java UTF-16 order when needed so values written by older Spark versions remain readable; and
• accepting both canonical UTF-8 order and legacy UTF-16 order during schema validation, while preserving the established schema field order.

This follows the compatibility direction discussed in apache/parquet-java#3736.

Why are the changes needed?

The Variant specification orders object keys by unsigned UTF-8 bytes, but Spark used String.compareTo, which orders UTF-16 code units. These orders differ for some valid keys. For example, UTF-16 places U+10000 before U+FFFF, while unsigned UTF-8 places U+FFFF first.

As a result, Spark wrote non-canonical Variant objects and could miss fields when binary-searching canonical values produced by another implementation.

Does this PR introduce any user-facing change?

Yes. Newly written Variant objects use the specification's unsigned UTF-8 field order. Spark continues to read affected values written in the legacy UTF-16 order, and schema output keeps its existing field order.

How was this patch tested?

Added regressions for canonical and legacy object lookup, nested objects, schema_of_variant, and Parquet shredding-schema inference:

build/sbt \
  'catalyst/testOnly *VariantExpressionSuite -- -z "SPARK-58949"' \
  'sql/testOnly *VariantInferShreddingSuite -- -z "SPARK-58949"'

Both suites passed (1 test each). The affected modules also passed Java checkstyle and main/test scalastyle.

I also ran a temporary lookup microbenchmark on an Apple M4 with Zulu OpenJDK 21.0.6, comparing upstream/master at 9da9f8d with this patch at b3a050b. Each result is the median of three alternating JVM fork medians. Each fork used a fixed 2 GiB heap, 5 seconds of warmup per case, and 9 measured rounds of 5,000,000 lookups. Object construction was excluded. Lower is better.

Object / lookup	master (ns/op)	patch (ns/op)	Change
16 fields, ASCII present	193.7	94.6	-51.2%
16 fields, ASCII absent	180.3	87.2	-51.6%
256 fields, canonical present	102.2	140.6	+37.6%
256 fields, canonical absent	111.5	147.0	+31.8% (approx.)
256 fields, legacy fallback present	196.5	366.3	+86.4%
256 fields, legacy fallback absent	201.7	367.0	+82.0%

The correctness assertion for a canonical U+10000 lookup changes from false on master to true with this patch. Following parquet-java's implementation, each canonical binary-search probe decodes the metadata key and re-encodes it as UTF-8 instead of maintaining a separate raw-metadata comparator. This keeps the implementation simple but adds allocations: the measured 256-field canonical cases regress by 32-38%, and the legacy fallback cases regress by 82-86%. The 16-field cases improve by about 51% because they now use binary search instead of the previous linear scan. One canonical-absent fork was noisy, so that aggregate is marked approximate.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

[peterxcli]

cc @sunchao @cloud-fan @yadavay-amzn @steveloughran PTAL. Thanks!

[uros-b]

Thank you @peterxcli! cc @harshmotw-db for variant

[sunchao]

LGTM. Reviewed b3a050b3dc02e39c59893a016c8fde58758ad658 with five independent passes covering lookup correctness, encoding/writing, schema/shredding, compatibility/tests, and performance/interoperability. No actionable new P1/P2 findings.

The unsigned UTF-8 ordering matches the Variant specification, the UTF-16 fallback preserves reading of legacy Spark values, and schema normalization preserves the ordering required by schema merging.

Validation

• 2,144,172 standalone JVM lookup, writer, compatibility, and interoperability checks passed against the reviewed Java sources.
• Java/Python encodings matched byte-for-byte across 30 flat/nested fixtures; Python decoded both canonical and legacy-reordered values correctly.
• Focused schema/shredding probes passed, including mixed legacy/canonical inputs and partial projections. These used cached Spark dependencies, not a full Spark build.
• The patched reader passed the base-writer compatibility check. Current upstream/fork CI has 32 successful checks and one skipped check.

Full Spark suites and Parquet-file integration tests were not run locally.

Acknowledged caveats

The author's benchmarks report 32-38% slower canonical lookups and 82-86% slower legacy-fallback lookups for 256-field objects, alongside faster 16-field lookups. Those timings were not independently reproduced in this review. Also, older readers can still miss the affected keys in newly canonical objects; the compatibility guarantee here is that the patched reader continues to read legacy data.

[sunchao]

Merged to master / branch-4.x, thanks @peterxcli for the contribution and @uros-b for the review!

[steveloughran]

probably worth adding some high unicode variant strings to the variant test files in the parquet reference file repo apache/parquet-testing, so as to make sure everyone reads these consistently

[peterxcli]

probably worth adding some high unicode variant strings to the variant test files in the parquet reference file repo apache/parquet-testing, so as to make sure everyone reads these consistently

Great suggestions, sounds fair. will let you know whenever PR is ready.

[peterxcli]

Thanks everyone for reviewing!

[peterxcli]

@sunchao @steveloughran @uros-b I think this should also close https://issues.apache.org/jira/browse/SPARK-56637. wdyt?

[peterxcli]

probably worth adding some high unicode variant strings to the variant test files in the parquet reference file repo apache/parquet-testing, so as to make sure everyone reads these consistently

Great suggestions, sounds fair. will let you know whenever PR is ready.

@steveloughran apache/parquet-testing#126, the PR is ready. Looking forward to your review, thanks!