Conversation
Prevent path-keyed lookups from mixing primitive types, pages, and indexes from different physical roots. Validate footer ordinal integrity before using column metadata while retaining lazy column-key decryption. Closes apache#3795
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Closes #3795.
Duplicate primitive root names must not let name-path lookup substitute a different physical leaf's type, pages, or indexes. The consumer reproduction and standalone fixture source are in apache/datafusion-comet#5964: apache/datafusion-comet#5964.
What changes are included in this PR?
This targets master at
7a9665a392551143548477f733c9321c4775b32e. The initial investigation used parquet-java 1.16.0; this port preserves master's writer-version handling and uses its Java 17/JUnit 5 toolchain. No release pin or Java 8 compatibility claim is made.Are these changes tested?
On JDK 17, Maven 3.9.11, and required Thrift 0.24.0:
The equivalent Maven 3.9.11 invocation passed in 67 seconds: 112 tests, zero failures/errors/skips (metadata 74, index filtering 24, duplicate identity 13, encryption options 1). Spotless and normal lifecycle checks passed. The selected module's prerequisite modules were built; their full test suites and the full repository reactor were not run.
Earlier exact-release consumer validation passed Spark's duplicate suite (8 tests) and V1/V2 filter suites (101 tests), but that result is not validation of this current-master port or a published downstream dependency.
Are there any user-facing changes?
Duplicate primitive root selection retains the first physical chunk and corresponding index rather than overwriting it with a later namesake. Invalid footer count/path/type combinations fail with
ParquetDecodingException. No general nested/encrypted duplicate support is claimed. A downstream Spark fix still needs coordinated descriptor/filter handling and a normally published Parquet artifact; this PR alone does not claim to fix every consumer path.