Skip to content

GH-3795: Preserve physical leaf identity for duplicate primitive roots - #3796

Open
ErikBPF wants to merge 1 commit into
apache:masterfrom
ErikBPF:fix/duplicate-root-physical-identity
Open

ErikBPF wants to merge 1 commit into
apache:masterfrom
ErikBPF:fix/duplicate-root-physical-identity

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Sep 16, 2026

Copy link
Copy Markdown

Rationale for this change

Closes #3795.

Duplicate primitive root names must not let name-path lookup substitute a different physical leaf's type, pages, or indexes. The consumer reproduction and standalone fixture source are in apache/datafusion-comet#5964: apache/datafusion-comet#5964.

What changes are included in this PR?

  • Resolve footer metadata using schema leaf ordinal; validate row-group column count and available physical path/type before path construction and crypto registration.
  • Preserve the first chunk and matching index for duplicate primitive root paths in ordinary and filtered row-group reads.
  • Retain empty-schema projections and lazy column-key metadata decryption.
  • Add 13 JUnit 5/AssertJ regression tests with generated fixtures, distinct page layouts, partial indexed reads, malformed controls, and encryption controls.

This targets master at 7a9665a392551143548477f733c9321c4775b32e. The initial investigation used parquet-java 1.16.0; this port preserves master's writer-version handling and uses its Java 17/JUnit 5 toolchain. No release pin or Java 8 compatibility claim is made.

Are these changes tested?

On JDK 17, Maven 3.9.11, and required Thrift 0.24.0:

export MAVEN_OPTS='-Xmx2g -XX:ActiveProcessorCount=2'
export JAVA_TOOL_OPTIONS='-Xmx2g -XX:ActiveProcessorCount=2'
timeout 600 ./mvnw -B -ntp -pl parquet-hadoop -am spotless:check package \
  -Dtest=TestDuplicateRootIdentity,TestParquetMetadataConverter,TestColumnIndexFiltering,TestEncryptionOptions \
  -Dsurefire.failIfNoSpecifiedTests=false

The equivalent Maven 3.9.11 invocation passed in 67 seconds: 112 tests, zero failures/errors/skips (metadata 74, index filtering 24, duplicate identity 13, encryption options 1). Spotless and normal lifecycle checks passed. The selected module's prerequisite modules were built; their full test suites and the full repository reactor were not run.

Earlier exact-release consumer validation passed Spark's duplicate suite (8 tests) and V1/V2 filter suites (101 tests), but that result is not validation of this current-master port or a published downstream dependency.

Are there any user-facing changes?

Duplicate primitive root selection retains the first physical chunk and corresponding index rather than overwriting it with a later namesake. Invalid footer count/path/type combinations fail with ParquetDecodingException. No general nested/encrypted duplicate support is claimed. A downstream Spark fix still needs coordinated descriptor/filter handling and a normally published Parquet artifact; this PR alone does not claim to fix every consumer path.

Prevent path-keyed lookups from mixing primitive types, pages, and indexes from different physical roots. Validate footer ordinal integrity before using column metadata while retaining lazy column-key decryption.

Closes apache#3795
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve physical leaf identity for duplicate primitive root paths

1 participant