fix(parquet): decode DECIMAL page statistics by physical type

[Joob1n]

Purpose

Linked issue: close #345

ColumnIndexFilter::CompareEncodedWithLiteral inferred the physical type of DECIMAL page statistics from the length of the encoded min/max — 4 bytes meant INT32, 8 bytes meant INT64 (both decoded with a native-endian memcpy), anything else was decoded as big-endian two's complement.

Parquet allows DECIMAL as FIXED_LEN_BYTE_ARRAY for any precision, with length ceil((precision * log2(10) + 1) / 8). Precision 7-9 yields exactly 4 bytes and precision 17-18 exactly 8, so those columns took the INT32/INT64 branch and had their big-endian bounds read little-endian. Page min 1.00 (unscaled 100, stored as 00 00 00 64) read back as 1,677,721,600, so col < 5.00 answered "no" and dropped a page that does match. Page-index filtering is pruning, not filtering, so nothing downstream brings those rows back.

This passes the ::parquet::SchemaDescriptor down from FileReaderWrapper::CalculateFilteredRowRanges through ColumnIndexFilter and dispatches on the column's physical type, the way parquet_stats_extractor.cpp already does for row-group statistics. INT32/INT64 keep the plain little-endian decode; FIXED_LEN_BYTE_ARRAY and BYTE_ARRAY go through Decimal::FromUnscaledBytes. When no schema is available the comparison is skipped, so no page is pruned.

Paimon's own writers use INT32 for precision <= 9 and INT64 for precision <= 18, so this only reaches users through files written elsewhere: Arrow with the default store_decimal_as_integer = false, Spark with spark.sql.parquet.writeLegacyFormat = true, or Hive.

Tests

New cases in src/paimon/format/parquet/column_index_filter_test.cpp:

• ColumnIndexFilterTest.DecimalFixedLenByteArrayPages — writes DECIMAL(9,2) and DECIMAL(18,2) columns as FIXED_LEN_BYTE_ARRAY with the page index enabled, asserts the writer really produced FLBA of length 4 and 8, then checks that <, > and = keep exactly the pages holding matching rows and prune the rest. Both cases fail on the previous decoding logic and pass with this change.
• ColumnIndexFilterTest.DecimalWithoutSchemaKeepsAllPages — with no schema descriptor, no page is pruned.

Run locally (macOS arm64, Debug, -Dfmt_SOURCE=BUNDLED):

• ./build/debug/paimon-parquet-format-test --gtest_filter='ColumnIndexFilterTest.*:RowRangesTest.*' — 36/36 pass.
• Full paimon-parquet-format-test (225 cases, run one per process): 147 pass, 78 fail. Those 78 fail identically on unmodified main — they abort in fixture setup at arrow/ipc/json_simple.cc:156 DCHECK(*out) inside ArrayFromJSON, which is unrelated to this change and Debug-only.

API and Format

No. ColumnIndexFilter::CalculateRowRanges gains a const ::parquet::SchemaDescriptor* parameter, but the class is internal to src/paimon/format/parquet/ and is not part of the public headers in include/. No storage format or protocol change.

Documentation

No.

Generative AI tooling

Generated-by: Claude Code (Claude Opus 5)

https://claude.ai/code/session_01NnX2Q5dAwrmeWYmf9n1mCC

[Joob1n]

@lxy-9602 @SteNicholas could you take a look when you have a moment?

This is a small correctness fix in the page-index path: ColumnIndexFilter was guessing the physical type of DECIMAL page bounds from their encoded length, so a DECIMAL stored as a 4-byte (precision 7-9) or 8-byte (precision 17-18) FIXED_LEN_BYTE_ARRAY had its big-endian bounds read little-endian and matching pages were pruned away. The fix threads the SchemaDescriptor down to the leaf and dispatches on physical_type(), mirroring what parquet_stats_extractor.cpp already does for row-group statistics.

The one API-shaped detail worth your eye: ColumnIndexFilter::CalculateRowRanges gains a const ::parquet::SchemaDescriptor* parameter (internal to src/paimon/format/parquet/, nothing in include/), and a null schema means the DECIMAL comparison is skipped so no page is pruned. Happy to change that to a hard requirement instead if you prefer.

Details and test results are in the PR description; it closes #345.

[zjw1111]

+1

[lxy-9602]

+1