From 81a6f8626320b2b4cc255a38a372407aab611853 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 09:27:15 +0800 Subject: [PATCH 01/11] [opt](parquet) Fuse fragmented nullable selection planning (#66397) - Fuse nullable definition-level runs with the row filter in one traversal. - Produce physical decode ranges, the selected NULL map, and selected value counts without first materializing and rescanning a row-wise selection map. - Reuse the existing selected-decoder strategies and nullable in-place expansion. - Restrict fusion to batches with at least 1,024 rows, at least 10% NULLs, and materially fragmented definition-level runs. No-NULL, low-NULL, clustered, nested, and non-expandable shapes keep the legacy path. The full benchmark matrix includes no-NULL, low-NULL, and clustered level plans as negative controls. Those shapes do not remove enough legacy work to guarantee a win, so this change deliberately leaves them unchanged. Decoder selection and encoding-specific materialization are not modified. - ASAN: `NativeNullableSelectionTest.*` and benchmark scenario tests: 16/16 passed. - ASAN: `ParquetV2NativeDecoderTest.*`: 118/118 passed. | Coverage | Legacy/fused pairs | Correctness | Regressions | Mean CPU change | Least improvement | |---|---:|---|---:|---:|---:| | Full scenario matrix | 100 | Identical ranges and NULL maps | N/A (includes negative controls) | N/A | N/A | | Production-eligible scenarios | 30 | Identical ranges and NULL maps | 0 | -42.68% | 8.08% | | Scenario | Repetitions | Legacy median CPU | Fused median CPU | CPU change | Legacy CV | Fused CV | |---|---:|---:|---:|---:|---:|---:| | 10% selectivity / 50% NULL, fragmented | 10 | 400,220 ns | 177,905 ns | -55.55% | 0.58% | 1.28% | | 99% selectivity / 50% NULL, high-selectivity boundary | 10 | 687,385 ns | 460,657 ns | -32.98% | 0.60% | 1.32% | The microbenchmark isolates nullable selection planning; it is not presented as an end-to-end query speedup. --- be/benchmark/parquet/AGENTS.md | 12 +- be/benchmark/parquet/README.md | 11 +- .../parquet/benchmark_parquet_kernels.hpp | 202 ++++++++++++++++++ .../parquet/parquet_benchmark_scenarios.h | 37 ++++ .../reader/native/column_chunk_reader.cpp | 59 +++++ .../reader/native/column_chunk_reader.h | 8 + .../parquet/reader/native/column_reader.cpp | 43 +++- .../parquet/reader/native/column_reader.h | 1 + .../parquet/reader/native/common.cpp | 128 +++++++++++ .../format_v2/parquet/reader/native/common.h | 11 + .../parquet_benchmark_scenarios_test.cpp | 26 +++ .../parquet/parquet_reader_control_test.cpp | 74 +++++++ 12 files changed, 601 insertions(+), 11 deletions(-) diff --git a/be/benchmark/parquet/AGENTS.md b/be/benchmark/parquet/AGENTS.md index 4c1d3cf4d5e197..4d8c0610f17b5d 100644 --- a/be/benchmark/parquet/AGENTS.md +++ b/be/benchmark/parquet/AGENTS.md @@ -51,7 +51,7 @@ be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetDecoder/' # currently 228 be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetKernel/' # currently 92 + | grep -c '^ParquetKernel/' # currently 292 be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetSelection/' # currently 25 @@ -146,13 +146,19 @@ cache to manufacture a cold run. | DELTA_LENGTH_BYTE_ARRAY | BYTE_ARRAY | | DELTA_BYTE_ARRAY | BYTE_ARRAY | -`ParquetKernel` contains 92 cases across six decode and selection stages: BYTE_STREAM_SPLIT, -DELTA_PREFIX_SUM, DICTIONARY_GATHER, NULLABLE_EXPAND, RAW_PREDICATE, and NESTED_SELECTION. It covers +`ParquetKernel` contains 292 cases across seven decode and selection stages: BYTE_STREAM_SPLIT, +DELTA_PREFIX_SUM, DICTIONARY_GATHER, NULLABLE_EXPAND, NULLABLE_SELECTION, RAW_PREDICATE, and +NESTED_SELECTION. It covers the applicable four- and eight-byte types, three dictionary working-set sizes, 0% through 90% null rates with both placement patterns, 0% through 100% raw-predicate selectivities, and 1%, 10%, and 50% nested parent-row selectivities with both placement patterns. Nested selection registers the legacy and fused implementations in the same binary and validates both against an independent source-level oracle before timing. +Nullable selection contributes 200 legacy/fused cases across five selectivities, five null rates, +and independent clustered or alternating selection/null placement. Each pair is validated for +identical physical ranges and null maps before timing. Treat no-NULL, low-NULL, and clustered +level-plan cases as negative controls: production fusion is gated to batches with at least 1,024 +rows, at least 10% NULLs, and materially fragmented definition-level runs. `ParquetSelection` contains 25 cases that isolate the selection-vector work used by Parquet predicate evaluation. It measures identity initialization, one raw-row filter, and two successive diff --git a/be/benchmark/parquet/README.md b/be/benchmark/parquet/README.md index 156302b24238d1..891fd02d8375ed 100644 --- a/be/benchmark/parquet/README.md +++ b/be/benchmark/parquet/README.md @@ -45,14 +45,21 @@ be/output/lib/benchmark_test \ ## SIMD kernel cases -`ParquetKernel` isolates six decode and selection stages from reader setup and virtual consumer +`ParquetKernel` isolates seven decode and selection stages from reader setup and virtual consumer overhead: byte-stream-split transpose, delta prefix sum, numeric dictionary gather, nullable -expansion, raw predicate evaluation, and repeated-level sparse selection. It covers the applicable +expansion, nullable selection planning, raw predicate evaluation, and repeated-level sparse +selection. It covers the applicable 4-byte and 8-byte integer and floating-point physical types, raw-predicate selectivities from 0% through 100%, and nullable rates from 0% through 90% with clustered and alternating placement. Nested selection covers 1%, 10%, and 50% surviving parent rows with both placement patterns. Each nested-selection scenario registers both `impl_legacy` and `impl_fused`; both paths use the same source levels and are checked against an independent oracle before timing. +Nullable selection planning registers legacy and fused pairs across five selectivities, five null +rates, and independent clustered or alternating selection/null placement. Both implementations are +checked for identical physical ranges and null maps before timing. The full matrix also acts as a +negative control: production fusion is limited to batches with at least 1,024 rows, at least 10% +NULLs, and fragmented definition-level runs; no-NULL, low-NULL, and clustered pages retain the +legacy planner. Dictionary gather uses 32-, 4,096-, and 262,144-entry working sets to separate cache-resident and cache-miss-dominated behavior. diff --git a/be/benchmark/parquet/benchmark_parquet_kernels.hpp b/be/benchmark/parquet/benchmark_parquet_kernels.hpp index 619d58fd8cf8ac..de3a15c254b8db 100644 --- a/be/benchmark/parquet/benchmark_parquet_kernels.hpp +++ b/be/benchmark/parquet/benchmark_parquet_kernels.hpp @@ -63,6 +63,191 @@ struct NestedSelectionScratch { size_t ancestor_null_count = 0; }; +struct NullableSelectionScratch { + format::parquet::native::ColumnSelectVector legacy_selection; + ParquetSelection physical_selection; + NullMap output_nulls; + NullMap selected_nulls; + size_t num_filtered = 0; +}; + +inline void append_nullable_run(std::vector* runs, bool is_null, size_t run_length, + bool* previous_is_null) { + if (runs->empty()) { + if (is_null) { + runs->push_back(0); + } + } else if (*previous_is_null == is_null) { + runs->push_back(0); + } + while (run_length > USHRT_MAX) { + runs->push_back(USHRT_MAX); + runs->push_back(0); + run_length -= USHRT_MAX; + } + runs->push_back(static_cast(run_length)); + *previous_is_null = is_null; +} + +inline std::vector build_nullable_runs(const NullMap& nulls) { + std::vector runs; + bool previous_is_null = false; + size_t row = 0; + while (row < nulls.size()) { + const bool is_null = nulls[row] != 0; + const size_t begin = row++; + while (row < nulls.size() && (nulls[row] != 0) == is_null) { + ++row; + } + append_nullable_run(&runs, is_null, row - begin, &previous_is_null); + } + return runs; +} + +inline Status run_legacy_nullable_selection(NullableSelectionScratch* scratch, + const std::vector& null_runs, + size_t num_values, + format::parquet::native::FilterMap* filter) { + using ReadType = format::parquet::native::ColumnSelectVector::DataReadType; + scratch->output_nulls.clear(); + scratch->selected_nulls.clear(); + scratch->physical_selection.ranges.clear(); + scratch->physical_selection.total_values = 0; + scratch->physical_selection.selected_values = 0; + RETURN_IF_ERROR(scratch->legacy_selection.init(null_runs, num_values, &scratch->output_nulls, + filter, 0)); + scratch->num_filtered = scratch->legacy_selection.num_filtered(); + + size_t physical_cursor = 0; + ReadType type; + while (const size_t run_length = scratch->legacy_selection.get_next_run(&type)) { + switch (type) { + case ReadType::CONTENT: + if (!scratch->physical_selection.ranges.empty() && + scratch->physical_selection.ranges.back().first + + scratch->physical_selection.ranges.back().count == + physical_cursor) { + scratch->physical_selection.ranges.back().count += run_length; + } else { + scratch->physical_selection.ranges.push_back( + {.first = physical_cursor, .count = run_length}); + } + scratch->physical_selection.selected_values += run_length; + scratch->selected_nulls.resize_fill(scratch->selected_nulls.size() + run_length, 0); + physical_cursor += run_length; + break; + case ReadType::NULL_DATA: + scratch->selected_nulls.resize_fill(scratch->selected_nulls.size() + run_length, 1); + break; + case ReadType::FILTERED_CONTENT: + physical_cursor += run_length; + break; + case ReadType::FILTERED_NULL: + break; + } + } + scratch->physical_selection.total_values = physical_cursor; + return Status::OK(); +} + +inline Status run_nullable_selection_once(NullableSelectionScratch* scratch, + const std::vector& null_runs, size_t num_values, + size_t num_nulls, + format::parquet::native::FilterMap* filter, + NullableSelectionImplementation implementation) { + if (implementation == NullableSelectionImplementation::LEGACY) { + return run_legacy_nullable_selection(scratch, null_runs, num_values, filter); + } + scratch->output_nulls.clear(); + return format::parquet::native::build_filtered_nullable_selection( + null_runs, num_values, num_nulls, &scratch->output_nulls, filter, 0, + &scratch->physical_selection, &scratch->selected_nulls, &scratch->num_filtered); +} + +inline bool equal_selection(const ParquetSelection& lhs, const ParquetSelection& rhs) { + if (lhs.total_values != rhs.total_values || lhs.selected_values != rhs.selected_values || + lhs.ranges.size() != rhs.ranges.size()) { + return false; + } + for (size_t range = 0; range < lhs.ranges.size(); ++range) { + if (lhs.ranges[range].first != rhs.ranges[range].first || + lhs.ranges[range].count != rhs.ranges[range].count) { + return false; + } + } + return true; +} + +inline void run_nullable_selection_kernel(benchmark::State& state, + const NullableSelectionScenario& scenario) { + using format::parquet::native::FilterMap; + + std::vector filter_data(KERNEL_ROWS, 0); + const auto selected = make_selection_plan(KERNEL_ROWS, scenario.selectivity_percent, + scenario.selection_pattern); + visit_selected_rows(selected, [&](size_t row) { filter_data[row] = 1; }); + FilterMap filter; + auto status = filter.init(filter_data.data(), filter_data.size(), false); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + + NullMap nulls; + nulls.resize_fill(KERNEL_ROWS, 0); + const auto null_plan = + make_selection_plan(KERNEL_ROWS, scenario.null_percent, scenario.null_pattern); + visit_selected_rows(null_plan, [&](size_t row) { nulls[row] = 1; }); + const auto null_runs = build_nullable_runs(nulls); + + NullableSelectionScratch legacy; + NullableSelectionScratch fused; + status = run_nullable_selection_once(&legacy, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, NullableSelectionImplementation::LEGACY); + if (status.ok()) { + status = + run_nullable_selection_once(&fused, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, NullableSelectionImplementation::FUSED); + } + if (!status.ok() || !equal_selection(legacy.physical_selection, fused.physical_selection) || + legacy.output_nulls != fused.output_nulls || + legacy.selected_nulls != fused.selected_nulls || + legacy.num_filtered != fused.num_filtered) { + if (status.ok()) { + state.SkipWithError("nullable selection implementations disagree"); + } else { + state.SkipWithError(status.to_string().c_str()); + } + return; + } + + NullableSelectionScratch scratch; + status = run_nullable_selection_once(&scratch, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, scenario.implementation); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + for (auto _ : state) { + status = run_nullable_selection_once(&scratch, null_runs, KERNEL_ROWS, + null_plan.selected_rows, &filter, + scenario.implementation); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + benchmark::DoNotOptimize(scratch.physical_selection.ranges.data()); + benchmark::DoNotOptimize(scratch.selected_nulls.data()); + benchmark::ClobberMemory(); + } + + state.SetItemsProcessed(static_cast(state.iterations()) * + static_cast(KERNEL_ROWS)); + state.counters["rows"] = static_cast(KERNEL_ROWS); + state.counters["selected_rows"] = static_cast(selected.selected_rows); + state.counters["null_rows"] = static_cast(null_plan.selected_rows); +} + inline NestedSelectionOracle build_nested_selection_oracle( const std::vector& repetition_levels, const std::vector& definition_levels, @@ -493,7 +678,24 @@ inline bool register_kernel_benchmarks() { return true; } +inline bool register_nullable_selection_benchmarks() { + for (const auto& scenario : nullable_selection_scenarios()) { + const std::string name = "ParquetKernel/nullable_selection/sel_" + + std::to_string(scenario.selectivity_percent) + "/null_" + + std::to_string(scenario.null_percent) + "/selection_" + + to_string(scenario.selection_pattern) + "/nulls_" + + to_string(scenario.null_pattern) + "/impl_" + + to_string(scenario.implementation); + benchmark::RegisterBenchmark(name.c_str(), [=](benchmark::State& state) { + run_nullable_selection_kernel(state, scenario); + })->Unit(benchmark::kNanosecond); + } + return true; +} + inline const bool KERNEL_BENCHMARKS_REGISTERED = register_kernel_benchmarks(); +inline const bool NULLABLE_SELECTION_BENCHMARKS_REGISTERED = + register_nullable_selection_benchmarks(); } // namespace detail } // namespace doris::parquet_benchmark diff --git a/be/benchmark/parquet/parquet_benchmark_scenarios.h b/be/benchmark/parquet/parquet_benchmark_scenarios.h index a9c58c15d8cff1..f7eddf0b11a5aa 100644 --- a/be/benchmark/parquet/parquet_benchmark_scenarios.h +++ b/be/benchmark/parquet/parquet_benchmark_scenarios.h @@ -55,6 +55,7 @@ enum class Kernel { NESTED_SELECTION }; enum class NestedSelectionImplementation { LEGACY, FUSED }; +enum class NullableSelectionImplementation { LEGACY, FUSED }; struct DecoderScenario { Encoding encoding; @@ -89,6 +90,14 @@ struct SelectionScenario { Pattern pattern; }; +struct NullableSelectionScenario { + int selectivity_percent; + int null_percent; + Pattern selection_pattern; + Pattern null_pattern; + NullableSelectionImplementation implementation; +}; + struct SelectionRange { size_t first; size_t count; @@ -177,6 +186,24 @@ inline std::vector selection_scenarios() { return scenarios; } +inline std::vector nullable_selection_scenarios() { + std::vector scenarios; + for (const int selectivity : {1, 10, 50, 90, 99}) { + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto selection_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto null_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto implementation : {NullableSelectionImplementation::LEGACY, + NullableSelectionImplementation::FUSED}) { + scenarios.push_back({selectivity, null_percent, selection_pattern, + null_pattern, implementation}); + } + } + } + } + } + return scenarios; +} + inline std::vector reader_scenarios() { std::vector scenarios; std::set::materialize_values( return Status::OK(); } +template +bool ColumnChunkReader::supports_fused_nullable_selection( + IColumn& column) const { + return visit_nullable_expandable_column(column, [](auto&) {}); +} + +template +Status ColumnChunkReader::materialize_fused_nullable_values( + MutableColumnPtr& doris_column, const DataTypeSerDe& serde, ParquetDecodeContext& context, + ParquetMaterializationState& state, size_t num_values, size_t num_nulls, + const NullMap& selected_nulls) { + if (num_values == 0) { + return Status::OK(); + } + SCOPED_RAW_TIMER(&_chunk_statistics.decode_value_time); + DORIS_CHECK_GT(num_nulls, 0); + const size_t physical_values = num_values - num_nulls; + DORIS_CHECK_EQ(state.selection.total_values, physical_values); + DORIS_CHECK_LE(state.selection.selected_values, selected_nulls.size()); + if (UNLIKELY(_empty_value_section && physical_values != 0)) { + return Status::Corruption( + "Parquet definition levels require {} values from an empty value section", + physical_values); + } + if (UNLIKELY((doris_column->is_column_dictionary() || context.dictionary_index_only) && + !_has_dict && physical_values != 0)) { + return Status::IOError("Not dictionary coded"); + } + if (UNLIKELY(_remaining_num_values < num_values)) { + return Status::IOError("Decode too many values in current page"); + } + RETURN_IF_ERROR(translate_value_encoding(_current_encoding, &context.encoding)); + + ++_chunk_statistics.hybrid_selection_batches; + const auto status = decode_prepared_nullable_values(*doris_column, serde, *_page_decoder, + context, state, selected_nulls, + &_chunk_statistics.materialization_time); + _chunk_statistics.hybrid_selection_ranges += state.selection.ranges.size(); + RETURN_IF_ERROR(status); + _remaining_num_values -= num_values; + return Status::OK(); +} + template bool ColumnChunkReader::can_filter_fixed_width_values( const VExprSPtrs& conjuncts, int column_id, const DataTypeSerDe* serde, diff --git a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h index 35bf5200d7aebb..a50b7e4bc080ff 100644 --- a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h @@ -177,6 +177,14 @@ class ColumnChunkReader { ParquetDecodeContext& context, ParquetMaterializationState& state, ColumnSelectVector& select_vector); + bool supports_fused_nullable_selection(IColumn& column) const; + + Status materialize_fused_nullable_values(MutableColumnPtr& doris_column, + const DataTypeSerDe& serde, + ParquetDecodeContext& context, + ParquetMaterializationState& state, size_t num_values, + size_t num_nulls, const NullMap& selected_nulls); + static bool supports_raw_fixed_filter_encoding(tparquet::Encoding::type encoding, tparquet::Type::type physical_type) { switch (encoding) { diff --git a/be/src/format_v2/parquet/reader/native/column_reader.cpp b/be/src/format_v2/parquet/reader/native/column_reader.cpp index f762cb0f9d35f5..b5ea931b29fdeb 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native/column_reader.cpp @@ -799,6 +799,8 @@ void ScalarColumnReader::release_batch_scratch( max_retained_bytes); release_selection |= release_vector_if_oversized(&_materialization_state.selection.ranges, max_retained_bytes); + release_selection |= + release_filter_if_oversized(&_fused_nullable_selection_nulls, max_retained_bytes); release_selection |= release_filter_if_oversized(&_fixed_width_predicate_nulls, max_retained_bytes); release_selection |= @@ -832,6 +834,7 @@ void ScalarColumnReader::release_batch_scratch( release_selection |= release_vector_for_aggregate(&_nested_filter_map_data); release_selection |= release_vector_for_aggregate(&_materialization_state.dictionary_indices); release_selection |= release_vector_for_aggregate(&_materialization_state.selection.ranges); + release_selection |= release_filter_for_aggregate(&_fused_nullable_selection_nulls); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_nulls); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_matches); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_conversion_nulls); @@ -860,7 +863,8 @@ size_t ScalarColumnReader::retained_batch_scratch_b _def_levels.capacity() * sizeof(level_t) + _null_run_lengths.capacity() * sizeof(uint16_t) + _nested_filter_map_data.capacity() * sizeof(uint8_t) + - _fixed_width_predicate_nulls.capacity() + _fixed_width_predicate_matches.capacity() + + _fused_nullable_selection_nulls.capacity() + _fixed_width_predicate_nulls.capacity() + + _fixed_width_predicate_matches.capacity() + _fixed_width_predicate_conversion_nulls.capacity() + _materialization_state.dictionary_indices.capacity() * sizeof(uint32_t) + _materialization_state.selection.ranges.capacity() * sizeof(ParquetSelectionRange) + @@ -875,7 +879,8 @@ size_t ScalarColumnReader::active_batch_scratch_byt _serde == nullptr ? 0 : _serde->active_parquet_raw_predicate_scratch_bytes(); return decoder_bytes + serde_bytes + _rep_levels.size() * sizeof(level_t) + _def_levels.size() * sizeof(level_t) + _null_run_lengths.size() * sizeof(uint16_t) + - _nested_filter_map_data.size() * sizeof(uint8_t) + _fixed_width_predicate_nulls.size() + + _nested_filter_map_data.size() * sizeof(uint8_t) + + _fused_nullable_selection_nulls.size() + _fixed_width_predicate_nulls.size() + _fixed_width_predicate_matches.size() + _fixed_width_predicate_conversion_nulls.size() + _materialization_state.dictionary_indices.size() * sizeof(uint32_t) + _materialization_state.selection.ranges.size() * sizeof(ParquetSelectionRange) + @@ -892,6 +897,7 @@ void ScalarColumnReader::reserve_batch_scratch_for_ _nested_filter_map_data.reserve(elements); _materialization_state.dictionary_indices.reserve(elements); _materialization_state.selection.ranges.reserve(elements); + _fused_nullable_selection_nulls.reserve(elements); _ancestor_null_indices.reserve(elements); } @@ -955,6 +961,7 @@ Status ScalarColumnReader::_read_values(size_t num_ } MutableColumnPtr data_column; _null_run_lengths.clear(); + size_t num_nulls = 0; NullMap* map_data_column = nullptr; doris_column = IColumn::mutate(std::move(doris_column)); if (is_column_nullable(*doris_column)) { @@ -977,6 +984,9 @@ Status ScalarColumnReader::_read_values(size_t num_ } bool is_null = def_level < _field_schema->definition_level; + if (is_null) { + num_nulls += loop_read; + } if (!(prev_is_null ^ is_null)) { _null_run_lengths.emplace_back(0); } @@ -1006,10 +1016,26 @@ Status ScalarColumnReader::_read_values(size_t num_ } _null_run_lengths.emplace_back((u_short)remaining); } + const bool use_fused_nullable_selection = + map_data_column != nullptr && filter_map.has_filter() && num_nulls > 0 && + should_use_fused_nullable_selection(num_values, num_nulls, _null_run_lengths.size()) && + _chunk_reader->supports_fused_nullable_selection(*data_column); { SCOPED_RAW_TIMER(&_decode_null_map_time); - RETURN_IF_ERROR(_select_vector.init(_null_run_lengths, num_values, map_data_column, - &filter_map, _filter_map_index)); + if (use_fused_nullable_selection) { + size_t num_filtered = 0; + // The fused path owns both the physical ranges and selected NULL layout. Restrict it + // to fragmented, materially nullable level plans: clustered, low-NULL, and no-NULL + // pages already collapse into a few cheap legacy runs, while fusing them adds planning + // branches without removing enough work to guarantee a win. + RETURN_IF_ERROR(build_filtered_nullable_selection( + _null_run_lengths, num_values, num_nulls, map_data_column, &filter_map, + _filter_map_index, &_materialization_state.selection, + &_fused_nullable_selection_nulls, &num_filtered)); + } else { + RETURN_IF_ERROR(_select_vector.init(_null_run_lengths, num_values, map_data_column, + &filter_map, _filter_map_index)); + } _filter_map_index += num_values; } DORIS_CHECK(_serde != nullptr); @@ -1020,8 +1046,13 @@ Status ScalarColumnReader::_read_values(size_t num_ conversion_failure_map(*_field_schema, type, _materialization_state.enable_strict_mode, map_data_column, &compatibility_scratch); const size_t materialization_start_row = data_column->size(); - const auto status = _chunk_reader->materialize_values(data_column, *_serde, _decode_context, - _materialization_state, _select_vector); + const auto status = + use_fused_nullable_selection + ? _chunk_reader->materialize_fused_nullable_values( + data_column, *_serde, _decode_context, _materialization_state, + num_values, num_nulls, _fused_nullable_selection_nulls) + : _chunk_reader->materialize_values(data_column, *_serde, _decode_context, + _materialization_state, _select_vector); _materialization_state.conversion_failure_null_map = nullptr; if (status.ok()) { mark_local_timestamp_defaults(*_field_schema, type, diff --git a/be/src/format_v2/parquet/reader/native/column_reader.h b/be/src/format_v2/parquet/reader/native/column_reader.h index 18bcdd866f83d7..594f0ac0921523 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_reader.h @@ -432,6 +432,7 @@ class ScalarColumnReader : public ColumnReader { std::vector _null_run_lengths; std::unordered_set _ancestor_null_indices; std::vector _nested_filter_map_data; + NullMap _fused_nullable_selection_nulls; NullMap _fixed_width_predicate_nulls; IColumn::Filter _fixed_width_predicate_matches; IColumn::Filter _fixed_width_predicate_conversion_nulls; diff --git a/be/src/format_v2/parquet/reader/native/common.cpp b/be/src/format_v2/parquet/reader/native/common.cpp index c50488575e4993..020f613c5d267f 100644 --- a/be/src/format_v2/parquet/reader/native/common.cpp +++ b/be/src/format_v2/parquet/reader/native/common.cpp @@ -17,6 +17,7 @@ #include "format_v2/parquet/reader/native/common.h" +#include #include #include "core/types.h" @@ -68,6 +69,133 @@ bool FilterMap::can_filter_all(size_t remaining_num_values, size_t filter_map_in remaining_num_values) == remaining_num_values; } +bool should_use_fused_nullable_selection(size_t num_values, size_t num_nulls, + size_t num_null_runs) { + constexpr size_t MIN_BATCH_VALUES = 1024; + constexpr size_t MIN_NULL_RUNS = 32; + constexpr size_t MAX_AVERAGE_NULL_RUN = 64; + constexpr size_t MIN_NULL_RATIO_DENOMINATOR = 10; + if (num_values < MIN_BATCH_VALUES || num_nulls < num_values / MIN_NULL_RATIO_DENOMINATOR) { + return false; + } + return num_null_runs >= std::max(MIN_NULL_RUNS, num_values / MAX_AVERAGE_NULL_RUN); +} + +Status build_filtered_nullable_selection(const std::vector& run_length_null_map, + size_t num_values, size_t num_nulls, + NullMap* output_null_map, FilterMap* filter_map, + size_t filter_map_index, ParquetSelection* selection, + NullMap* selected_nulls, size_t* num_filtered) { + if (output_null_map == nullptr || filter_map == nullptr || selection == nullptr || + selected_nulls == nullptr || num_filtered == nullptr) { + return Status::InvalidArgument( + "Nullable selection planning requires non-null output state"); + } + if (!filter_map->has_filter()) { + return Status::InvalidArgument("Nullable selection planning requires a row filter"); + } + if (!filter_map->filter_all() && + (filter_map->filter_map_data() == nullptr || + filter_map_index + num_values > filter_map->filter_map_size())) { + return Status::InvalidArgument("Nullable selection filter range [{}, {}) exceeds size {}", + filter_map_index, filter_map_index + num_values, + filter_map->filter_map_size()); + } + if (num_nulls > num_values) { + return Status::InvalidArgument("Nullable selection has {} nulls for {} values", num_nulls, + num_values); + } + + selection->ranges.clear(); + selection->total_values = num_values - num_nulls; + selection->selected_values = 0; + selected_nulls->clear(); + *num_filtered = 0; + if (filter_map->filter_all()) { + *num_filtered = num_values; + return Status::OK(); + } + + selected_nulls->reserve(num_values); + const uint8_t* filter = filter_map->filter_map_data() + filter_map_index; + const auto select_physical_values = [&](size_t physical_index, size_t count) { + if (!selection->ranges.empty() && + selection->ranges.back().first + selection->ranges.back().count == physical_index) { + selection->ranges.back().count += count; + } else { + selection->ranges.push_back({.first = physical_index, .count = count}); + } + selection->selected_values += count; + }; + + if (num_nulls == 0) { + size_t row = 0; + while (row < num_values) { + const bool selected = filter[row] != 0; + const size_t run_start = row++; + while (row < num_values && (filter[row] != 0) == selected) { + ++row; + } + const size_t run_length = row - run_start; + if (selected) { + select_physical_values(run_start, run_length); + } else { + *num_filtered += run_length; + } + } + selected_nulls->resize_fill(selection->selected_values, 0); + } else { + size_t logical_index = 0; + size_t physical_index = 0; + size_t observed_nulls = 0; + bool is_null = false; + for (const size_t run_length : run_length_null_map) { + if (logical_index + run_length > num_values) { + return Status::InvalidArgument("Nullable selection run lengths exceed {} values", + num_values); + } + const size_t run_end = logical_index + run_length; + while (logical_index < run_end) { + const bool selected = filter[logical_index] != 0; + const size_t filter_run_start = logical_index++; + while (logical_index < run_end && (filter[logical_index] != 0) == selected) { + ++logical_index; + } + const size_t filter_run_length = logical_index - filter_run_start; + if (selected) { + selected_nulls->resize_fill(selected_nulls->size() + filter_run_length, + static_cast(is_null)); + if (!is_null) { + select_physical_values(physical_index, filter_run_length); + } + } else { + *num_filtered += filter_run_length; + } + if (!is_null) { + physical_index += filter_run_length; + } else { + observed_nulls += filter_run_length; + } + } + is_null = !is_null; + } + if (logical_index != num_values || observed_nulls != num_nulls || + physical_index != selection->total_values) { + return Status::InvalidArgument( + "Nullable selection level plan is inconsistent: values={}, nulls={}", + logical_index, observed_nulls); + } + } + + const size_t old_null_size = output_null_map->size(); + output_null_map->resize(old_null_size + selected_nulls->size()); + if (!selected_nulls->empty()) { + memcpy(output_null_map->data() + old_null_size, selected_nulls->data(), + selected_nulls->size()); + } + return Status::OK(); +} + Status FilterMap::generate_nested_filter_map(const std::vector& rep_levels, std::vector& nested_filter_map_data, std::unique_ptr* nested_filter_map, diff --git a/be/src/format_v2/parquet/reader/native/common.h b/be/src/format_v2/parquet/reader/native/common.h index eb6848ee299f10..bd687616f226ce 100644 --- a/be/src/format_v2/parquet/reader/native/common.h +++ b/be/src/format_v2/parquet/reader/native/common.h @@ -25,6 +25,7 @@ #include "common/status.h" #include "core/column/column_nullable.h" +#include "core/data_type_serde/parquet_decode_source.h" namespace doris::format::parquet::native { @@ -116,4 +117,14 @@ class ColumnSelectVector { size_t _read_index = 0; }; +Status build_filtered_nullable_selection(const std::vector& run_length_null_map, + size_t num_values, size_t num_nulls, + NullMap* output_null_map, FilterMap* filter_map, + size_t filter_map_index, ParquetSelection* selection, + NullMap* selected_nulls, size_t* num_filtered); + +// Fusion pays for its additional planning branches only when definition levels are materially +// nullable and fragmented. Keep compact/no-NULL batches on the run-oriented legacy path. +bool should_use_fused_nullable_selection(size_t num_values, size_t num_nulls, size_t num_null_runs); + } // namespace doris::format::parquet::native diff --git a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp index 2145b6ab60da43..fe1d871816ed26 100644 --- a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp +++ b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp @@ -147,6 +147,32 @@ TEST(ParquetBenchmarkScenariosTest, SelectionMatrixCoversIdentityAndSuccessiveCo } } +TEST(ParquetBenchmarkScenariosTest, NullableSelectionPairsLegacyAndFusedAcrossRowShapes) { + const auto scenarios = nullable_selection_scenarios(); + EXPECT_EQ(scenarios.size(), size_t {200}); + for (const int selectivity : {1, 10, 50, 90, 99}) { + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto selection_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto null_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto implementation : {NullableSelectionImplementation::LEGACY, + NullableSelectionImplementation::FUSED}) { + EXPECT_TRUE(std::ranges::any_of( + scenarios, + [&](const NullableSelectionScenario& scenario) { + return scenario.selectivity_percent == selectivity && + scenario.null_percent == null_percent && + scenario.selection_pattern == selection_pattern && + scenario.null_pattern == null_pattern && + scenario.implementation == implementation; + })) + << "missing nullable selection comparison shape"; + } + } + } + } + } +} + TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversNullableSparseAndProjectionAxes) { const auto scenarios = reader_scenarios(); // Keep the exact count aligned with the upstream complex-residual scenario retained by rebase. diff --git a/be/test/format_v2/parquet/parquet_reader_control_test.cpp b/be/test/format_v2/parquet/parquet_reader_control_test.cpp index e21439e885570d..37deb32d5bf37b 100644 --- a/be/test/format_v2/parquet/parquet_reader_control_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_control_test.cpp @@ -27,6 +27,7 @@ #include "core/column/column_string.h" #include "core/column/column_vector.h" #include "core/data_type/data_type_number.h" +#include "core/data_type_serde/parquet_decode_source.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_scan.h" #include "format_v2/parquet/reader/column_reader.h" @@ -186,6 +187,79 @@ TEST(SelectionVectorTest, IdentitySelectionDoesNotMaterializeFilter) { EXPECT_EQ(filter, nullptr); } +TEST(NativeNullableSelectionTest, BuildsPhysicalRangesAndSelectedNullsInOnePass) { + using native::FilterMap; + + const std::vector null_runs {2, 1, 3, 2, 2}; + const std::vector filter_data {1, 0, 1, 1, 0, 1, 1, 1, 0, 1}; + FilterMap filter; + ASSERT_TRUE(filter.init(filter_data.data(), filter_data.size(), false).ok()); + ParquetSelection selection; + NullMap output_nulls {1}; + NullMap selected_nulls; + size_t num_filtered = 0; + + ASSERT_TRUE(native::build_filtered_nullable_selection(null_runs, filter_data.size(), 3, + &output_nulls, &filter, 0, &selection, + &selected_nulls, &num_filtered) + .ok()); + + EXPECT_EQ(selection.total_values, 7); + EXPECT_EQ(selection.selected_values, 4); + ASSERT_EQ(selection.ranges.size(), 4); + EXPECT_EQ(selection.ranges[0].first, 0); + EXPECT_EQ(selection.ranges[0].count, 1); + EXPECT_EQ(selection.ranges[1].first, 2); + EXPECT_EQ(selection.ranges[1].count, 1); + EXPECT_EQ(selection.ranges[2].first, 4); + EXPECT_EQ(selection.ranges[2].count, 1); + EXPECT_EQ(selection.ranges[3].first, 6); + EXPECT_EQ(selection.ranges[3].count, 1); + EXPECT_EQ(selected_nulls, (NullMap {0, 1, 0, 0, 1, 1, 0})); + EXPECT_EQ(output_nulls, (NullMap {1, 0, 1, 0, 0, 1, 1, 0})); + EXPECT_EQ(num_filtered, 3); +} + +TEST(NativeNullableSelectionTest, UsesDirectPhysicalCoordinatesWithoutNulls) { + using native::FilterMap; + + const std::vector no_nulls {10}; + const std::vector filter_data {1, 1, 0, 1, 0, 0, 1, 1, 1, 0}; + FilterMap filter; + ASSERT_TRUE(filter.init(filter_data.data(), filter_data.size(), false).ok()); + ParquetSelection selection; + NullMap output_nulls; + NullMap selected_nulls; + size_t num_filtered = 0; + + ASSERT_TRUE(native::build_filtered_nullable_selection(no_nulls, filter_data.size(), 0, + &output_nulls, &filter, 0, &selection, + &selected_nulls, &num_filtered) + .ok()); + + EXPECT_EQ(selection.total_values, 10); + EXPECT_EQ(selection.selected_values, 6); + ASSERT_EQ(selection.ranges.size(), 3); + EXPECT_EQ(selection.ranges[0].first, 0); + EXPECT_EQ(selection.ranges[0].count, 2); + EXPECT_EQ(selection.ranges[1].first, 3); + EXPECT_EQ(selection.ranges[1].count, 1); + EXPECT_EQ(selection.ranges[2].first, 6); + EXPECT_EQ(selection.ranges[2].count, 3); + EXPECT_EQ(selected_nulls, (NullMap {0, 0, 0, 0, 0, 0})); + EXPECT_EQ(output_nulls, selected_nulls); + EXPECT_EQ(num_filtered, 4); +} + +TEST(NativeNullableSelectionTest, EnablesFusionOnlyForMateriallyFragmentedNullableBatches) { + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 0, 3)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 655, 1311)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 32768, 3)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(512, 256, 512)); + EXPECT_TRUE(native::should_use_fused_nullable_selection(65536, 6553, 13107)); + EXPECT_TRUE(native::should_use_fused_nullable_selection(65536, 32768, 65536)); +} + TEST(NativeNestedSelectionTest, BuildsSelectionAndCompactsSurvivingParentLevels) { using native::ColumnSelectVector; using native::FilterMap; From 50c1af7d253af380d71cf365f162d323e6559f7b Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 11:30:10 +0800 Subject: [PATCH 02/11] [feature](variant) Support reading Iceberg Variant from Parquet (#66302) Forward-port the Iceberg Variant Parquet reader to the plugin-driven connector on master while preserving mixed-version scan safety and delete-only merge behavior. --- be/src/core/block/block.cpp | 5 +- .../column/variant_v2/column_variant_v2.cpp | 172 ++- .../column/variant_v2/column_variant_v2.h | 63 +- .../column_variant_v2_read_view.cpp | 3 + be/src/exec/operator/file_scan_operator.cpp | 43 + be/src/exec/operator/scan_operator.h | 12 + be/src/exec/scan/access_path_parser.cpp | 95 +- be/src/exec/scan/access_path_parser.h | 6 + be/src/exec/scan/file_scanner_v2.cpp | 37 +- be/src/exec/scan/file_scanner_v2.h | 2 + be/src/exec/scan/scanner.cpp | 9 + be/src/exec/scan/scanner.h | 4 + be/src/exec/scan/split_source_connector.h | 17 + be/src/exec/sink/viceberg_merge_sink.cpp | 26 +- be/src/exec/sink/viceberg_merge_sink.h | 1 + .../function/function_variant_element_v2.cpp | 45 +- be/src/format_v2/column_data.h | 24 + be/src/format_v2/column_mapper.cpp | 371 ++++-- be/src/format_v2/column_mapper.h | 17 + be/src/format_v2/column_mapper_nested.cpp | 116 -- be/src/format_v2/column_mapper_nested.h | 2 - be/src/format_v2/file_reader.cpp | 8 + be/src/format_v2/file_reader.h | 76 +- .../format_v2/parquet/native_schema_desc.cpp | 434 +++++++ be/src/format_v2/parquet/native_schema_desc.h | 4 + .../format_v2/parquet/native_schema_node.cpp | 3 +- .../parquet/parquet_column_schema.cpp | 63 +- .../format_v2/parquet/parquet_column_schema.h | 5 + be/src/format_v2/parquet/parquet_profile.cpp | 20 + be/src/format_v2/parquet/parquet_profile.h | 14 + be/src/format_v2/parquet/parquet_reader.cpp | 123 ++ be/src/format_v2/parquet/parquet_reader.h | 9 + be/src/format_v2/parquet/parquet_scan.cpp | 29 +- .../format_v2/parquet/parquet_statistics.cpp | 458 ++++++- be/src/format_v2/parquet/parquet_statistics.h | 2 +- be/src/format_v2/parquet/parquet_type.h | 2 + .../parquet/reader/count_column_reader.cpp | 11 + .../parquet/reader/native_column_reader.cpp | 209 +++- .../parquet/reader/native_column_reader.h | 8 +- .../parquet/reader/variant_column_reader.cpp | 1112 +++++++++++++++++ .../parquet/reader/variant_column_reader.h | 67 + be/src/format_v2/schema_projection.cpp | 5 + be/src/format_v2/table/iceberg_reader.cpp | 88 ++ be/src/format_v2/table/iceberg_reader.h | 4 + be/src/format_v2/table_reader.h | 36 +- .../core/column/column_variant_v2_test.cpp | 27 + be/test/exec/scan/access_path_parser_test.cpp | 87 ++ be/test/exec/scan/file_scanner_v2_test.cpp | 34 + .../scan/scanner_late_arrival_rf_test.cpp | 49 +- .../exec/sink/viceberg_merge_sink_test.cpp | 38 + be/test/format_v2/column_mapper_test.cpp | 904 +++++--------- .../format_v2/parquet/parquet_reader_test.cpp | 570 +++++++++ .../format_v2/parquet/parquet_schema_test.cpp | 397 ++++++ .../parquet/parquet_statistics_test.cpp | 437 ++++++- .../parquet/variant_column_reader_test.cpp | 927 ++++++++++++++ .../format_v2/table/iceberg_reader_test.cpp | 39 + .../format_v2/table_reader_request_test.cpp | 26 + .../iceberg/IcebergScanPlanProvider.java | 13 + .../connector/iceberg/IcebergTypeMapping.java | 6 +- .../iceberg/IcebergWritePlanProvider.java | 28 +- .../iceberg/IcebergScanPlanProviderTest.java | 16 + .../iceberg/IcebergTypeMappingReadTest.java | 10 +- .../iceberg/IcebergWritePlanProviderTest.java | 13 + .../spi/handle/ConnectorWriteHandle.java | 8 + .../spi/scan/ConnectorScanPlanProvider.java | 11 + .../converter/ConnectorColumnConverter.java | 5 + .../datasource/scan/PluginDrivenScanNode.java | 63 + .../translator/PhysicalPlanTranslator.java | 9 +- ...nkToPhysicalExternalRowLevelMergeSink.java | 1 + .../AccessPathExpressionCollector.java | 9 + .../rewrite/AccessPathPlanCollector.java | 7 + .../rules/rewrite/NestedColumnPruning.java | 5 + .../rules/rewrite/SlotTypeReplacer.java | 5 + .../ExternalRowLevelMergePlanBuilder.java | 2 + .../ExternalRowLevelUpdatePlanBuilder.java | 1 + .../LogicalExternalRowLevelMergeSink.java | 40 +- .../PhysicalExternalRowLevelMergeSink.java | 58 +- .../doris/planner/PluginDrivenTableSink.java | 27 +- .../ConnectorColumnConverterTest.java | 7 + ...PluginDrivenScanNodeCompatibilityTest.java | 50 + .../rules/rewrite/PruneNestedColumnTest.java | 19 + .../rewrite/VariantPruningLogicTest.java | 19 + .../planner/PluginDrivenTableSinkTest.java | 14 + gensrc/thrift/DataSinks.thrift | 2 + .../iceberg/iceberg_variant_shredded.parquet | Bin 0 -> 34528 bytes ...-7100-4eb0-a42e-e52ddc62d9e3.metadata.json | 74 ++ ...958052-e154-425f-8850-f0011d0272c5-m0.avro | Bin 0 -> 7778 bytes ...-b7958052-e154-425f-8850-f0011d0272c5.avro | Bin 0 -> 4758 bytes .../iceberg/test_iceberg_varbinary.out | Bin 1626 -> 1648 bytes .../iceberg/test_iceberg_variant_read.out | 123 ++ .../iceberg/test_iceberg_varbinary.groovy | 20 +- .../iceberg/test_iceberg_variant_read.groovy | 551 ++++++++ .../variant_p0/variant_with_rowstore.groovy | 7 +- 93 files changed, 7629 insertions(+), 989 deletions(-) create mode 100644 be/src/format_v2/parquet/reader/variant_column_reader.cpp create mode 100644 be/src/format_v2/parquet/reader/variant_column_reader.h create mode 100644 be/test/format_v2/parquet/variant_column_reader_test.cpp create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/b7958052-e154-425f-8850-f0011d0272c5-m0.avro create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro create mode 100644 regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out create mode 100644 regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy diff --git a/be/src/core/block/block.cpp b/be/src/core/block/block.cpp index 8b92c2e3db9156..f2315aaacff03f 100644 --- a/be/src/core/block/block.cpp +++ b/be/src/core/block/block.cpp @@ -828,6 +828,7 @@ void Block::clear() { data.clear(); } +// Both clear paths must preserve shared children even when a composite column is top-level exclusive. void Block::clear_column_data(int64_t column_size) { SCOPED_SKIP_MEMORY_CHECK(); // data.size() greater than column_size, means here have some @@ -839,7 +840,7 @@ void Block::clear_column_data(int64_t column_size) { } for (auto& d : data) { if (d.column) { - if (d.column->is_exclusive()) { + if (is_recursively_exclusive(*d.column)) { d.column->assert_mutable()->clear(); } else { d.column = d.column->clone_empty(); @@ -854,7 +855,7 @@ void Block::clear_column_data(const std::vector& columns_to_clear) { DCHECK_LT(col, data.size()); auto& column = data[col].column; if (column) { - if (column->is_exclusive()) { + if (is_recursively_exclusive(*column)) { column->assert_mutable()->clear(); } else { column = column->clone_empty(); diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index 6a74ac091ee5cb..0d5a7fb74bb6df 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -347,7 +347,8 @@ ColumnVariantV2::ColumnVariantV2(const ColumnVariantV2& other) _meta_ids(other._meta_ids), _values(other._values), _typed(other._typed), - _typed_type(other._typed_type) {} + _typed_type(other._typed_type), + _shredded(other._shredded) {} ColumnVariantV2::MutablePtr ColumnVariantV2::create_typed(ColumnPtr column, DataTypePtr scalar_type) { @@ -359,6 +360,15 @@ ColumnVariantV2::MutablePtr ColumnVariantV2::create_typed(ColumnPtr column, return result; } +ColumnVariantV2::MutablePtr ColumnVariantV2::create_shredded( + std::shared_ptr state) { + DORIS_CHECK(state != nullptr) << "shredded ColumnVariantV2 state must not be null"; + auto result = ColumnVariantV2::create(); + result->_shredded = std::move(state); + result->_check_invariants(); + return result; +} + const IColumn& ColumnVariantV2::typed_column() const { DORIS_CHECK(_typed != nullptr) << "typed_column requires ColumnVariantV2 typed state"; return *_typed; @@ -369,7 +379,30 @@ const DataTypePtr& ColumnVariantV2::typed_type() const { return _typed_type; } +std::optional ColumnVariantV2::find_shredded_typed_value( + std::span path) const { + if (!_shredded) { + return std::nullopt; + } + return _shredded->find_typed_value(path); +} + void ColumnVariantV2::ensure_encoded() { + if (_shredded) { + const ColumnVariantV2& materialized = _shredded->materialized_column(); + DORIS_CHECK(!materialized.is_shredded()) + << "shredded state materializer returned another shredded column"; + // The shredded state may cache and share its canonical materialization across readers. + // Detach every mutable buffer before dropping that owner so later COW mutations stay legal. + _metadatas = materialized._metadatas->clone_resized(materialized._metadatas->size()); + _meta_ids = materialized._meta_ids->clone_resized(materialized._meta_ids->size()); + _values = materialized._values->clone_resized(materialized._values->size()); + _typed = materialized._typed == nullptr + ? nullptr + : materialized._typed->clone_resized(materialized._typed->size()); + _typed_type = materialized._typed_type; + _shredded.reset(); + } if (!_typed) { DCHECK(_typed_type == nullptr); return; @@ -394,6 +427,9 @@ void ColumnVariantV2::ensure_encoded() { } std::string ColumnVariantV2::get_name() const { + if (_shredded) { + return "variant_v2(shredded)"; + } if (_typed) { DORIS_CHECK(_typed_type != nullptr); return "variant_v2(typed=" + _typed_type->get_name() + ")"; @@ -403,6 +439,9 @@ std::string ColumnVariantV2::get_name() const { } size_t ColumnVariantV2::size() const { + if (_shredded) { + return _shredded->size(); + } if (_typed) { DCHECK(_typed_type != nullptr); DCHECK(_metadatas->empty()); @@ -417,6 +456,9 @@ size_t ColumnVariantV2::size() const { } size_t ColumnVariantV2::byte_size() const { + if (_shredded) { + return _shredded->byte_size(); + } if (_typed) { DCHECK(_metadatas->empty()); DCHECK(_meta_ids->empty()); @@ -428,6 +470,9 @@ size_t ColumnVariantV2::byte_size() const { } size_t ColumnVariantV2::allocated_bytes() const { + if (_shredded) { + return _shredded->allocated_bytes(); + } if (_typed) { DCHECK(_metadatas->empty()); DCHECK(_meta_ids->empty()); @@ -441,6 +486,9 @@ size_t ColumnVariantV2::allocated_bytes() const { bool ColumnVariantV2::has_enough_capacity(const IColumn& src) const { const auto& source = assert_cast(src); + if (_shredded || source._shredded) { + return false; + } if (static_cast(_typed) != static_cast(source._typed)) { return false; } @@ -460,6 +508,11 @@ bool ColumnVariantV2::structure_equals(const IColumn& rhs) const { } void ColumnVariantV2::sanity_check() const { + if (_shredded) { + _shredded->sanity_check(); + _check_invariants(); + return; + } if (_typed) { _typed->sanity_check(); } else { @@ -487,6 +540,10 @@ void ColumnVariantV2::sanity_check() const { } void ColumnVariantV2::for_each_subcolumn(ColumnCallback callback) const { + if (_shredded) { + _shredded->for_each_subcolumn(callback); + return; + } if (_typed) { callback(*static_cast(_typed)); } else { @@ -497,6 +554,11 @@ void ColumnVariantV2::for_each_subcolumn(ColumnCallback callback) const { } void ColumnVariantV2::mutate_subcolumns() { + if (_shredded) { + // Shredded state is immutable and reference-counted, so keep a partial leaf projection + // intact until an operation explicitly requires canonical bytes. + return; + } if (_typed) { mutate_subcolumn(_typed); } else { @@ -507,6 +569,11 @@ void ColumnVariantV2::mutate_subcolumns() { } void ColumnVariantV2::clear() { + if (_shredded) { + _shredded.reset(); + _check_invariants(); + return; + } if (_typed) { mutate_subcolumn(_typed); _typed->clear(); @@ -528,7 +595,7 @@ void ColumnVariantV2::clear() { // Validate the encoded batch before appending metadata, ids, and values. void ColumnVariantV2::insert_encoded_rows( // NOLINT(readability-function-size) const EncodedDataView& data) { - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } DORIS_CHECK(_typed_type == nullptr) << "encoded state cannot retain a typed data type"; @@ -606,7 +673,7 @@ void ColumnVariantV2::insert_encoded_rows( // NOLINT(readability-function-size) } void ColumnVariantV2::insert_encoded_batch(const VariantBatchBuilder& block) { - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } DORIS_CHECK(_typed_type == nullptr) << "encoded state cannot retain a typed data type"; @@ -639,6 +706,9 @@ void ColumnVariantV2::insert_encoded_batch(const VariantBatchBuilder& block) { } VariantRef ColumnVariantV2::get_value_ref(size_t row) const { + if (_shredded) { + return _shredded->materialized_column().get_value_ref(row); + } DCHECK(!_typed); DCHECK(_typed_type == nullptr); DCHECK_LT(row, size()); @@ -672,7 +742,7 @@ void ColumnVariantV2::insert_many_defaults(size_t length) { return; } - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } @@ -724,6 +794,39 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) return; } + if (!_shredded && !_typed && empty() && _metadatas->empty() && source._shredded) { + // IColumn::cut() inserts into an empty clone. Select the physical tree directly because an + // incomplete leaf projection cannot be reconstructed merely to copy a row range. + _shredded = start == 0 && length == source.size() + ? source._shredded + : source._shredded->select_range(start, length); + _check_invariants(); + return; + } + if (_shredded && source._shredded) { + auto selected_source = start == 0 && length == source.size() + ? source._shredded + : source._shredded->select_range(start, length); + // C++20 libc++ removed shared_ptr::unique(); use_count preserves the same COW invariant + // on every supported toolchain before mutating the format-owned state. + if (_shredded.use_count() != 1) { + _shredded = _shredded->select_range(0, size()); + } + // A partial physical projection has no metadata/value pair to encode. Preserve that + // invariant by merging compatible scanner batches before the canonical fallback below. + if (_shredded->try_append(*selected_source)) { + _check_invariants(); + return; + } + } + if (_shredded) { + ensure_encoded(); + } + if (source._shredded) { + insert_range_from(source._shredded->materialized_column(), start, length); + return; + } + if (_typed && source._typed && exact_typed_identity(_typed_type, source._typed_type)) { mutate_subcolumn(_typed); _typed->insert_range_from(*source._typed, start, length); @@ -808,6 +911,21 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } + if (_shredded) { + ensure_encoded(); + } + if (!_typed && empty() && _metadatas->empty() && source._shredded) { + // Gather into the native shredded representation for the same reason as range selection: + // row selection does not require, and may not have, a complete logical Variant value. + _shredded = source._shredded->select_indices(indices_begin, indices_end); + _check_invariants(); + return; + } + if (source._shredded) { + insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); + return; + } + if (_typed && source._typed && exact_typed_identity(_typed_type, source._typed_type)) { mutate_subcolumn(_typed); _typed->insert_indices_from(*source._typed, indices_begin, indices_end); @@ -885,6 +1003,9 @@ void ColumnVariantV2::pop_back(size_t length) { if (length == 0) { return; } + if (_shredded) { + ensure_encoded(); + } if (_typed) { mutate_subcolumn(_typed); _typed->pop_back(length); @@ -1152,6 +1273,9 @@ void ColumnVariantV2::replace_column_null_data(const uint8_t* __restrict null_ma if (std::none_of(null_map, null_map + size(), [](uint8_t value) { return value != 0; })) { return; } + if (_shredded) { + ensure_encoded(); + } // Hash joins serialize the nested value even for a null-safe NULL key. Normalize those hidden // values to the canonical Variant default so build and probe keys compare byte-for-byte. @@ -1176,6 +1300,9 @@ void ColumnVariantV2::replace_column_null_data(const uint8_t* __restrict null_ma ColumnPtr ColumnVariantV2::filter(const Filter& filter, ssize_t result_size_hint) const { column_match_filter_size(size(), filter.size()); + if (_shredded) { + return ColumnVariantV2::create_shredded(_shredded->filter(filter, result_size_hint)); + } if (_typed) { ColumnPtr filtered = _typed->filter(filter, result_size_hint); auto result = ColumnVariantV2::create(); @@ -1201,6 +1328,13 @@ ColumnPtr ColumnVariantV2::filter(const Filter& filter, ssize_t result_size_hint size_t ColumnVariantV2::filter(const Filter& filter) { column_match_filter_size(size(), filter.size()); + if (_shredded) { + // Scanner-side compaction is a row-selection operation, not a request for canonical + // Variant bytes. Keep partial Parquet projections in their physical representation. + _shredded = _shredded->filter(filter, -1); + _check_invariants(); + return size(); + } if (_typed) { ColumnPtr filtered = static_cast(_typed)->filter(filter, -1); const size_t filtered_size = filtered->size(); @@ -1232,6 +1366,10 @@ MutableColumnPtr ColumnVariantV2::permute(const Permutation& permutation, size_t } } + if (_shredded) { + return _shredded->materialized_column().permute(permutation, limit); + } + if (_typed) { MutableColumnPtr permuted = _typed->permute(permutation, result_size); auto result = ColumnVariantV2::create(); @@ -1257,6 +1395,20 @@ MutableColumnPtr ColumnVariantV2::permute(const Permutation& permutation, size_t } MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { + if (_shredded) { + if (new_size == 0) { + // Empty scanner placeholders carry no rows and therefore need no physical shredded + // state. Avoid forcing a partial leaf projection through full materialization. + return ColumnVariantV2::create(); + } + if (new_size == size()) { + auto result = ColumnVariantV2::create(); + result->_shredded = _shredded; + result->_check_invariants(); + return result; + } + return _shredded->materialized_column().clone_resized(new_size); + } if (_typed) { auto result = ColumnVariantV2::create(); if (new_size <= size()) { @@ -1295,6 +1447,9 @@ MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { void ColumnVariantV2::resize(size_t new_size) { const size_t old_size = size(); + if (_shredded && new_size != old_size) { + ensure_encoded(); + } if (_typed) { if (new_size == old_size) { return; @@ -1357,6 +1512,7 @@ void ColumnVariantV2::_adopt_state_from(ColumnVariantV2& replacement) { _values = std::move(replacement._values); _typed = std::move(replacement._typed); _typed_type = std::move(replacement._typed_type); + _shredded = std::move(replacement._shredded); _check_invariants(); } @@ -1368,6 +1524,14 @@ void ColumnVariantV2::_detach_metadata_for_write() { } void ColumnVariantV2::_check_invariants() const { + if (_shredded) { + DORIS_CHECK(_typed == nullptr) << "shredded state cannot contain a typed column"; + DORIS_CHECK(_typed_type == nullptr) << "shredded state cannot retain a typed data type"; + DORIS_CHECK(_metadatas->empty()) << "shredded state cannot contain encoded metadata"; + DORIS_CHECK(_meta_ids->empty()) << "shredded state cannot contain encoded metadata ids"; + DORIS_CHECK(_values->empty()) << "shredded state cannot contain encoded values"; + return; + } if (_typed) { DORIS_CHECK(_typed_type != nullptr) << "typed state requires a data type"; const IColumn* typed_column = static_cast(_typed).get(); diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index e39bef3454719e..abe4df4b8223d4 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -19,6 +19,8 @@ #include #include +#include +#include #include #include @@ -36,9 +38,58 @@ namespace doris { class DataTypeVariantV2SerDe; class VariantBatchBuilder; +class ColumnVariantV2; -// ColumnVariantV2 stores a whole column in exactly one state: encoded Variant bytes or one nullable -// typed scalar column. Mixed operations materialize the typed state as encoded bytes on demand. +struct VariantShreddedPathSegment { + enum class Kind : uint8_t { OBJECT_KEY, ARRAY_INDEX }; + + Kind kind = Kind::OBJECT_KEY; + StringRef key; + int64_t index = 0; +}; + +struct VariantShreddedTypedValue { + // The state owns the same immutable column. Keeping a ColumnPtr here lets expression results + // retain the decoded leaf without copying it or depending on scanner lifetime. + ColumnPtr column; + DataTypePtr type; +}; + +// Format readers keep their native shredded representation behind this interface. Core Variant +// code sees only logical paths and an explicit late-materialization boundary. +class VariantShreddedState { +public: + virtual ~VariantShreddedState() = default; + + virtual size_t size() const = 0; + virtual size_t byte_size() const = 0; + virtual size_t allocated_bytes() const = 0; + virtual void sanity_check() const = 0; + // Shredded columns are immutable and shared. Expose their physical tree only through the + // immutable callback contract. + virtual void for_each_subcolumn(IColumn::ColumnCallback callback) const = 0; + // Row selection must remain in the native shredded representation. A scanner may compact a + // predicate column before every logical Variant value is available for materialization. + virtual std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t result_size_hint) const = 0; + virtual std::shared_ptr select_range(size_t start, + size_t length) const = 0; + virtual std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; + // Appends another state only when both format-owned physical layouts have identical semantics. + // An incompatible source must leave this state unchanged and return false. + virtual bool try_append(const VariantShreddedState& source) = 0; + virtual std::optional find_typed_value( + std::span path) const = 0; + + // The returned column is cached and owned by this state, so borrowed VariantRef values remain + // valid for the state lifetime. Implementations must not materialize before this is called. + virtual const ColumnVariantV2& materialized_column() const = 0; +}; + +// ColumnVariantV2 stores a whole column in exactly one state: encoded Variant bytes, one nullable +// typed scalar column, or a format-owned shredded tree. Mixed operations materialize typed or +// shredded state as encoded bytes only when canonical row bytes are required. class ColumnVariantV2 final : public COWHelper { public: struct EncodedDataView { @@ -87,10 +138,14 @@ class ColumnVariantV2 final : public COWHelper { // The input must be an exact, non-Const ColumnNullable whose nested column matches the // non-nullable supported scalar type. static MutablePtr create_typed(ColumnPtr column, DataTypePtr scalar_type); + static MutablePtr create_shredded(std::shared_ptr state); bool is_typed() const noexcept { return _typed != nullptr; } + bool is_shredded() const noexcept { return _shredded != nullptr; } const IColumn& typed_column() const; const DataTypePtr& typed_type() const; + std::optional find_shredded_typed_value( + std::span path) const; void ensure_encoded(); ReadView read_view() const; @@ -197,6 +252,10 @@ class ColumnVariantV2 final : public COWHelper { // single type described by _typed_type. IColumn::WrappedPtr _typed; DataTypePtr _typed_type; + + // A non-null state owns the decoded format columns. Encoded and typed storage stay empty until + // an operation explicitly requests canonical Variant bytes. + std::shared_ptr _shredded; }; template diff --git a/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp b/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp index 3ce60aed8dd7c7..5e276fd0279220 100644 --- a/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp @@ -89,6 +89,9 @@ const DataTypePtr& ColumnVariantV2::ReadView::typed_type() const { } ColumnVariantV2::ReadView ColumnVariantV2::read_view() const { + if (_shredded) { + return _shredded->materialized_column().read_view(); + } if (_typed) { DORIS_CHECK(_typed_type != nullptr) << "typed state requires a data type"; return {static_cast(_typed).get(), &_typed_type}; diff --git a/be/src/exec/operator/file_scan_operator.cpp b/be/src/exec/operator/file_scan_operator.cpp index 370f77bed8b1c8..5290e3078fb171 100644 --- a/be/src/exec/operator/file_scan_operator.cpp +++ b/be/src/exec/operator/file_scan_operator.cpp @@ -19,8 +19,14 @@ #include +#include #include +#include "core/assert_cast.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" #include "exec/operator/olap_scan_operator.h" #include "exec/operator/scan_operator.h" #include "exec/scan/file_scanner.h" @@ -31,6 +37,29 @@ #include "storage/tablet/tablet_manager.h" namespace doris { +namespace { + +bool contains_variant_type(const DataTypePtr& input) { + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + +} // namespace PushDownType FileScanLocalState::_should_push_down_binary_predicate( VectorizedFnCall* fn_call, VExprContext* expr_ctx, Field& constant_val, @@ -163,6 +192,20 @@ Status FileScanLocalState::_init_scanners(std::list* scanners) { const bool use_file_scanner_v2 = _should_use_file_scanner_v2(state()->query_options(), is_load, *scan_params); _operator_profile->add_info_string("UseScannerV2", use_file_scanner_v2 ? "true" : "false"); + const auto* output_tuple_desc = state()->desc_tbl().get_tuple_descriptor(_output_tuple_id); + DORIS_CHECK(output_tuple_desc != nullptr); + const bool metadata_only_count = + is_count_star_pushdown() && _split_source->all_ranges_have_table_level_row_count(); + if (!is_load && !use_file_scanner_v2 && !metadata_only_count && + std::ranges::any_of(output_tuple_desc->slots(), [](const SlotDescriptor* slot) { + return contains_variant_type(slot->get_data_type_ptr()); + })) { + // A syntactic COUNT(*) alone is insufficient: every assigned range must prove that the + // legacy scanner will emit metadata counts without decoding a Variant carrier. + return Status::NotSupported( + "External VARIANT columns require FileScannerV2; the legacy file scanner does " + "not support VARIANT"); + } for (int i = 0; i < _max_scanners; ++i) { ScannerSPtr scanner; if (use_file_scanner_v2) { diff --git a/be/src/exec/operator/scan_operator.h b/be/src/exec/operator/scan_operator.h index e11eb506a772ab..cb9effb23882ce 100644 --- a/be/src/exec/operator/scan_operator.h +++ b/be/src/exec/operator/scan_operator.h @@ -77,6 +77,18 @@ class ScanLocalStateBase : public PipelineXLocalState<> { virtual TPushAggOp::type get_push_down_agg_type() = 0; virtual const std::optional>& get_push_down_count_slot_ids() const = 0; + static bool is_count_star_pushdown(TPushAggOp::type agg_type, + const std::optional>& count_slot_ids) { + // An absent argument field is an old plan with unknown semantics. Only an explicitly empty + // argument list proves COUNT(*)/COUNT(1) and permits placeholder slots to be ignored. + return agg_type == TPushAggOp::type::COUNT && count_slot_ids.has_value() && + count_slot_ids->empty(); + } + + bool is_count_star_pushdown() { + return is_count_star_pushdown(get_push_down_agg_type(), get_push_down_count_slot_ids()); + } + // If scan operator is serial operator(like topn), its real parallelism is 1. // Otherwise, its real parallelism is query_parallel_instance_num. // query_parallel_instance_num of olap table is usually equal to session var parallel_pipeline_task_num. diff --git a/be/src/exec/scan/access_path_parser.cpp b/be/src/exec/scan/access_path_parser.cpp index c294f86f10437d..0d0a5b0a547405 100644 --- a/be/src/exec/scan/access_path_parser.cpp +++ b/be/src/exec/scan/access_path_parser.cpp @@ -189,6 +189,20 @@ void insert_access_path(AccessPathNode* root, const std::vector& pa insert_access_path(&root->children[path[path_idx]], path, path_idx + 1); } +void collect_variant_access_paths(const AccessPathNode& node, std::vector* path, + std::vector>* result) { + DORIS_CHECK(path != nullptr && result != nullptr); + for (const auto& [segment, child] : node.children) { + path->push_back(segment); + if (child.project_all || child.children.empty()) { + result->push_back(*path); + } else { + collect_variant_access_paths(child, path, result); + } + path->pop_back(); + } +} + Status build_nested_children_from_access_node(format::ColumnDefinition* column, const DataTypePtr& type, const AccessPathNode& node, const std::string& path, @@ -445,6 +459,19 @@ Status build_nested_children_from_access_node(format::ColumnDefinition* column, return build_map_children_from_access_node( column, assert_cast(*nested_type), node, path, schema_column, prefer_exact_name_match); + case TYPE_VARIANT: { + // A Variant nested below STRUCT/ARRAY/MAP owns paths relative to this terminal. Keeping + // them on the nested ColumnDefinition lets ColumnMapper select the same physical leaves + // as a root Variant without flattening away the surrounding container. + column->variant_access_paths.clear(); + std::vector variant_path; + collect_variant_access_paths(node, &variant_path, &column->variant_access_paths); + std::ranges::sort(column->variant_access_paths); + column->variant_access_paths.erase(std::unique(column->variant_access_paths.begin(), + column->variant_access_paths.end()), + column->variant_access_paths.end()); + return Status::OK(); + } default: return Status::NotSupported("AccessPathParser does not support access path {} for slot {}", path, column->name); @@ -461,6 +488,44 @@ Status AccessPathParser::build_nested_children(format::ColumnDefinition* column, if (is_scanner_materialized_virtual_column(column->name)) { return Status::OK(); } + if (remove_nullable(column->type)->get_primitive_type() == TYPE_VARIANT) { + column->variant_access_paths.clear(); + for (const auto& access_path : access_paths) { + if (access_path.type != TAccessPathType::DATA || + !access_path.__isset.data_access_path) { + return Status::NotSupported( + "AccessPathParser only supports DATA access paths for Variant slot {}", + column->name); + } + const auto& path = access_path.data_access_path.path; + if (path.empty()) { + // Match the generic access-path tree: an empty DATA path denotes the whole slot + // and dominates every narrower Variant path in the same request. + column->variant_access_paths.clear(); + return Status::OK(); + } + int32_t top_level_id = -1; + if (to_lower(path.front()) != to_lower(column->name) && + (!parse_non_negative_int(path.front(), &top_level_id) || + !column->has_identifier_field_id() || + top_level_id != column->get_identifier_field_id())) { + return Status::NotSupported( + "AccessPathParser access path {} does not match Variant slot {}", + access_path_to_string(path), column->name); + } + if (path.size() == 1) { + // A whole-root access covers every subpath and must disable physical leaf pruning. + column->variant_access_paths.clear(); + return Status::OK(); + } + column->variant_access_paths.emplace_back(path.begin() + 1, path.end()); + } + std::ranges::sort(column->variant_access_paths); + column->variant_access_paths.erase(std::unique(column->variant_access_paths.begin(), + column->variant_access_paths.end()), + column->variant_access_paths.end()); + return Status::OK(); + } if (!is_complex_type(remove_nullable(column->type)->get_primitive_type())) { return Status::OK(); } @@ -506,8 +571,36 @@ Status AccessPathParser::build_nested_children(format::ColumnDefinition* column, bool prefer_exact_name_match) { DORIS_CHECK(column != nullptr); DORIS_CHECK(slot_desc != nullptr); - return build_nested_children(column, slot_desc->all_access_paths(), schema_column, + return build_nested_children(column, slot_desc->all_access_paths(), + slot_desc->predicate_access_paths(), schema_column, prefer_exact_name_match); } +Status AccessPathParser::build_nested_children( + format::ColumnDefinition* column, const std::vector& all_access_paths, + const std::vector& predicate_access_paths, + const format::ColumnDefinition* schema_column, bool prefer_exact_name_match) { + DORIS_CHECK(column != nullptr); + auto predicate_column = *column; + RETURN_IF_ERROR(build_nested_children(column, all_access_paths, schema_column, + prefer_exact_name_match)); + column->has_predicate_access_paths = !predicate_access_paths.empty(); + column->predicate_children.clear(); + column->predicate_variant_access_paths.clear(); + if (predicate_access_paths.empty()) { + return Status::OK(); + } + + predicate_column.children.clear(); + predicate_column.variant_access_paths.clear(); + predicate_column.has_predicate_access_paths = false; + predicate_column.predicate_children.clear(); + predicate_column.predicate_variant_access_paths.clear(); + RETURN_IF_ERROR(build_nested_children(&predicate_column, predicate_access_paths, schema_column, + prefer_exact_name_match)); + column->predicate_children = std::move(predicate_column.children); + column->predicate_variant_access_paths = std::move(predicate_column.variant_access_paths); + return Status::OK(); +} + } // namespace doris diff --git a/be/src/exec/scan/access_path_parser.h b/be/src/exec/scan/access_path_parser.h index 0be785a33906e8..650993aca08b27 100644 --- a/be/src/exec/scan/access_path_parser.h +++ b/be/src/exec/scan/access_path_parser.h @@ -38,6 +38,12 @@ class AccessPathParser { const std::vector& access_paths, const format::ColumnDefinition* schema_column, bool prefer_exact_name_match = true); + + static Status build_nested_children( + format::ColumnDefinition* column, + const std::vector& all_access_paths, + const std::vector& predicate_access_paths, + const format::ColumnDefinition* schema_column, bool prefer_exact_name_match = true); }; } // namespace doris diff --git a/be/src/exec/scan/file_scanner_v2.cpp b/be/src/exec/scan/file_scanner_v2.cpp index 052d465daa2005..a0840b7ae50024 100644 --- a/be/src/exec/scan/file_scanner_v2.cpp +++ b/be/src/exec/scan/file_scanner_v2.cpp @@ -36,7 +36,10 @@ #include "core/block/column_with_type_and_name.h" #include "core/column/column.h" #include "core/data_type/data_type.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" #include "core/data_type_serde/data_type_serde.h" #include "core/string_ref.h" #include "exec/common/util.hpp" @@ -79,6 +82,8 @@ namespace { constexpr int kIcebergPositionDeleteContent = 1; constexpr int kIcebergDeletionVectorContent = 3; +std::string table_format_name(const TFileRangeDesc& range); + std::string table_format_name(const TFileRangeDesc& range) { return range.__isset.table_format_params ? range.table_format_params.table_format_type : "NotSet"; @@ -89,6 +94,26 @@ TFileFormatType::type get_range_format_type(const TFileScanRangeParams& params, return range.__isset.format_type ? range.format_type : params.format_type; } +bool contains_variant_type(const DataTypePtr& input) { + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + bool is_supported_table_format(const TFileRangeDesc& range) { const auto table_format = table_format_name(range); if (table_format == "hudi" && range.__isset.table_format_params && @@ -486,6 +511,13 @@ Status FileScannerV2::_filter_output_block(Block* block) { _get_current_format_type()); } +bool FileScannerV2::_can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const { + // A Variant access expression is evaluated above the file reader. Keep each file-local + // shredded schema intact until that projection turns complete and leaf-only states into a + // common logical result column. + return !_has_variant_projection; +} + Status FileScannerV2::_contextualize_output_filter_status(Status status, TFileFormatType::type format_type) { if (!status.ok() && format_type == TFileFormatType::FORMAT_ORC) { @@ -672,8 +704,9 @@ Status FileScannerV2::_create_table_reader_for_format( Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range, std::map partition_values) { + const auto format_type = get_range_format_type(*_params, range); format::FileFormat current_split_format; - RETURN_IF_ERROR(_to_file_format(get_range_format_type(*_params, range), ¤t_split_format)); + RETURN_IF_ERROR(_to_file_format(format_type, ¤t_split_format)); VExprContextSPtrs conjuncts; RETURN_IF_ERROR(_build_table_conjuncts(&conjuncts)); VExprContextSPtrs partition_prune_conjuncts; @@ -797,6 +830,7 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ _projected_columns.clear(); _projected_columns.reserve(_params->required_slots.size()); _need_global_rowid_column = false; + _has_variant_projection = false; format::ProjectedColumnBuildContext build_context { .scan_params = _params, .range = &_current_range, @@ -814,6 +848,7 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ slot_info.slot_id); } auto column = _build_table_column(it->second); + _has_variant_projection = _has_variant_projection || contains_variant_type(column.type); build_context.slot_desc = it->second; if (column.name.starts_with(BeConsts::GLOBAL_ROWID_COL)) { _need_global_rowid_column = true; diff --git a/be/src/exec/scan/file_scanner_v2.h b/be/src/exec/scan/file_scanner_v2.h index 03e5f4d6bbc1a0..3ccc4e075ef209 100644 --- a/be/src/exec/scan/file_scanner_v2.h +++ b/be/src/exec/scan/file_scanner_v2.h @@ -114,6 +114,7 @@ class FileScannerV2 final : public Scanner { protected: Status _get_block_impl(RuntimeState* state, Block* block, bool* eof) override; + bool _can_merge_padding_blocks(const Block& left, const Block& right) const override; Status _filter_output_block(Block* block) override; void _collect_profile_before_close() override; bool _should_update_load_counters() const override; @@ -190,6 +191,7 @@ class FileScannerV2 final : public Scanner { // the reader is format-specific, so it is rebuilt whenever this stops matching the range. std::string _table_reader_format; std::vector _projected_columns; + bool _has_variant_projection = false; // File formats without embedded schema, such as CSV, still need the FE slot descriptors in // file-column order. This mirrors old FileScanner::_file_slot_descs and is passed only to // readers that cannot derive their schema from file metadata. diff --git a/be/src/exec/scan/scanner.cpp b/be/src/exec/scan/scanner.cpp index cbdf5fa7eeec55..e30543b9343613 100644 --- a/be/src/exec/scan/scanner.cpp +++ b/be/src/exec/scan/scanner.cpp @@ -88,6 +88,15 @@ Status Scanner::get_block_after_projects(RuntimeState* state, Block* block, bool auto& row_descriptor = _local_state->_parent->row_descriptor(); if (_output_row_descriptor) { _origin_block.clear_column_data(row_descriptor.num_materialized_slots()); + if (!_can_merge_padding_blocks(_padding_block, _origin_block)) { + DORIS_CHECK(_padding_block.empty()) + << "padding policy must remain stable for one scanner"; + // Some physical columns carry file-local state that an upper projection must consume + // before the next split is read. Padding those blocks first would make correctness + // depend on whether two file tails happen to share one output batch. + RETURN_IF_ERROR(get_block(state, &_origin_block, eos)); + return _do_projections(&_origin_block, block); + } const auto min_batch_size = std::max(state->batch_size() / 2, 1); const auto block_max_bytes = state->preferred_block_size_bytes(); while (_padding_block.rows() < min_batch_size && _padding_block.bytes() < block_max_bytes && diff --git a/be/src/exec/scan/scanner.h b/be/src/exec/scan/scanner.h index 1bf4cdb5bb2ea1..75e583ecdfce61 100644 --- a/be/src/exec/scan/scanner.h +++ b/be/src/exec/scan/scanner.h @@ -123,6 +123,10 @@ class Scanner { // Subclass should implement this to return data. virtual Status _get_block_impl(RuntimeState* state, Block* block, bool* eof) = 0; + virtual bool _can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const { + return true; + } + Status _merge_padding_block() { if (_padding_block.empty()) { _padding_block.swap(_origin_block); diff --git a/be/src/exec/scan/split_source_connector.h b/be/src/exec/scan/split_source_connector.h index 5926baff303cbf..f13190cf0641c2 100644 --- a/be/src/exec/scan/split_source_connector.h +++ b/be/src/exec/scan/split_source_connector.h @@ -17,6 +17,8 @@ #pragma once +#include + #include "common/config.h" #include "core/custom_allocator.h" #include "runtime/runtime_state.h" @@ -45,6 +47,8 @@ class SplitSourceConnector { virtual TFileScanRangeParams* get_params() = 0; + virtual bool all_ranges_have_table_level_row_count() const { return false; } + protected: template , typename V2 = std::vector> requires(std::is_same_v, @@ -125,6 +129,19 @@ class LocalSplitSourceConnector : public SplitSourceConnector { throw Exception( Status::FatalError("Unreachable, params is got by file_scan_range_params_map")); } + + bool all_ranges_have_table_level_row_count() const override { + // Every assigned range must carry a proven count; one fallback range would still require + // decoding the projected carrier through the selected scanner. + return !_scan_ranges.empty() && std::ranges::all_of(_scan_ranges, [](const auto& params) { + const auto& ranges = params.scan_range.ext_scan_range.file_scan_range.ranges; + return !ranges.empty() && std::ranges::all_of(ranges, [](const auto& range) { + return range.__isset.table_format_params && + range.table_format_params.__isset.table_level_row_count && + range.table_format_params.table_level_row_count >= 0; + }); + }); + } }; /** diff --git a/be/src/exec/sink/viceberg_merge_sink.cpp b/be/src/exec/sink/viceberg_merge_sink.cpp index 5ff5a0a1f28150..5008b217228cd2 100644 --- a/be/src/exec/sink/viceberg_merge_sink.cpp +++ b/be/src/exec/sink/viceberg_merge_sink.cpp @@ -55,13 +55,15 @@ VIcebergMergeSink::~VIcebergMergeSink() = default; Status VIcebergMergeSink::init_properties(ObjectPool* pool, const RowDescriptor& row_desc) { RETURN_IF_ERROR(_build_inner_sinks()); - _table_writer = std::make_unique(_table_sink, _table_output_expr_ctxs, - nullptr, nullptr); - _table_writer->defer_file_cleanup_until_outer_close(); + if (_writes_data_files) { + _table_writer = std::make_unique(_table_sink, _table_output_expr_ctxs, + nullptr, nullptr); + _table_writer->defer_file_cleanup_until_outer_close(); + RETURN_IF_ERROR(_table_writer->init_properties(pool, row_desc)); + } _delete_writer = std::make_unique(_delete_sink, _delete_output_expr_ctxs, nullptr, nullptr); _delete_writer->defer_file_cleanup_until_outer_close(); - RETURN_IF_ERROR(_table_writer->init_properties(pool, row_desc)); RETURN_IF_ERROR(_delete_writer->init_properties(pool)); return Status::OK(); } @@ -88,10 +90,13 @@ Status VIcebergMergeSink::open(RuntimeState* state, RuntimeProfile* profile) { RETURN_IF_ERROR(_prepare_output_layout()); - RuntimeProfile* table_profile = profile->create_child("IcebergMergeTableWriter", true, true); RuntimeProfile* delete_profile = profile->create_child("IcebergMergeDeleteWriter", true, true); - RETURN_IF_ERROR(_table_writer->open(state, table_profile)); + if (_table_writer) { + RuntimeProfile* table_profile = + profile->create_child("IcebergMergeTableWriter", true, true); + RETURN_IF_ERROR(_table_writer->open(state, table_profile)); + } RETURN_IF_ERROR(_delete_writer->open(state, delete_profile)); return Status::OK(); @@ -153,6 +158,13 @@ Status VIcebergMergeSink::write(RuntimeState* state, Block& block) { _delete_row_count += delete_rows; _insert_row_count += insert_rows; + // A delete-only plan deliberately omits the data writer so Variant target schemas never enter + // the unsupported Iceberg data-write path. Reject a mismatched FE plan before dereferencing it. + if (has_insert && !_writes_data_files) { + return Status::InternalError( + "Iceberg delete-only merge sink received a data insert operation"); + } + bool skip_io = false; #ifdef BE_TEST skip_io = _skip_io; @@ -338,6 +350,8 @@ Status VIcebergMergeSink::_build_inner_sinks() { } const auto& merge_sink = _t_sink.iceberg_merge_sink; + // An old FE cannot produce delete-only plans, so an unset flag retains its data-writer path. + _writes_data_files = !merge_sink.__isset.writes_data_files || merge_sink.writes_data_files; // Missing means an old FE plan, which predates SQL MERGE cardinality validation. _require_merge_cardinality_check = merge_sink.__isset.require_merge_cardinality_check && merge_sink.require_merge_cardinality_check; diff --git a/be/src/exec/sink/viceberg_merge_sink.h b/be/src/exec/sink/viceberg_merge_sink.h index f3733a3318230e..88e9ef89121b74 100644 --- a/be/src/exec/sink/viceberg_merge_sink.h +++ b/be/src/exec/sink/viceberg_merge_sink.h @@ -78,6 +78,7 @@ class VIcebergMergeSink final : public AsyncResultWriter { std::vector _data_column_indices; std::map _matched_row_positions; size_t _matched_row_id_state_size = sizeof(std::map); + bool _writes_data_files = true; bool _require_merge_cardinality_check = false; VExprContextSPtrs _table_output_expr_ctxs; diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 644a033f4af356..90863fe86c5c72 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -131,6 +131,42 @@ Status extract_encoded_variant_element(const ColumnVariantV2& source, const ResolvedVariantElementV2Path& path, std::span outer_nulls, ColumnPtr* output); +std::optional extract_shredded_typed_variant_element( + const ColumnVariantV2& source, const ResolvedVariantElementV2Path& path, + std::span outer_nulls) { + DorisVector shredded_path; + shredded_path.reserve(path.size()); + for (size_t position = 0; position < path.size(); ++position) { + VariantShreddedPathSegment segment; + if (path.kind_at(position) == VariantElementV2PathSegment::Kind::OBJECT_KEY) { + segment.kind = VariantShreddedPathSegment::Kind::OBJECT_KEY; + segment.key = path.object_key_at(position); + } else { + segment.kind = VariantShreddedPathSegment::Kind::ARRAY_INDEX; + segment.index = path.array_index_at(position); + } + shredded_path.push_back(segment); + } + + auto match = source.find_shredded_typed_value(shredded_path); + if (!match.has_value()) { + return std::nullopt; + } + const auto& leaf = assert_cast(*match->column); + auto nulls = leaf.get_null_map_column().clone_resized(source.size()); + auto& null_data = assert_cast(*nulls).get_data(); + for (size_t row = 0; row < source.size(); ++row) { + null_data[row] = + static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); + } + + // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null + // map is produced here, so predicates and casts can consume the leaf without reconstructing + // canonical Variant rows. + auto values = ColumnVariantV2::create_typed(match->column, match->type); + return ColumnNullable::create(std::move(values), std::move(nulls)); +} + Status make_all_null_variant_element_result(size_t rows, ColumnPtr* output); } // namespace @@ -217,7 +253,14 @@ Status extract_variant_element_v2(const ColumnVariantV2& source, ColumnPtr candidate; try { - if (!source.is_typed()) { + if (source.is_shredded()) { + if (auto typed = extract_shredded_typed_variant_element(source, path, outer_nulls)) { + candidate = std::move(*typed); + } else { + RETURN_IF_ERROR( + extract_encoded_variant_element(source, path, outer_nulls, &candidate)); + } + } else if (!source.is_typed()) { RETURN_IF_ERROR(extract_encoded_variant_element(source, path, outer_nulls, &candidate)); } else { // A typed Variant is one scalar root value per row. String payloads are strings, not diff --git a/be/src/format_v2/column_data.h b/be/src/format_v2/column_data.h index e122b48555113a..40f360e29d01f2 100644 --- a/be/src/format_v2/column_data.h +++ b/be/src/format_v2/column_data.h @@ -261,6 +261,17 @@ struct ColumnDefinition { // Full table-schema identity subtree before access-path pruning. ID-less physical complex // wrappers must be discovered from this view without adding unrequested children to output. std::vector identity_children {}; + // Logical object-key paths requested from a Variant column. An empty collection means the + // whole Variant is required; non-empty paths may be resolved to format-specific shredded + // physical children after the per-file schema is known. + std::vector> variant_access_paths {}; + // Predicate access paths are kept separately from the final union projection. File Scanner V2 + // can lower this smaller semantic tree to an eager predicate projection while deferring the + // final children until rows survive. The flag distinguishes no predicate metadata from a + // whole-root predicate, whose child/path collections are intentionally empty. + bool has_predicate_access_paths = false; + std::vector predicate_children {}; + std::vector> predicate_variant_access_paths {}; // Expression used to materialize missing/default/generated values when the column is not read // directly from the file. VExprContextSPtr default_expr = nullptr; @@ -371,6 +382,19 @@ struct LocalColumnIndex { std::string debug_string() const; }; +inline bool same_local_column_index(const LocalColumnIndex& lhs, const LocalColumnIndex& rhs) { + if (lhs.index != rhs.index || lhs.project_all_children != rhs.project_all_children || + lhs.children.size() != rhs.children.size()) { + return false; + } + for (size_t i = 0; i < lhs.children.size(); ++i) { + if (!same_local_column_index(lhs.children[i], rhs.children[i])) { + return false; + } + } + return true; +} + inline bool is_full_projection(const LocalColumnIndex* projection) { return projection == nullptr || projection->project_all_children; } diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index 6da3b9b9cd6198..7e48255e0fcdb1 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -18,6 +18,7 @@ #include "format_v2/column_mapper.h" #include +#include #include #include #include @@ -1278,6 +1279,12 @@ static bool needs_projected_file_type_rebuild(const ColumnMapping& mapping) { remove_nullable(mapping.table_type)->get_primitive_type()) { return true; } + if (remove_nullable(mapping.file_type)->get_primitive_type() == TYPE_STRUCT && + mapping.child_mappings.size() != mapping.original_file_children.size()) { + // A predicate access path keeps the parent Struct type but intentionally carries only the + // referenced child descriptors; type equality alone must not restore the pruned siblings. + return true; + } if (!mapping.table_type->equals(*mapping.file_type)) { return true; } @@ -1646,17 +1653,24 @@ static bool has_projected_file_children(const ColumnMapping& mapping) { return false; } -static bool needs_nested_file_projection(const ColumnMapping& mapping) { +static bool needs_nested_file_projection(const ColumnMapping& mapping, + bool include_variant_access_paths = false) { if (has_projected_file_children(mapping)) { // Return True if the projected child column is missing / re-ordered return true; } - return std::ranges::any_of(mapping.child_mappings, [](const ColumnMapping& child_mapping) { - return needs_nested_file_projection(child_mapping); - }); + if (include_variant_access_paths && !mapping.variant_access_paths.empty()) { + return true; + } + return std::ranges::any_of( + mapping.child_mappings, [include_variant_access_paths](const ColumnMapping& child) { + return needs_nested_file_projection(child, include_variant_access_paths); + }); } -static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection); +static bool build_variant_projection(const ColumnMapping& mapping, LocalColumnIndex* projection); +static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection, + bool enable_variant_leaf_projection = false); // Build the projected file children/type according to the pruned complex projection. For example, // if we have a struct column `s` with children `id` and `name`, and the projection only keeps @@ -1694,11 +1708,15 @@ static Status rebuild_projected_file_children_and_type( // projected output shape; file readers still read full keys to construct ColumnMap offsets and keep // key semantics unchanged. If a caller tries to project only/prune the key child, the common schema // projection helper rejects it. -static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection) { +static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection, + bool enable_variant_leaf_projection) { if (projection == nullptr) { return Status::InvalidArgument("projection is null"); } DORIS_CHECK(mapping.file_local_id.has_value()); + if (enable_variant_leaf_projection && build_variant_projection(mapping, projection)) { + return Status::OK(); + } *projection = LocalColumnIndex::local(*mapping.file_local_id); projection->project_all_children = mapping.child_mappings.empty(); projection->children.clear(); @@ -1712,7 +1730,8 @@ static Status build_complex_projection(const ColumnMapping& mapping, LocalColumn } for (const auto* child_mapping : present_children) { LocalColumnIndex child_projection; - RETURN_IF_ERROR(build_complex_projection(*child_mapping, &child_projection)); + RETURN_IF_ERROR(build_complex_projection(*child_mapping, &child_projection, + enable_variant_leaf_projection)); projection->children.push_back(std::move(child_projection)); } if (!projection->project_all_children && projection->children.empty()) { @@ -1722,8 +1741,6 @@ static Status build_complex_projection(const ColumnMapping& mapping, LocalColumn return Status::OK(); } -using FilterProjectionMap = std::map; - // Update the mapping's file type according to the projection, and determine whether the projection // is trivial (i.e. the projected file type is the same as the table type, so no need to // rematerialize the complex value back to table layout after reading from file). @@ -1748,50 +1765,148 @@ static Status apply_projection_to_mapping_file_type(const LocalColumnIndex& proj return Status::OK(); } -static Status merge_filter_projection(const FilterProjectionMap* filter_projections, - LocalColumnIndex* projection) { - DORIS_CHECK(projection != nullptr); - if (filter_projections == nullptr) { - return Status::OK(); +static const ColumnDefinition* find_file_child_by_name( + const std::vector& children, std::string_view name) { + const auto child_it = std::ranges::find_if( + children, [name](const ColumnDefinition& child) { return child.name == name; }); + return child_it == children.end() ? nullptr : &*child_it; +} + +static bool variant_leaf_type_preserves_physical_identity(const ColumnDefinition& leaf) { + if (!leaf.children.empty() || leaf.type == nullptr) { + return false; } - const auto filter_projection_it = filter_projections->find(projection->column_id()); - if (filter_projection_it == filter_projections->end()) { - return Status::OK(); + // ColumnDefinition does not transport Parquet's raw-binary/UUID and timestamp-unit tags. + // Limit direct leaves to identities fully described by the Doris scalar type; every ambiguous + // identity must retain the complete wrapper so reconstruction can inspect its physical schema. + switch (remove_nullable(leaf.type)->get_primitive_type()) { + case TYPE_BOOLEAN: + case TYPE_TINYINT: + case TYPE_SMALLINT: + case TYPE_INT: + case TYPE_BIGINT: + case TYPE_FLOAT: + case TYPE_DOUBLE: + case TYPE_DECIMAL128I: + case TYPE_DATEV2: + return true; + default: + return false; } - // Merge predicate-only nested paths into the root projection that is about to be scanned. - // Example: `SELECT s.a WHERE s.b > 1` first builds the output projection `s -> a` from - // ColumnMapping, while build_nested_struct_filter_projection_map() records `s -> b`. This merge - // produces one file scan projection `s -> a,b`. - RETURN_IF_ERROR(merge_local_column_index(projection, filter_projection_it->second)); - return Status::OK(); } -static bool table_root_is_map(const ColumnMapping& mapping) { - if (mapping.table_type == nullptr) { +static bool build_variant_leaf_path_projection(const ColumnMapping& mapping, + const std::vector& path, + LocalColumnIndex* root_projection) { + DORIS_CHECK(root_projection != nullptr); + const auto is_numeric_selector = [](std::string_view value) { + if (value.empty()) { + return false; + } + const size_t digits_begin = value.front() == '+' || value.front() == '-' ? 1 : 0; + return digits_begin < value.size() && + std::ranges::all_of(value.substr(digits_begin), + [](unsigned char c) { return std::isdigit(c); }); + }; + if (path.size() != 1 || path[0].empty() || path[0] == "NULL" || + path[0].find('.') != std::string::npos || is_numeric_selector(path[0]) || + !mapping.file_local_id.has_value()) { + // Thrift currently carries access paths as strings without segment-kind or escaping + // metadata. Signed numeric tokens are therefore also ambiguous between an array selector + // and an object key, so only a single unambiguous key can be mapped losslessly to a leaf. + return false; + } + *root_projection = LocalColumnIndex::partial_local(*mapping.file_local_id); + const auto* root_typed = find_file_child_by_name(mapping.original_file_children, "typed_value"); + if (root_typed == nullptr || root_typed->children.empty() || root_typed->type == nullptr || + remove_nullable(root_typed->type)->get_primitive_type() != TYPE_STRUCT) { return false; } - return remove_nullable(mapping.table_type)->get_primitive_type() == TYPE_MAP; + root_projection->children.push_back( + LocalColumnIndex::partial_local(root_typed->file_local_id())); + auto* current_projection = &root_projection->children.back(); + const auto* typed_children = &root_typed->children; + for (size_t position = 0; position < path.size(); ++position) { + const auto* wrapper = find_file_child_by_name(*typed_children, path[position]); + if (wrapper == nullptr) { + return false; + } + current_projection->children.push_back( + LocalColumnIndex::partial_local(wrapper->file_local_id())); + current_projection = ¤t_projection->children.back(); + const auto* typed = find_file_child_by_name(wrapper->children, "typed_value"); + if (typed == nullptr) { + return false; + } + auto typed_projection = LocalColumnIndex::partial_local(typed->file_local_id()); + const bool leaf = position + 1 == path.size(); + if (leaf) { + // Only primitive typed values can be returned as a direct vector. Complex shredded + // values still need their wrapper shape and therefore keep the full Variant fallback. + if (!variant_leaf_type_preserves_physical_identity(*typed)) { + return false; + } + typed_projection.project_all_children = true; + } + current_projection->children.push_back(std::move(typed_projection)); + current_projection = ¤t_projection->children.back(); + typed_children = &typed->children; + } + return true; } -static Status add_scan_column(FileScanRequest* file_request, ColumnMapping* mapping, - bool is_predicate_column, bool force_full_complex_scan_projection, - const FilterProjectionMap* filter_projections = nullptr) { +static bool build_variant_projection(const ColumnMapping& mapping, LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); + if (mapping.table_type == nullptr || mapping.variant_access_paths.empty() || + remove_nullable(mapping.table_type)->get_primitive_type() != TYPE_VARIANT) { + return false; + } + std::optional merged; + for (const auto& path : mapping.variant_access_paths) { + LocalColumnIndex path_projection; + if (!build_variant_leaf_path_projection(mapping, path, &path_projection)) { + return false; + } + if (!merged.has_value()) { + merged = std::move(path_projection); + } else if (!merge_local_column_index(&*merged, path_projection).ok()) { + return false; + } + } + if (!merged.has_value()) { + return false; + } + *projection = std::move(*merged); + return true; +} + +static Status build_scan_projection(ColumnMapping* mapping, bool force_full_complex_scan_projection, + bool enable_variant_leaf_projection, + LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); const auto file_column_id = LocalColumnId(mapping->file_local_id.value()); - LocalColumnIndex projection = LocalColumnIndex::top_level(file_column_id); + *projection = LocalColumnIndex::top_level(file_column_id); // Columnar readers can turn a complex mapping into a nested file projection, but // row-oriented readers must scan the full top-level complex field because all children are // encoded in the same text cell. - if (!force_full_complex_scan_projection && needs_nested_file_projection(*mapping)) { - RETURN_IF_ERROR(build_complex_projection(*mapping, &projection)); - } - if (is_predicate_column && !force_full_complex_scan_projection) { - DCHECK(filter_projections != nullptr); - // If a projected complex root is also used by a predicate, rebuild the predicate scan - // projection from the output mapping before merging predicate-only children. For - // `SELECT s.a WHERE s.b > 1`, build_complex_projection() produces `s -> a` and - // merge_filter_projection() adds `s -> b`, so the predicate column reads both children. - RETURN_IF_ERROR(merge_filter_projection(filter_projections, &projection)); + if (enable_variant_leaf_projection && !force_full_complex_scan_projection && + build_variant_projection(*mapping, projection)) { + // The per-file Parquet reader will validate residual-value statistics before honoring this + // physical leaf projection; unsafe files atomically fall back to the complete Variant. + } else if (!force_full_complex_scan_projection && + needs_nested_file_projection(*mapping, enable_variant_leaf_projection)) { + RETURN_IF_ERROR( + build_complex_projection(*mapping, projection, enable_variant_leaf_projection)); } + return Status::OK(); +} + +static Status add_scan_column(FileScanRequest* file_request, ColumnMapping* mapping, + bool is_predicate_column, bool force_full_complex_scan_projection, + bool enable_variant_leaf_projection) { + LocalColumnIndex projection; + RETURN_IF_ERROR(build_scan_projection(mapping, force_full_complex_scan_projection, + enable_variant_leaf_projection, &projection)); FileScanRequestBuilder builder(file_request); if (is_predicate_column) { return builder.add_predicate_column(std::move(projection)); @@ -1812,18 +1927,21 @@ static const LocalColumnIndex* find_scan_projection( // mapping.file_type/projected_file_children from the original file schema to the exact shape that // FileReader will return. // -// Example: for `SELECT s.a WHERE s.b > 1`, add_scan_column() keeps only one predicate scan -// projection `s -> a,b`. Applying that projection changes the mapping's file type from the full -// file struct `s` to the projected file struct `s`, so later filter rewrite and -// TableReader final materialization use the same column shape as the file-local block. +// Applying the selected projection changes a mapping's file type to the exact nested shape exposed +// by FileReader, so later filter rewrite and TableReader materialization agree with the file block. static Status apply_scan_projection_to_mapping_file_type(const FileScanRequest& file_request, - ColumnMapping* mapping) { + ColumnMapping* mapping, + bool predicate_mapping = false) { DORIS_CHECK(mapping != nullptr); DORIS_CHECK(mapping->file_local_id.has_value()); const auto file_column_id = LocalColumnId(*mapping->file_local_id); - // Predicate columns are the actual scan projection when a column is used by row-level filters: - // add_scan_column() removes the duplicate non-predicate projection in that case. - const auto* projection = find_scan_projection(file_request.predicate_columns, file_column_id); + const LocalColumnIndex* projection = nullptr; + if (!predicate_mapping && file_request.has_deferred_non_predicate_column(file_column_id)) { + projection = find_scan_projection(file_request.non_predicate_columns, file_column_id); + } + if (projection == nullptr) { + projection = find_scan_projection(file_request.predicate_columns, file_column_id); + } if (projection == nullptr) { projection = find_scan_projection(file_request.non_predicate_columns, file_column_id); } @@ -1831,76 +1949,6 @@ static Status apply_scan_projection_to_mapping_file_type(const FileScanRequest& return apply_projection_to_mapping_file_type(*projection, mapping); } -// Build extra scan projections required only by row-level filters on nested struct children. -// -// Example: for `SELECT s.a FROM t WHERE s.b.c > 1`, the output projection may only contain `s.a`, -// but the file reader must also read `s.b.c` to evaluate the predicate. This function collects the -// table-side filter path, resolves it through ColumnMapping first, and records the corresponding -// file-side projection in filter_projections. This keeps renamed fields consistent between the scan -// projection and row-level conjunct rewrite. Example: -// table filter path: s -> renamed_b -> c -// old file path: s -> b -> c -// recorded path: s -> b -> c -// When add_scan_column() adds the same root as a predicate column, it rebuilds that root from the -// output mapping, merges this filter-only projection into it, and removes the duplicate -// non-predicate root entry. -static Status build_nested_struct_filter_projection_map( - const std::vector& table_filters, const std::vector& mappings, - FilterProjectionMap* filter_projections) { - DORIS_CHECK(filter_projections != nullptr); - filter_projections->clear(); - for (const auto& table_filter : table_filters) { - if (table_filter.conjunct == nullptr) { - continue; - } - // Collect all nested struct paths in the table filter. For example, for - // `s.id > 5 AND element_at(s, 'renamed_name') = 'abc'`, collect the table paths - // `s -> id` and `s -> renamed_name`, then resolve each one to its file-side projection. - std::vector paths; - collect_nested_struct_paths(table_filter.conjunct->root(), &paths); - for (const auto& path : paths) { - auto mapping_it = std::ranges::find_if(mappings, [&](const ColumnMapping& mapping) { - return mapping.global_index == path.root_global_index; - }); - if (mapping_it == mappings.end() || !mapping_it->file_local_id.has_value() || - path.selectors.empty()) { - continue; - } - - ResolvedNestedStructPath resolved; - LocalColumnIndex root_projection; - if (!resolve_nested_struct_path_for_file(path, mappings, &resolved)) { - if (!table_root_is_map(*mapping_it)) { - continue; - } - // Direct map value filters such as `m.value.a > 1` need the value leaf for row - // evaluation even when the query only projects another value child. This is only a - // scan projection fallback; complex map/array expressions are still not rewritten - // into file-local conjuncts. - LocalColumnIndex child_projection; - RETURN_IF_ERROR(build_file_child_projection_from_schema( - mapping_it->original_file_children, path.selectors, &child_projection)); - if (child_projection.local_id() < 0) { - continue; - } - root_projection = LocalColumnIndex::partial_local(*mapping_it->file_local_id); - root_projection.children.push_back(std::move(child_projection)); - } else { - root_projection = std::move(resolved.file_projection); - } - auto filter_projection_it = filter_projections->find(root_projection.column_id()); - if (filter_projection_it == filter_projections->end()) { - filter_projections->emplace(root_projection.column_id(), - std::move(root_projection)); - continue; - } - RETURN_IF_ERROR( - merge_local_column_index(&filter_projection_it->second, root_projection)); - } - } - return Status::OK(); -} - static void rebuild_projection(ColumnMapping* mapping, LocalIndex block_position) { DORIS_CHECK(mapping->file_local_id.has_value()); if (mapping->is_trivial || needs_complex_rematerialize(*mapping)) { @@ -1993,6 +2041,7 @@ Status TableColumnMapper::_create_mapping_for_column(const ColumnDefinition& tab mapping->global_index = global_index; mapping->table_column_name = table_column.name; mapping->table_type = table_column.type; + mapping->variant_access_paths = table_column.variant_access_paths; // Row-lineage names are Iceberg metadata contracts, not reserved names in generic Hive, // Hudi, or Paimon schemas. Only the Iceberg reader may opt into virtual synthesis. const auto row_lineage_type = @@ -2117,6 +2166,19 @@ Status TableColumnMapper::create_mapping(const std::vector& pr RETURN_IF_ERROR(_create_mapping_for_column(projected_columns[column_idx], GlobalIndex(column_idx), &mapping)); _mappings.push_back(std::move(mapping)); + if (enable_independent_predicate_projection() && + projected_columns[column_idx].has_predicate_access_paths) { + auto predicate_column = projected_columns[column_idx]; + predicate_column.children = predicate_column.predicate_children; + predicate_column.variant_access_paths = predicate_column.predicate_variant_access_paths; + predicate_column.has_predicate_access_paths = false; + predicate_column.predicate_children.clear(); + predicate_column.predicate_variant_access_paths.clear(); + ColumnMapping predicate_mapping; + RETURN_IF_ERROR(_create_mapping_for_column(predicate_column, GlobalIndex(column_idx), + &predicate_mapping)); + _predicate_mappings.push_back(std::move(predicate_mapping)); + } } return Status::OK(); } @@ -2124,7 +2186,13 @@ Status TableColumnMapper::create_mapping(const std::vector& pr std::vector TableColumnMapper::_filter_visible_mappings() const { std::vector mappings; mappings.reserve(_mappings.size() + _hidden_mappings.size()); - mappings.insert(mappings.end(), _mappings.begin(), _mappings.end()); + for (const auto& mapping : _mappings) { + const auto predicate_it = std::ranges::find_if( + _predicate_mappings, [&](const ColumnMapping& predicate_mapping) { + return predicate_mapping.global_index == mapping.global_index; + }); + mappings.push_back(predicate_it == _predicate_mappings.end() ? mapping : *predicate_it); + } mappings.insert(mappings.end(), _hidden_mappings.begin(), _hidden_mappings.end()); return mappings; } @@ -2166,6 +2234,7 @@ Status TableColumnMapper::create_scan_request( // continues to address the same physical column. file_request->local_positions = *fixed_local_positions; } + file_request->non_predicate_positions.clear(); file_request->conjuncts.clear(); file_request->delete_conjuncts.clear(); _filter_entries.clear(); @@ -2188,7 +2257,8 @@ Status TableColumnMapper::create_scan_request( } if (!used_by_filter || !enable_lazy_materialization()) { RETURN_IF_ERROR(add_scan_column(file_request, mapping, false, - force_full_complex_scan_projection())); + force_full_complex_scan_projection(), + enable_variant_leaf_projection())); } } } @@ -2227,12 +2297,12 @@ Status TableColumnMapper::create_scan_request( if (!mapping.file_local_id.has_value()) { continue; } - auto position_it = - file_request->local_positions.find(LocalColumnId(*mapping.file_local_id)); + const auto local_id = LocalColumnId(*mapping.file_local_id); + const auto position_it = file_request->local_positions.find(local_id); DORIS_CHECK(position_it != file_request->local_positions.end()) << file_request->local_positions.size() << " " << *mapping.file_local_id << " " << mapping.file_column_name; - rebuild_projection(&mapping, position_it->second); + rebuild_projection(&mapping, file_request->non_predicate_position(local_id)); } return Status::OK(); } @@ -2246,7 +2316,19 @@ ColumnMapping* TableColumnMapper::_find_mapping(GlobalIndex global_index) { return nullptr; } +ColumnMapping* TableColumnMapper::_find_predicate_mapping(GlobalIndex global_index) { + for (auto& mapping : _predicate_mappings) { + if (mapping.global_index == global_index) { + return &mapping; + } + } + return nullptr; +} + ColumnMapping* TableColumnMapper::_find_filter_mapping(GlobalIndex global_index) { + if (auto* mapping = _find_predicate_mapping(global_index); mapping != nullptr) { + return mapping; + } if (auto* mapping = _find_mapping(global_index); mapping != nullptr) { return mapping; } @@ -2262,10 +2344,7 @@ Status TableColumnMapper::localize_filters(const std::vector& table FileScanRequest* file_request, RuntimeState* runtime_state) { std::set localized_predicate_columns; - FilterProjectionMap filter_projections; auto filter_mappings = _filter_visible_mappings(); - RETURN_IF_ERROR(build_nested_struct_filter_projection_map(table_filters, filter_mappings, - &filter_projections)); for (const auto& table_filter : table_filters) { for (const auto& global_index : table_filter.global_indices) { auto* mapping = _find_filter_mapping(global_index); @@ -2273,9 +2352,29 @@ Status TableColumnMapper::localize_filters(const std::vector& table !filter_conversion_has_local_source(mapping->filter_conversion)) { continue; } + // Nested eager projection is an FE contract. Without predicate_access_paths the + // all-access-path mapping is read as one unit instead of inferring another subtree + // from VExpr and risking a shape that disagrees with final materialization. RETURN_IF_ERROR(add_scan_column(file_request, mapping, enable_lazy_materialization(), force_full_complex_scan_projection(), - &filter_projections)); + enable_variant_leaf_projection())); + auto* output_mapping = _find_mapping(global_index); + if (!enable_independent_predicate_projection() || output_mapping == nullptr || + mapping == output_mapping || !output_mapping->file_local_id.has_value()) { + continue; + } + LocalColumnIndex output_projection; + RETURN_IF_ERROR( + build_scan_projection(output_mapping, force_full_complex_scan_projection(), + enable_variant_leaf_projection(), &output_projection)); + const auto* predicate_projection = find_scan_projection(file_request->predicate_columns, + output_projection.column_id()); + DORIS_CHECK(predicate_projection != nullptr); + if (!same_local_column_index(*predicate_projection, output_projection)) { + FileScanRequestBuilder builder(file_request); + RETURN_IF_ERROR( + builder.add_deferred_non_predicate_column(std::move(output_projection))); + } } } // Rebuild the file type for every scan-local mapping before expression rewrite. Predicate-only @@ -2286,6 +2385,13 @@ Status TableColumnMapper::localize_filters(const std::vector& table RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request, &mapping)); } } + for (auto& mapping : _predicate_mappings) { + if (mapping.file_local_id.has_value() && + file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) { + RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request, &mapping, + /*predicate_mapping=*/true)); + } + } for (auto& mapping : _hidden_mappings) { if (mapping.file_local_id.has_value() && file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) { @@ -2368,14 +2474,15 @@ Status TableColumnMapper::localize_filters(const std::vector& table // Candidate columns are added before expression rewriting because their file-block positions // are needed to localize slot refs. If rewriting rejects every filter that references a visible - // column, move its already-merged output/filter projection to the lazy non-predicate set - // instead of forcing it through the eager predicate path. + // column, move its all-access-path projection to the lazy non-predicate set instead of forcing + // it through the eager predicate path. for (auto& mapping : _mappings) { if (!mapping.file_local_id.has_value()) { continue; } const auto local_id = LocalColumnId(*mapping.file_local_id); - if (localized_predicate_columns.contains(local_id)) { + if (localized_predicate_columns.contains(local_id) || + file_request->has_deferred_non_predicate_column(local_id)) { continue; } const auto predicate_it = std::ranges::find_if( @@ -2419,6 +2526,9 @@ Status TableColumnMapper::_create_direct_mapping(const ColumnDefinition& table_c mapping->original_file_children = file_field.children; mapping->projected_file_children = file_field.children; mapping->file_type = file_field.type; + // Access paths are relative to the Variant terminal, so recursive complex mappings must carry + // them instead of leaving them only on the top-level table column. + mapping->variant_access_paths = table_column.variant_access_paths; mapping->is_trivial = mapping_can_use_file_column_directly(*mapping); mapping->filter_conversion = direct_filter_conversion(*mapping); mapping->child_mappings.clear(); @@ -2484,6 +2594,7 @@ Status TableColumnMapper::_create_direct_mapping(const ColumnDefinition& table_c child_mapping.table_type = table_child.type; child_mapping.file_type = table_child.type; child_mapping.default_expr = table_child.default_expr; + child_mapping.variant_access_paths = table_child.variant_access_paths; child_mapping.filter_conversion = FilterConversionType::FINALIZE_ONLY; mapping->child_mappings.push_back(std::move(child_mapping)); continue; diff --git a/be/src/format_v2/column_mapper.h b/be/src/format_v2/column_mapper.h index a3f743378e4fc4..ccfbd090407a52 100644 --- a/be/src/format_v2/column_mapper.h +++ b/be/src/format_v2/column_mapper.h @@ -128,6 +128,8 @@ struct ColumnMapping { // schema, not table child order. TableReader uses this to map table-output children back to the // file-local block layout when projection, predicate-only children, and schema evolution mix. std::vector projected_file_children; + // Table-side Variant object-key paths retained until the physical shredding schema is known. + std::vector> variant_access_paths; // Split/file-local constant entry when this mapping is produced from partition/default/virtual // expression instead of physical file data. std::optional constant_index; @@ -213,6 +215,7 @@ class TableColumnMapper { RuntimeState* runtime_state = nullptr); void clear() { _mappings.clear(); + _predicate_mappings.clear(); _hidden_mappings.clear(); _constant_map.clear(); _filter_entries.clear(); @@ -233,6 +236,12 @@ class TableColumnMapper { // delimited text field. They must scan the whole complex top-level field and let TableReader // rematerialize the requested table child after row-level filters have run. virtual bool force_full_complex_scan_projection() const { return false; } + // Only Parquet currently has a Variant physical shredding schema and a reader that can + // validate residual-value completeness before honoring a typed-leaf projection. + virtual bool enable_variant_leaf_projection() const { return false; } + // Parquet can keep two independent readers/cursors for the same complex root: a narrow eager + // predicate subtree and the final output subtree materialized only for surviving rows. + virtual bool enable_independent_predicate_projection() const { return false; } const ColumnDefinition* _find_file_field( const ColumnDefinition& table_column, @@ -255,6 +264,7 @@ class TableColumnMapper { std::vector _filter_visible_mappings() const; ColumnMapping* _find_mapping(GlobalIndex global_index); + ColumnMapping* _find_predicate_mapping(GlobalIndex global_index); ColumnMapping* _find_filter_mapping(GlobalIndex global_index); TableColumnMapperOptions _options; @@ -262,6 +272,9 @@ class TableColumnMapper { // describes how to get one table/global column from file-local sources, and carries metadata // for filter localization and result finalize. std::vector _mappings; + // Optional mappings built from SlotDescriptor::predicate_access_paths. They deliberately keep + // a different file type/projection shape from the final output mappings above. + std::vector _predicate_mappings; // Predicate-only top-level columns are not output projection columns, so keep their mappings // here. They are visible only to filter localization and file-reader predicate construction. std::vector _hidden_mappings; @@ -278,6 +291,10 @@ class TableColumnMapper { class ParquetColumnMapper final : public TableColumnMapper { public: using TableColumnMapper::TableColumnMapper; + +protected: + bool enable_variant_leaf_projection() const override { return true; } + bool enable_independent_predicate_projection() const override { return true; } }; // Mapper for readers that always materialize every required file column before filtering. The diff --git a/be/src/format_v2/column_mapper_nested.cpp b/be/src/format_v2/column_mapper_nested.cpp index b172c6ecc874d7..f3cb37f8403c96 100644 --- a/be/src/format_v2/column_mapper_nested.cpp +++ b/be/src/format_v2/column_mapper_nested.cpp @@ -30,87 +30,12 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/primitive_type.h" #include "exprs/vexpr.h" -#include "format_v2/expr/cast.h" #include "gen_cpp/Exprs_types.h" namespace doris::format { namespace { -static bool is_cast_expr(const VExprSPtr& expr) { - return dynamic_cast(expr.get()) != nullptr; -} - -static bool is_signed_integer_type(PrimitiveType type) { - switch (type) { - case TYPE_TINYINT: - case TYPE_SMALLINT: - case TYPE_INT: - case TYPE_BIGINT: - case TYPE_LARGEINT: - return true; - default: - return false; - } -} - -static int primitive_integer_width(PrimitiveType type) { - switch (type) { - case TYPE_TINYINT: - return 1; - case TYPE_SMALLINT: - return 2; - case TYPE_INT: - return 4; - case TYPE_BIGINT: - return 8; - case TYPE_LARGEINT: - return 16; - default: - return 0; - } -} - -static bool is_decimal_type(PrimitiveType type) { - switch (type) { - case TYPE_DECIMAL32: - case TYPE_DECIMAL64: - case TYPE_DECIMALV2: - case TYPE_DECIMAL128I: - case TYPE_DECIMAL256: - return true; - default: - return false; - } -} - -static bool is_order_preserving_safe_cast(const DataTypePtr& from_type, - const DataTypePtr& to_type) { - if (from_type == nullptr || to_type == nullptr) { - return false; - } - const auto from_nested_type = remove_nullable(from_type); - const auto to_nested_type = remove_nullable(to_type); - if (from_nested_type->equals(*to_nested_type)) { - return true; - } - - const auto from_primitive_type = from_nested_type->get_primitive_type(); - const auto to_primitive_type = to_nested_type->get_primitive_type(); - if (is_signed_integer_type(from_primitive_type) && is_signed_integer_type(to_primitive_type)) { - return primitive_integer_width(to_primitive_type) >= - primitive_integer_width(from_primitive_type); - } - if (from_primitive_type == TYPE_FLOAT && to_primitive_type == TYPE_DOUBLE) { - return true; - } - if (is_decimal_type(from_primitive_type) && is_decimal_type(to_primitive_type)) { - return from_nested_type->get_scale() == to_nested_type->get_scale() && - to_nested_type->get_precision() >= from_nested_type->get_precision(); - } - return false; -} - static bool parse_struct_child_selector(const VExprSPtr& expr, StructChildSelector* selector) { DORIS_CHECK(selector != nullptr); if (expr == nullptr || !expr->is_literal()) { @@ -191,27 +116,6 @@ static bool extract_nested_struct_path(const VExprSPtr& expr, NestedStructPath* return true; } -static bool extract_nested_struct_path_for_pruning(const VExprSPtr& expr, NestedStructPath* path) { - DORIS_CHECK(path != nullptr); - // Simple `ELEMENT_AT` - if (extract_nested_struct_path(expr, path)) { - return true; - } - - // `ELEMENT_AT` with `CAST` - if (!is_cast_expr(expr) || expr->get_num_children() != 1) { - return false; - } - const auto& child = expr->children()[0]; - if (!is_order_preserving_safe_cast(child->data_type(), expr->data_type())) { - return false; - } - // A safe widening cast is null-preserving and keeps the comparison ordering of the nested - // primitive leaf, so file-layer pruning can target the original leaf statistics. The row-level - // filter still evaluates the original cast expression after read. - return extract_nested_struct_path_for_pruning(child, path); -} - static const ColumnDefinition* resolve_file_child(const std::vector& children, const StructChildSelector& selector) { if (selector.by_name) { @@ -493,26 +397,6 @@ bool resolve_nested_struct_expr_for_file(const VExprSPtr& expr, return resolve_nested_struct_path_for_file(path, mappings, resolved, true); } -// Collect nested struct leaf references that can be turned into file-reader projections. For -// example, from `s.a > 1 AND element_at(s, 'b') = 2`, this records two paths rooted at `s`: -// `s -> a` and `s -> b`. Non-struct expressions are traversed recursively, while a recognized -// struct path is emitted once so the caller can merge it into the scan projection for that -// top-level file column. -void collect_nested_struct_paths(const VExprSPtr& expr, std::vector* paths) { - DORIS_CHECK(paths != nullptr); - if (expr == nullptr) { - return; - } - NestedStructPath path; - if (extract_nested_struct_path_for_pruning(expr, &path)) { - paths->push_back(std::move(path)); - return; - } - for (const auto& child : expr->children()) { - collect_nested_struct_paths(child, paths); - } -} - std::vector present_child_mappings_in_file_order( const std::vector& child_mappings) { std::vector result; diff --git a/be/src/format_v2/column_mapper_nested.h b/be/src/format_v2/column_mapper_nested.h index 7b2e8cb1513cb4..ab96512e1709ed 100644 --- a/be/src/format_v2/column_mapper_nested.h +++ b/be/src/format_v2/column_mapper_nested.h @@ -86,8 +86,6 @@ bool resolve_nested_struct_expr_for_file(const VExprSPtr& expr, const std::vector& mappings, ResolvedNestedStructPath* resolved); -void collect_nested_struct_paths(const VExprSPtr& expr, std::vector* paths); - std::vector present_child_mappings_in_file_order( const std::vector& child_mappings); diff --git a/be/src/format_v2/file_reader.cpp b/be/src/format_v2/file_reader.cpp index 9bbf7c3a8fc066..a2ca4894044404 100644 --- a/be/src/format_v2/file_reader.cpp +++ b/be/src/format_v2/file_reader.cpp @@ -65,6 +65,14 @@ std::string FileScanRequest::debug_string() const { } out << column_id << ":" << block_position; } + out << "}, non_predicate_positions={"; + position_idx = 0; + for (const auto& [column_id, block_position] : non_predicate_positions) { + if (position_idx++ > 0) { + out << ", "; + } + out << column_id << ":" << block_position; + } out << "}, conjunct_count=" << conjuncts.size() << ", delete_conjunct_count=" << delete_conjuncts.size() << ", count_star_placeholder_columns={"; diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 65a2d03417c605..0256b8c1eebcb2 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -68,8 +68,8 @@ struct FileScanRequest { // Columns that must be read before row-level filtering. They are materialized eagerly because // conjuncts/delete_conjuncts need them to decide the selected rows. std::vector predicate_columns; - // Columns read after row-level filtering. Predicate columns are also available for output and - // should not be duplicated here. + // Columns read after row-level filtering. A complex root may intentionally also appear in + // predicate_columns when its eager predicate subtree is smaller than its final output subtree. std::vector non_predicate_columns; // Predicate columns introduced only to evaluate hidden filter slots. Their values are dead // after all file-local predicates run, although the shared file block still needs row-shaped @@ -77,6 +77,10 @@ struct FileScanRequest { std::vector predicate_only_columns; // file-local column id -> file-local output block position. std::map local_positions; + // Optional output position for a root that has independent eager-predicate and deferred-output + // projections. local_positions continues to identify the position referenced by localized + // predicate expressions. + std::map non_predicate_positions; // Row-level filters converted to file-local expressions from table-level predicates. VExprContextSPtrs conjuncts; // Delete predicates converted to file-local expressions. A TRUE result means that the row is @@ -98,6 +102,26 @@ struct FileScanRequest { bool is_predicate_only(LocalColumnId column_id) const { return std::ranges::find(predicate_only_columns, column_id) != predicate_only_columns.end(); } + + LocalIndex non_predicate_position(LocalColumnId column_id) const { + const auto it = non_predicate_positions.find(column_id); + return it == non_predicate_positions.end() ? local_positions.at(column_id) : it->second; + } + + bool has_deferred_non_predicate_column(LocalColumnId column_id) const { + return non_predicate_positions.contains(column_id); + } + + size_t block_column_count() const { + size_t count = 0; + for (const auto& [_, position] : local_positions) { + count = std::max(count, position.value() + 1); + } + for (const auto& [_, position] : non_predicate_positions) { + count = std::max(count, position.value() + 1); + } + return count; + } }; // Helper for constructing the scan-column layout in FileScanRequest. @@ -105,9 +129,10 @@ struct FileScanRequest { // as Parquet can read predicate columns first, filter rows, and then lazily read the remaining // projected columns. The two lists still share one file-local output block, whose positions are // stored in local_positions. This builder centralizes the mechanical rules for that shared layout: -// - each root file column gets one stable block position; +// - each root file column gets one stable predicate block position; // - predicate columns dominate non-predicate columns because they are already returned in the file // block and can be reused for final materialization; +// - a smaller complex predicate subtree may get a second deferred output position; // - repeated nested projections for the same root are merged instead of duplicated. // TableColumnMapper should still own table-to-file semantic resolution. This helper only owns the // FileScanRequest layout contract after a file-local projection has been produced. @@ -127,6 +152,38 @@ class FileScanRequestBuilder { /*is_predicate_column=*/false); } + Status add_deferred_non_predicate_column(LocalColumnIndex projection) { + const auto file_column_id = projection.column_id(); + DORIS_CHECK(file_column_id != LocalColumnId::invalid()); + DORIS_CHECK(_request->local_positions.contains(file_column_id)); + DORIS_CHECK(std::ranges::any_of(_request->predicate_columns, + [&](const LocalColumnIndex& predicate) { + return predicate.column_id() == file_column_id; + })); + + if (!_request->non_predicate_positions.contains(file_column_id)) { + _request->non_predicate_positions.emplace(file_column_id, + _next_block_position(*_request)); + } + _sort_projection_children_by_file_id(&projection); + auto existing = std::ranges::find_if(_request->non_predicate_columns, + [&](const LocalColumnIndex& output) { + return output.column_id() == file_column_id; + }); + if (existing == _request->non_predicate_columns.end()) { + _request->non_predicate_columns.push_back(std::move(projection)); + } else { + RETURN_IF_ERROR(merge_local_column_index(&*existing, projection)); + _sort_projection_children_by_file_id(&*existing); + } + if (!_request->is_predicate_only(file_column_id)) { + // The eager complex value has a different physical shape from the final value and + // must never leak into table materialization after its predicates have run. + _request->predicate_only_columns.push_back(file_column_id); + } + return Status::OK(); + } + Status add_predicate_column(LocalColumnId column_id) { return add_predicate_column(LocalColumnIndex::top_level(column_id)); } @@ -141,6 +198,9 @@ class FileScanRequestBuilder { for (const auto& [_, block_position] : request.local_positions) { next_position = std::max(next_position, block_position.value() + 1); } + for (const auto& [_, block_position] : request.non_predicate_positions) { + next_position = std::max(next_position, block_position.value() + 1); + } return LocalIndex(next_position); } @@ -164,9 +224,11 @@ class FileScanRequestBuilder { const auto file_column_id = projection.column_id(); DORIS_CHECK(file_column_id != LocalColumnId::invalid()); if (!is_predicate_column && - std::ranges::find_if(_request->predicate_columns, [&](const LocalColumnIndex& p) { - return p.column_id() == file_column_id; - }) != _request->predicate_columns.end()) { + std::ranges::find_if(_request->predicate_columns, + [&](const LocalColumnIndex& p) { + return p.column_id() == file_column_id; + }) != _request->predicate_columns.end() && + !_request->has_deferred_non_predicate_column(file_column_id)) { return Status::OK(); } if (!_request->local_positions.contains(file_column_id)) { @@ -184,7 +246,7 @@ class FileScanRequestBuilder { _sort_projection_children_by_file_id(&*existing_projection_it); } - if (is_predicate_column) { + if (is_predicate_column && !_request->has_deferred_non_predicate_column(file_column_id)) { auto it = std::ranges::find_if( _request->non_predicate_columns, [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); diff --git a/be/src/format_v2/parquet/native_schema_desc.cpp b/be/src/format_v2/parquet/native_schema_desc.cpp index 9eba6d8feaa55a..b56afb6f7170fb 100644 --- a/be/src/format_v2/parquet/native_schema_desc.cpp +++ b/be/src/format_v2/parquet/native_schema_desc.cpp @@ -20,7 +20,9 @@ #include #include +#include #include +#include #include #include "common/cast_set.h" @@ -55,6 +57,412 @@ static bool is_map_node(const tparquet::SchemaElement& schema) { (schema.__isset.logicalType && schema.logicalType.__isset.MAP); } +static bool is_variant_node(const tparquet::SchemaElement& schema) { + return schema.__isset.logicalType && schema.logicalType.__isset.VARIANT; +} + +enum class VariantPrimitiveAnnotation : uint8_t { + NONE, + INT8, + INT16, + DECIMAL, + DATE, + TIME_MICROS, + TIMESTAMP_MICROS, + TIMESTAMP_NANOS, + STRING, + UUID, + UNSUPPORTED, +}; + +static VariantPrimitiveAnnotation variant_logical_annotation( + const tparquet::SchemaElement& schema) { + if (!schema.__isset.logicalType) { + return VariantPrimitiveAnnotation::NONE; + } + const auto& logical = schema.logicalType; + if (logical.__isset.INTEGER) { + if (!logical.INTEGER.isSigned) { + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.INTEGER.bitWidth == 8) { + return VariantPrimitiveAnnotation::INT8; + } + if (logical.INTEGER.bitWidth == 16) { + return VariantPrimitiveAnnotation::INT16; + } + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.DECIMAL) { + return VariantPrimitiveAnnotation::DECIMAL; + } + if (logical.__isset.DATE) { + return VariantPrimitiveAnnotation::DATE; + } + if (logical.__isset.TIME) { + return !logical.TIME.isAdjustedToUTC && logical.TIME.unit.__isset.MICROS + ? VariantPrimitiveAnnotation::TIME_MICROS + : VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.TIMESTAMP) { + if (logical.TIMESTAMP.unit.__isset.MICROS) { + return VariantPrimitiveAnnotation::TIMESTAMP_MICROS; + } + if (logical.TIMESTAMP.unit.__isset.NANOS) { + return VariantPrimitiveAnnotation::TIMESTAMP_NANOS; + } + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.STRING) { + return VariantPrimitiveAnnotation::STRING; + } + if (logical.__isset.UUID) { + return VariantPrimitiveAnnotation::UUID; + } + const bool empty = !logical.__isset.MAP && !logical.__isset.LIST && !logical.__isset.ENUM && + !logical.__isset.UNKNOWN && !logical.__isset.JSON && !logical.__isset.BSON && + !logical.__isset.FLOAT16 && !logical.__isset.GEOMETRY && + !logical.__isset.GEOGRAPHY && !logical.__isset.VARIANT; + return empty ? VariantPrimitiveAnnotation::NONE : VariantPrimitiveAnnotation::UNSUPPORTED; +} + +static VariantPrimitiveAnnotation variant_converted_annotation( + const tparquet::SchemaElement& schema) { + if (!schema.__isset.converted_type) { + return VariantPrimitiveAnnotation::NONE; + } + switch (schema.converted_type) { + case tparquet::ConvertedType::INT_8: + return VariantPrimitiveAnnotation::INT8; + case tparquet::ConvertedType::INT_16: + return VariantPrimitiveAnnotation::INT16; + case tparquet::ConvertedType::DECIMAL: + return VariantPrimitiveAnnotation::DECIMAL; + case tparquet::ConvertedType::DATE: + return VariantPrimitiveAnnotation::DATE; + case tparquet::ConvertedType::TIME_MICROS: + return VariantPrimitiveAnnotation::TIME_MICROS; + case tparquet::ConvertedType::TIMESTAMP_MICROS: + return VariantPrimitiveAnnotation::TIMESTAMP_MICROS; + case tparquet::ConvertedType::UTF8: + return VariantPrimitiveAnnotation::STRING; + default: + return VariantPrimitiveAnnotation::UNSUPPORTED; + } +} + +static Status validate_variant_decimal(const tparquet::SchemaElement& schema, + tparquet::Type::type physical_type) { + int32_t precision = -1; + int32_t scale = -1; + if (schema.__isset.logicalType && schema.logicalType.__isset.DECIMAL) { + precision = schema.logicalType.DECIMAL.precision; + scale = schema.logicalType.DECIMAL.scale; + if ((schema.__isset.precision && schema.precision != precision) || + (schema.__isset.scale && schema.scale != scale)) { + return Status::Corruption( + "Parquet Variant DECIMAL logical and converted parameters disagree"); + } + } else if (schema.__isset.precision && schema.__isset.scale) { + precision = schema.precision; + scale = schema.scale; + } + if (precision <= 0 || precision > 38 || scale < 0 || scale > precision) { + return Status::Corruption("Parquet Variant DECIMAL({}, {}) is invalid", precision, scale); + } + + if ((physical_type == tparquet::Type::INT32 && precision > 9) || + (physical_type == tparquet::Type::INT64 && (precision < 10 || precision > 18)) || + ((physical_type == tparquet::Type::BYTE_ARRAY || + physical_type == tparquet::Type::FIXED_LEN_BYTE_ARRAY) && + precision < 19)) { + return Status::Corruption( + "Parquet Variant DECIMAL precision {} does not match physical type {}", precision, + physical_type); + } + if (physical_type == tparquet::Type::FIXED_LEN_BYTE_ARRAY) { + static constexpr int32_t MAX_PRECISION_BY_LENGTH[] = {2, 4, 6, 9, 11, 14, 16, 18, + 21, 23, 26, 28, 31, 33, 35, 38}; + if (!schema.__isset.type_length || schema.type_length <= 0 || schema.type_length > 16 || + precision > MAX_PRECISION_BY_LENGTH[schema.type_length - 1]) { + return Status::Corruption( + "Parquet Variant DECIMAL precision {} does not fit fixed length {}", precision, + schema.__isset.type_length ? schema.type_length : -1); + } + } + return Status::OK(); +} + +static Status validate_variant_primitive_type(const NativeFieldSchema& typed) { + const auto& schema = typed.parquet_schema; + auto logical = variant_logical_annotation(schema); + auto converted = variant_converted_annotation(schema); + if (logical == VariantPrimitiveAnnotation::UNSUPPORTED || + converted == VariantPrimitiveAnnotation::UNSUPPORTED || + (logical != VariantPrimitiveAnnotation::NONE && + converted != VariantPrimitiveAnnotation::NONE && logical != converted)) { + return Status::Corruption( + "Parquet Variant typed value {} has an unsupported logical annotation", typed.name); + } + const auto annotation = logical != VariantPrimitiveAnnotation::NONE ? logical : converted; + const auto physical = schema.type; + bool valid = false; + switch (physical) { + case tparquet::Type::BOOLEAN: + valid = annotation == VariantPrimitiveAnnotation::NONE; + break; + case tparquet::Type::INT32: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::INT8 || + annotation == VariantPrimitiveAnnotation::INT16 || + annotation == VariantPrimitiveAnnotation::DECIMAL || + annotation == VariantPrimitiveAnnotation::DATE; + break; + case tparquet::Type::INT64: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::DECIMAL || + annotation == VariantPrimitiveAnnotation::TIME_MICROS || + annotation == VariantPrimitiveAnnotation::TIMESTAMP_MICROS || + annotation == VariantPrimitiveAnnotation::TIMESTAMP_NANOS; + break; + case tparquet::Type::FLOAT: + case tparquet::Type::DOUBLE: + valid = annotation == VariantPrimitiveAnnotation::NONE; + break; + case tparquet::Type::BYTE_ARRAY: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::STRING || + annotation == VariantPrimitiveAnnotation::DECIMAL; + break; + case tparquet::Type::FIXED_LEN_BYTE_ARRAY: + valid = annotation == VariantPrimitiveAnnotation::DECIMAL || + (annotation == VariantPrimitiveAnnotation::UUID && schema.__isset.type_length && + schema.type_length == 16); + break; + default: + valid = false; + break; + } + if (!valid) { + return Status::Corruption( + "Parquet Variant typed value {} has unsupported physical/logical type pair", + typed.name); + } + if (annotation == VariantPrimitiveAnnotation::DECIMAL) { + RETURN_IF_ERROR(validate_variant_decimal(schema, physical)); + } + return Status::OK(); +} + +class ScopedBoolOverride { +public: + ScopedBoolOverride(bool& target, bool value) : _target(target), _original(target) { + _target = value; + } + ~ScopedBoolOverride() { _target = _original; } + +private: + bool& _target; + bool _original; +}; + +static Status validate_variant_layout(const tparquet::SchemaElement& group_schema, + const NativeFieldSchema& group_field) { + const auto& annotation = group_schema.logicalType.VARIANT; + if (annotation.__isset.specification_version && annotation.specification_version != 1) { + return Status::NotSupported("Parquet Variant specification version {} is not supported", + annotation.specification_version); + } + if (group_field.children.size() < 2 || group_field.children.size() > 3) { + return Status::Corruption( + "Parquet Variant {} must contain metadata, value, and optional typed_value", + group_schema.name); + } + + const NativeFieldSchema* metadata = nullptr; + const NativeFieldSchema* value = nullptr; + const NativeFieldSchema* typed_value = nullptr; + for (const auto& child : group_field.children) { + const NativeFieldSchema** target = nullptr; + if (child.name == "metadata") { + target = &metadata; + } else if (child.name == "value") { + target = &value; + } else if (child.name == "typed_value") { + target = &typed_value; + } else { + return Status::Corruption("Parquet Variant {} has unexpected child {}", + group_schema.name, child.name); + } + if (*target != nullptr) { + return Status::Corruption("Parquet Variant {} has duplicate child {}", + group_schema.name, child.name); + } + *target = &child; + } + if (metadata == nullptr || value == nullptr) { + return Status::Corruption("Parquet Variant {} requires metadata and value children", + group_schema.name); + } + if (!metadata->children.empty() || metadata->physical_type != tparquet::Type::BYTE_ARRAY || + metadata->parquet_schema.repetition_type != tparquet::FieldRepetitionType::REQUIRED) { + return Status::Corruption("Parquet Variant {} metadata must be a required BYTE_ARRAY", + group_schema.name); + } + const auto expected_value_repetition = typed_value == nullptr + ? tparquet::FieldRepetitionType::REQUIRED + : tparquet::FieldRepetitionType::OPTIONAL; + // SQL nullability belongs to the outer Variant group. Only shredding makes value optional, + // because typed_value may carry all or part of the logical value instead. + if (!value->children.empty() || value->physical_type != tparquet::Type::BYTE_ARRAY || + value->parquet_schema.repetition_type != expected_value_repetition) { + return Status::Corruption("Parquet Variant {} value must be a {} BYTE_ARRAY", + group_schema.name, + typed_value == nullptr ? "required" : "optional"); + } + if (typed_value != nullptr && + typed_value->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL) { + return Status::Corruption("Parquet Variant {} typed_value must be optional", + group_schema.name); + } + + enum class WrapperContext : uint8_t { OBJECT_FIELD, ARRAY_ELEMENT }; + std::function validate_typed_value; + std::function validate_wrapper; + validate_wrapper = [&](const NativeFieldSchema& wrapper, WrapperContext context) -> Status { + if (!wrapper.parquet_schema.__isset.repetition_type || + wrapper.parquet_schema.repetition_type != tparquet::FieldRepetitionType::REQUIRED) { + return Status::Corruption("Parquet Variant shredded wrapper {} must be required", + wrapper.name); + } + const NativeFieldSchema* fallback = nullptr; + const NativeFieldSchema* typed = nullptr; + for (const auto& child : wrapper.children) { + if (child.name == "value") { + if (fallback != nullptr) { + return Status::Corruption( + "Parquet Variant wrapper {} has duplicate value child", wrapper.name); + } + fallback = &child; + } else if (child.name == "typed_value") { + if (typed != nullptr) { + return Status::Corruption( + "Parquet Variant wrapper {} has duplicate typed_value child", + wrapper.name); + } + typed = &child; + } else { + return Status::Corruption("Parquet Variant wrapper {} has unexpected child {}", + wrapper.name, child.name); + } + } + if (fallback == nullptr && typed == nullptr) { + return Status::Corruption( + "Parquet Variant shredded wrapper {} requires at least one of value or " + "typed_value", + wrapper.name); + } + // Object fields always retain the fallback value carrier; only typed_value is optional. + // Array elements may omit either carrier when every element uses the remaining one. + if (context == WrapperContext::OBJECT_FIELD && fallback == nullptr) { + return Status::Corruption( + "Parquet Variant object wrapper {} requires an optional value child", + wrapper.name); + } + if (fallback != nullptr && + (!fallback->children.empty() || fallback->physical_type != tparquet::Type::BYTE_ARRAY || + !fallback->parquet_schema.__isset.repetition_type || + fallback->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL)) { + return Status::Corruption( + "Parquet Variant wrapper {} value must be an optional BYTE_ARRAY", + wrapper.name); + } + if (typed != nullptr) { + if (!typed->parquet_schema.__isset.repetition_type || + typed->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL) { + return Status::Corruption("Parquet Variant wrapper {} typed_value must be optional", + wrapper.name); + } + return validate_typed_value(*typed); + } + return Status::OK(); + }; + validate_typed_value = [&](const NativeFieldSchema& typed) -> Status { + if (!typed.unsupported_reason.empty()) { + return Status::NotSupported("Parquet Variant typed value {} is not supported: {}", + typed.name, typed.unsupported_reason); + } + if (typed.children.empty()) { + const auto& physical = typed.parquet_schema; + if (physical.__isset.logicalType && physical.logicalType.__isset.INTEGER && + !physical.logicalType.INTEGER.isSigned) { + return Status::Corruption( + "Parquet Variant unsigned integers are not valid typed values"); + } + if (physical.__isset.converted_type && + (physical.converted_type == tparquet::ConvertedType::UINT_8 || + physical.converted_type == tparquet::ConvertedType::UINT_16 || + physical.converted_type == tparquet::ConvertedType::UINT_32 || + physical.converted_type == tparquet::ConvertedType::UINT_64)) { + return Status::Corruption( + "Parquet Variant unsigned integers are not valid typed values"); + } + if (physical.__isset.logicalType && physical.logicalType.__isset.TIME) { + const auto& time = physical.logicalType.TIME; + // Variant v1 has one canonical TIME representation: local wall-clock MICROS. + // Accepting adjusted or lower-precision forms would make projection-dependent + // reconstruction disagree with the canonical Variant value. + if (time.isAdjustedToUTC) { + return Status::Corruption( + "Parquet Variant TIME must have isAdjustedToUTC=false"); + } + if (!time.unit.__isset.MICROS) { + return Status::Corruption( + "Parquet Variant TIME(MILLIS) is not supported; use TIME(MICROS)"); + } + } + if (physical.__isset.converted_type && + physical.converted_type == tparquet::ConvertedType::TIME_MILLIS) { + return Status::Corruption( + "Parquet Variant TIME(MILLIS) is not supported; use TIME(MICROS)"); + } + if (physical.__isset.logicalType && physical.logicalType.__isset.TIMESTAMP && + physical.logicalType.TIMESTAMP.unit.__isset.NANOS) { + // Reject at schema open so full reconstruction and direct typed-leaf access have + // the same precision contract instead of diverging after projection planning. + return Status::NotSupported("Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + // Preserve precise diagnostics above, then enforce the complete Variant matrix before + // generic Parquet inference can re-encode a value with another logical identity. + RETURN_IF_ERROR(validate_variant_primitive_type(typed)); + return Status::OK(); + } + + const PrimitiveType primitive = remove_nullable(typed.data_type)->get_primitive_type(); + if (primitive == TYPE_STRUCT) { + std::unordered_set field_names; + for (const auto& child : typed.children) { + if (!field_names.insert(child.name).second) { + // Name lookup selects one physical wrapper, so duplicates would otherwise make + // full reconstruction and leaf projection observe different logical values. + return Status::Corruption( + "Parquet Variant object has duplicate shredded field {}", child.name); + } + RETURN_IF_ERROR(validate_wrapper(child, WrapperContext::OBJECT_FIELD)); + } + return Status::OK(); + } + if (primitive == TYPE_ARRAY && typed.children.size() == 1) { + return validate_wrapper(typed.children[0], WrapperContext::ARRAY_ELEMENT); + } + return Status::Corruption("Invalid Parquet Variant typed_value schema {}", typed.name); + }; + if (typed_value != nullptr) { + RETURN_IF_ERROR(validate_typed_value(*typed_value)); + } + return Status::OK(); +} + static bool has_primitive_only_annotation(const tparquet::SchemaElement& schema) { if (schema.__isset.logicalType) { const auto& logical = schema.logicalType; @@ -265,6 +673,11 @@ Status NativeFieldDescriptor::parse_node_field( // nested structure or nullable list return parse_group_field(t_schemas, curr_pos, node_field); } + if (is_variant_node(t_schema)) { + return Status::InvalidArgument( + "Parquet Variant logical type requires a group node, got primitive {}", + t_schema.name); + } if (is_repeated_node(t_schema)) { // repeated (LIST) // produce required list @@ -517,6 +930,27 @@ Status NativeFieldDescriptor::parse_group_field( const std::vector& t_schemas, size_t curr_pos, NativeFieldSchema* group_field) { auto& group_schema = t_schemas[curr_pos]; + group_field->parquet_schema = group_schema; + if (is_variant_node(group_schema)) { + if (is_repeated_node(group_schema)) { + // A repeated annotated group needs an ARRAY carrier and Dremel-level remapping; treating + // it as a scalar Variant would expose the wrong row shape. + return Status::NotSupported("repeated Parquet Variant group {} is not supported", + group_schema.name); + } + // UTC-adjusted timestamps inside Variant carry an instant, independent of the catalog's + // presentation mapping. Parsing them as DATETIMEV2 would apply the session timezone and + // lose that instant when the shredded value is re-encoded into ColumnVariantV2. + { + ScopedBoolOverride timestamp_tz_mapping(_enable_mapping_timestamp_tz, true); + RETURN_IF_ERROR(parse_struct_field(t_schemas, curr_pos, group_field)); + } + RETURN_IF_ERROR(validate_variant_layout(group_schema, *group_field)); + group_field->variant_physical_type = group_field->data_type; + // Native page readers dispatch groups from data_type, so preserve the physical STRUCT + // here. The public Parquet schema maps it to logical Variant without losing this shape. + return Status::OK(); + } if ((group_schema.__isset.logicalType && group_schema.logicalType.__isset.ENUM) || (group_schema.__isset.converted_type && group_schema.converted_type == tparquet::ConvertedType::ENUM)) { diff --git a/be/src/format_v2/parquet/native_schema_desc.h b/be/src/format_v2/parquet/native_schema_desc.h index dfb669559fc71a..918be6d2c65c58 100644 --- a/be/src/format_v2/parquet/native_schema_desc.h +++ b/be/src/format_v2/parquet/native_schema_desc.h @@ -47,6 +47,10 @@ struct NativeFieldSchema { // Used to identify whether this field is a nested field. DataTypePtr data_type; + + // VARIANT is logically exposed as DataTypeVariantV2, while native page readers still need the + // physical STRUCT shape formed by metadata/value/typed_value. + DataTypePtr variant_physical_type; // Schema construction keeps a physical fallback so unprojected columns and metadata-only // queries remain readable, while projection validation reports the original logical failure. std::string unsupported_reason; diff --git a/be/src/format_v2/parquet/native_schema_node.cpp b/be/src/format_v2/parquet/native_schema_node.cpp index 052df93e4b4143..e1f573951fbbfc 100644 --- a/be/src/format_v2/parquet/native_schema_node.cpp +++ b/be/src/format_v2/parquet/native_schema_node.cpp @@ -64,7 +64,8 @@ Status build_native_schema_node(const DataTypePtr& projected_type, const auto type = remove_nullable(projected_type); switch (type->get_primitive_type()) { case TYPE_STRUCT: { - if (file_schema.kind != ParquetColumnSchemaKind::STRUCT) { + if (file_schema.kind != ParquetColumnSchemaKind::STRUCT && + file_schema.kind != ParquetColumnSchemaKind::VARIANT) { return Status::Corruption("Parquet column {} is not a STRUCT", file_schema.name); } const auto* struct_type = assert_cast(type.get()); diff --git a/be/src/format_v2/parquet/parquet_column_schema.cpp b/be/src/format_v2/parquet/parquet_column_schema.cpp index 7e541b57da2780..623f1ece3dd409 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.cpp +++ b/be/src/format_v2/parquet/parquet_column_schema.cpp @@ -20,7 +20,11 @@ #include #include +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "format_v2/parquet/native_schema_desc.h" #include "format_v2/parquet/parquet_type.h" @@ -63,7 +67,11 @@ void fill_native_type_descriptor(const NativeFieldSchema& field, ParquetTypeDesc result->fixed_length = schema.__isset.type_length ? schema.type_length : -1; if (schema.__isset.logicalType) { const auto& logical = schema.logicalType; - if (logical.__isset.DECIMAL) { + if (logical.__isset.STRING) { + result->is_string_annotation = true; + } else if (logical.__isset.UUID) { + result->is_uuid = true; + } else if (logical.__isset.DECIMAL) { result->is_decimal = true; result->decimal_precision = logical.DECIMAL.precision; result->decimal_scale = logical.DECIMAL.scale; @@ -87,6 +95,9 @@ void fill_native_type_descriptor(const NativeFieldSchema& field, ParquetTypeDesc } } else if (schema.__isset.converted_type) { switch (schema.converted_type) { + case tparquet::ConvertedType::UTF8: + result->is_string_annotation = true; + break; case tparquet::ConvertedType::DECIMAL: result->is_decimal = true; result->decimal_precision = schema.__isset.precision ? schema.precision : -1; @@ -167,13 +178,31 @@ void propagate_native_max_levels(ParquetColumnSchema* schema) { } } +bool contains_variant_node(const ParquetColumnSchema& schema) { + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + return true; + } + return std::ranges::any_of(schema.children, [](const auto& child) { + DORIS_CHECK(child != nullptr); + return contains_variant_node(*child); + }); +} + std::unique_ptr build_native_node_schema(const NativeFieldSchema& field, int32_t local_id) { auto result = std::make_unique(); result->local_id = local_id; result->parquet_field_id = field.field_id; result->name = field.name; - result->type = field.data_type; + result->variant_physical_type = field.variant_physical_type; + if (field.variant_physical_type != nullptr) { + DataTypePtr variant_type = std::make_shared(); + result->type = field.variant_physical_type->is_nullable() + ? make_nullable(std::move(variant_type)) + : std::move(variant_type); + } else { + result->type = field.data_type; + } result->definition_level = field.definition_level; result->repetition_level = field.repetition_level; result->max_definition_level = field.definition_level; @@ -191,7 +220,9 @@ std::unique_ptr build_native_node_schema(const NativeFieldS fill_native_type_descriptor(field, &result->type_descriptor); return result; } - if (primitive_type == TYPE_ARRAY) { + if (field.variant_physical_type != nullptr) { + result->kind = ParquetColumnSchemaKind::VARIANT; + } else if (primitive_type == TYPE_ARRAY) { result->kind = ParquetColumnSchemaKind::LIST; } else if (primitive_type == TYPE_MAP) { result->kind = ParquetColumnSchemaKind::MAP; @@ -203,6 +234,32 @@ std::unique_ptr build_native_node_schema(const NativeFieldS result->children.push_back( build_native_node_schema(field.children[child_idx], cast_set(child_idx))); } + // A nested Variant changes its public child type from the physical STRUCT carrier. Rebuild + // every enclosing complex type so file-block columns keep the same logical shape as readers. + if (result->kind != ParquetColumnSchemaKind::VARIANT && contains_variant_node(*result)) { + DataTypePtr logical_type; + if (result->kind == ParquetColumnSchemaKind::LIST) { + DORIS_CHECK(result->children.size() == 1); + logical_type = std::make_shared(result->children[0]->type); + } else if (result->kind == ParquetColumnSchemaKind::MAP) { + DORIS_CHECK(result->children.size() == 2); + logical_type = std::make_shared(make_nullable(result->children[0]->type), + make_nullable(result->children[1]->type)); + } else { + DataTypes child_types; + Strings child_names; + child_types.reserve(result->children.size()); + child_names.reserve(result->children.size()); + for (const auto& child : result->children) { + child_types.push_back(child->type); + child_names.push_back(child->name); + } + logical_type = std::make_shared(std::move(child_types), + std::move(child_names)); + } + result->type = result->type->is_nullable() ? make_nullable(std::move(logical_type)) + : std::move(logical_type); + } propagate_native_max_levels(result.get()); return result; } diff --git a/be/src/format_v2/parquet/parquet_column_schema.h b/be/src/format_v2/parquet/parquet_column_schema.h index 697bcd498382b1..4cb295018332fe 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.h +++ b/be/src/format_v2/parquet/parquet_column_schema.h @@ -32,6 +32,7 @@ enum class ParquetColumnSchemaKind { STRUCT, // Parquet group with STRUCT semantics LIST, // Parquet group with LIST semantics MAP, // Parquet group with MAP semantics + VARIANT, // Parquet Variant logical group }; // ============================================================================ @@ -46,6 +47,10 @@ struct ParquetColumnSchema { DataTypePtr type = nullptr; + // Set only for VARIANT. The public file type is DataTypeVariantV2, while this type describes + // the metadata/value/typed_value STRUCT consumed by the native decoder. + DataTypePtr variant_physical_type = nullptr; + int leaf_column_id = -1; ParquetTypeDescriptor type_descriptor {}; diff --git a/be/src/format_v2/parquet/parquet_profile.cpp b/be/src/format_v2/parquet/parquet_profile.cpp index e6717505cea898..5fd3100a2e5bcf 100644 --- a/be/src/format_v2/parquet/parquet_profile.cpp +++ b/be/src/format_v2/parquet/parquet_profile.cpp @@ -56,6 +56,8 @@ void ParquetProfile::init(RuntimeProfile* profile) { parquet_profile, 1); filtered_page_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "FilteredRowsByPage", TUnit::UNIT, parquet_profile, 1); + variant_leaf_projections = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantLeafProjections", + TUnit::UNIT, parquet_profile, 1); pages_skipped_by_data_page_filter = ADD_CHILD_COUNTER_WITH_LEVEL( profile, "PagesSkippedByDataPageFilter", TUnit::UNIT, parquet_profile, 1); data_page_filter_skip_bytes = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "DataPageFilterSkipBytes", @@ -88,6 +90,18 @@ void ParquetProfile::init(RuntimeProfile* profile) { ADD_CHILD_TIMER_WITH_LEVEL(profile, "LevelOnlySkipTime", parquet_profile, 1); materialization_time = ADD_CHILD_TIMER_WITH_LEVEL(profile, "MaterializationTime", parquet_profile, 1); + variant_reconstruction_time = + ADD_CHILD_TIMER_WITH_LEVEL(profile, "VariantReconstructionTime", parquet_profile, 1); + variant_reconstructed_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantReconstructedRows", + TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantDirectLeafRows", + TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_path_misses = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafPathMisses", TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_residual_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafResidualFallbacks", TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_unsupported_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafUnsupportedFallbacks", TUnit::UNIT, parquet_profile, 1); hybrid_selection_batches = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionBatches", TUnit::UNIT, parquet_profile, 1); hybrid_selection_ranges = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionRanges", @@ -298,6 +312,12 @@ ParquetColumnReaderProfile ParquetProfile::column_reader_profile() const { .level_only_read_time = level_only_read_time, .level_only_skip_time = level_only_skip_time, .materialization_time = materialization_time, + .variant_reconstruction_time = variant_reconstruction_time, + .variant_reconstructed_rows = variant_reconstructed_rows, + .variant_direct_leaf_rows = variant_direct_leaf_rows, + .variant_direct_leaf_path_misses = variant_direct_leaf_path_misses, + .variant_direct_leaf_residual_fallbacks = variant_direct_leaf_residual_fallbacks, + .variant_direct_leaf_unsupported_fallbacks = variant_direct_leaf_unsupported_fallbacks, .hybrid_selection_batches = hybrid_selection_batches, .hybrid_selection_ranges = hybrid_selection_ranges, .hybrid_selection_null_fallback_batches = hybrid_selection_null_fallback_batches, diff --git a/be/src/format_v2/parquet/parquet_profile.h b/be/src/format_v2/parquet/parquet_profile.h index 57c03c79336f06..ed1faa8f935134 100644 --- a/be/src/format_v2/parquet/parquet_profile.h +++ b/be/src/format_v2/parquet/parquet_profile.h @@ -38,6 +38,12 @@ struct ParquetColumnReaderProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; // value materialization time (ns) + RuntimeProfile::Counter* variant_reconstruction_time = nullptr; + RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; @@ -145,6 +151,8 @@ struct ParquetProfile { RuntimeProfile::Counter* selected_row_ranges = nullptr; RuntimeProfile::Counter* filtered_group_rows = nullptr; RuntimeProfile::Counter* filtered_page_rows = nullptr; + // File-level Variant access paths that safely retained a physical typed-leaf projection. + RuntimeProfile::Counter* variant_leaf_projections = nullptr; // ======== Page Skip ======== RuntimeProfile::Counter* pages_skipped_by_data_page_filter = nullptr; @@ -166,6 +174,12 @@ struct ParquetProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; + RuntimeProfile::Counter* variant_reconstruction_time = nullptr; + RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index 065adc63b66e1e..bd44316463f6b1 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -57,6 +58,117 @@ struct ParquetReaderScanState { bool enable_strict_mode = false; }; +const ParquetColumnSchema* projected_schema_child(const ParquetColumnSchema& schema, + int32_t local_id) { + const auto child_it = std::ranges::find_if( + schema.children, [local_id](const auto& child) { return child->local_id == local_id; }); + return child_it == schema.children.end() ? nullptr : child_it->get(); +} + +const ParquetColumnSchema* schema_child_by_name(const ParquetColumnSchema& schema, + std::string_view name) { + const auto child_it = std::ranges::find_if( + schema.children, [name](const auto& child) { return child->name == name; }); + return child_it == schema.children.end() ? nullptr : child_it->get(); +} + +bool collect_variant_residual_leaf_ids(const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection, + std::vector* residual_leaf_ids) { + DORIS_CHECK(residual_leaf_ids != nullptr); + const auto* value = schema_child_by_name(schema, "value"); + const auto* typed_value = schema_child_by_name(schema, "typed_value"); + if (value != nullptr && typed_value != nullptr) { + if (value->kind != ParquetColumnSchemaKind::PRIMITIVE || value->leaf_column_id < 0) { + return false; + } + residual_leaf_ids->push_back(value->leaf_column_id); + } + for (const auto& child_projection : projection.children) { + const auto* child = projected_schema_child(schema, child_projection.local_id()); + if (child == nullptr || + !collect_variant_residual_leaf_ids(*child, child_projection, residual_leaf_ids)) { + return false; + } + } + return true; +} + +bool detail::variant_projection_is_fully_shredded(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection) { + if (schema.kind != ParquetColumnSchemaKind::VARIANT || schema.max_repetition_level != 0 || + !format::is_partial_projection(&projection)) { + return false; + } + std::vector residual_leaf_ids; + if (!collect_variant_residual_leaf_ids(schema, projection, &residual_leaf_ids)) { + return false; + } + std::ranges::sort(residual_leaf_ids); + residual_leaf_ids.erase(std::unique(residual_leaf_ids.begin(), residual_leaf_ids.end()), + residual_leaf_ids.end()); + for (const auto& row_group : metadata.row_groups) { + for (const int leaf_id : residual_leaf_ids) { + if (leaf_id < 0 || leaf_id >= static_cast(row_group.columns.size())) { + return false; + } + const auto& chunk = row_group.columns[leaf_id]; + if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics || + !chunk.meta_data.statistics.__isset.null_count || + chunk.meta_data.statistics.null_count != row_group.num_rows) { + return false; + } + } + } + return true; +} + +size_t detail::finalize_variant_leaf_projection(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + format::LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); + if (!format::is_partial_projection(projection)) { + return 0; + } + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + if (variant_projection_is_fully_shredded(metadata, schema, *projection)) { + return 1; + } + // Unknown residual completeness must restore this Variant wrapper atomically. For a + // repeated ancestor, footer null_count is in the leaf-value domain rather than Variant + // instances, so variant_projection_is_fully_shredded() deliberately takes this fallback. + projection->project_all_children = true; + projection->children.clear(); + return 0; + } + + size_t retained = 0; + for (auto& child_projection : projection->children) { + const auto* child_schema = projected_schema_child(schema, child_projection.local_id()); + DORIS_CHECK(child_schema != nullptr); + retained += finalize_variant_leaf_projection(metadata, *child_schema, &child_projection); + } + return retained; +} + +size_t finalize_variant_leaf_projections( + const NativeParquetMetadata& metadata, + const std::vector>& file_schema, + std::vector* projections) { + DORIS_CHECK(projections != nullptr); + size_t retained = 0; + for (auto& projection : *projections) { + const int32_t local_id = projection.local_id(); + if (local_id < 0 || local_id >= static_cast(file_schema.size())) { + continue; + } + retained += detail::finalize_variant_leaf_projection(metadata.to_thrift(), + *file_schema[local_id], &projection); + } + return retained; +} + Status validate_all_projected_leaves_supported(const ParquetColumnSchema& column_schema) { if (column_schema.kind == ParquetColumnSchemaKind::PRIMITIVE) { if (!column_schema.type_descriptor.unsupported_reason.empty()) { @@ -406,6 +518,17 @@ Status ParquetReader::open(std::shared_ptr request) { } auto request_snapshot = request; DORIS_CHECK(request_snapshot != nullptr); + const size_t retained_variant_leaf_projections = + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->predicate_columns) + + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->non_predicate_columns); + if (_parquet_profile.variant_leaf_projections != nullptr) { + COUNTER_UPDATE(_parquet_profile.variant_leaf_projections, + retained_variant_leaf_projections); + } RETURN_IF_ERROR(format::FileReader::open(std::move(request))); // `local_positions.empty()` means all columns are needed by table reader diff --git a/be/src/format_v2/parquet/parquet_reader.h b/be/src/format_v2/parquet/parquet_reader.h index fe95b93a9e0101..0a2b3791502736 100644 --- a/be/src/format_v2/parquet/parquet_reader.h +++ b/be/src/format_v2/parquet/parquet_reader.h @@ -36,6 +36,15 @@ namespace doris::format::parquet { struct ParquetReaderScanState; +namespace detail { +bool variant_projection_is_fully_shredded(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection); +size_t finalize_variant_leaf_projection(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + format::LocalColumnIndex* projection); +} // namespace detail + // ============================================================================ // ============================================================================ // init() -> get_schema() -> open(request) -> get_block() [loop] -> close() diff --git a/be/src/format_v2/parquet/parquet_scan.cpp b/be/src/format_v2/parquet/parquet_scan.cpp index 18b42d90a96368..630e4d8121c970 100644 --- a/be/src/format_v2/parquet/parquet_scan.cpp +++ b/be/src/format_v2/parquet/parquet_scan.cpp @@ -259,7 +259,7 @@ void materialize_count_star_placeholders(const format::FileScanRequest& request, if (!request.is_count_star_placeholder(column.column_id())) { continue; } - const auto block_position = request.local_positions.at(column.column_id()).value(); + const auto block_position = request.non_predicate_position(column.column_id()).value(); auto placeholder = file_block->get_by_position(block_position).column->assert_mutable(); DCHECK(placeholder->empty()); placeholder->insert_many_defaults(rows); @@ -475,8 +475,9 @@ Status finalize_native_row_group_read_plan( std::vector page_selected_ranges; std::map page_skip_plans; RETURN_IF_ERROR(select_row_group_ranges_by_native_page_index( - thrift, page_indexes, file_schema, request, row_group_plan->row_group_rows, - &page_selected_ranges, &page_skip_plans, pruning_stats, timezone, runtime_state)); + thrift, thrift.row_groups[row_group_plan->row_group_id], page_indexes, file_schema, + request, row_group_plan->row_group_rows, &page_selected_ranges, &page_skip_plans, + pruning_stats, timezone, runtime_state)); row_group_plan->selected_ranges = intersect_row_ranges(row_group_plan->selected_ranges, page_selected_ranges); row_group_plan->page_skip_plans = std::move(page_skip_plans); @@ -1176,9 +1177,17 @@ Status ParquetScanScheduler::open_next_row_group( RETURN_IF_ERROR(detail::build_native_prefetch_ranges( thrift_metadata, file_schema, request_scan_columns(request), row_group_idx, file_context.native_file->size(), compat.parquet_816_padding, &native_ranges)); - _current_merge_range_active = file_context.set_native_random_access_ranges( - native_ranges, detail::average_prefetch_range_size(native_ranges), _profile, - _merge_read_slice_size); + if (request.non_predicate_positions.empty()) { + _current_merge_range_active = file_context.set_native_random_access_ranges( + native_ranges, detail::average_prefetch_range_size(native_ranges), _profile, + _merge_read_slice_size); + } else { + // Independent predicate/output readers may revisit the same physical leaf at different + // cursors. MergeRangeFileReader has one consumptive cache per range, so use the random + // access reader for this layout instead of sharing one sequential range cache. + _current_merge_range_active = file_context.set_native_random_access_ranges( + {}, 0, _profile, _merge_read_slice_size); + } for (const auto& col : request.predicate_columns) { const auto local_id = col.column_id(); @@ -2624,9 +2633,7 @@ Status ParquetScanScheduler::read_current_row_group_batch( // selection vector. This also merges pending range gaps with fully filtered batches. RETURN_IF_ERROR(flush_pending_non_predicate_skip_rows()); for (const auto& [fid, column_reader] : _current_non_predicate_columns) { - auto position_it = request.local_positions.find(fid); - DORIS_CHECK(position_it != request.local_positions.end()); - const auto block_position = position_it->second.value(); + const auto block_position = request.non_predicate_position(fid).value(); auto column = file_block->get_by_position(block_position).column->assert_mutable(); DCHECK_EQ(file_block->get_by_position(block_position).type->get_primitive_type(), column_reader->type()->get_primitive_type()) @@ -2694,9 +2701,7 @@ Status ParquetScanScheduler::materialize_pending_predicate_batch( SCOPED_TIMER(_scan_profile.column_read_time); RETURN_IF_ERROR(flush_pending_non_predicate_skip_rows()); for (const auto& [fid, column_reader] : _current_non_predicate_columns) { - auto position_it = request.local_positions.find(fid); - DORIS_CHECK(position_it != request.local_positions.end()); - const auto block_position = position_it->second.value(); + const auto block_position = request.non_predicate_position(fid).value(); auto column = file_block->get_by_position(block_position).column->assert_mutable(); [[maybe_unused]] const auto old_size = column->size(); RETURN_IF_ERROR(column_reader->select(_pending_output_selection, diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 7f35b087910353..b4f036773573bb 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -37,7 +37,10 @@ #include "core/data_type_serde/data_type_serde.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" +#include "exprs/vectorized_fn_call.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" +#include "exprs/vslot_ref.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_file_context.h" #include "format_v2/parquet/reader/native/block_split_bloom_filter.h" @@ -452,6 +455,322 @@ std::optional file_column_id_by_block_position( return std::nullopt; } +enum class VariantComparisonOp { EQ, NE, LT, LE, GT, GE }; + +struct VariantShreddedPredicate { + int slot_index = -1; + std::vector path; + DataTypePtr comparison_type; + DataTypePtr literal_type; + Field literal; + VariantComparisonOp op = VariantComparisonOp::EQ; +}; + +std::string callable_name(const VExprSPtr& expr) { + if (const auto function = std::dynamic_pointer_cast(expr); + function != nullptr) { + return function->function_name(); + } + return expr == nullptr ? std::string {} : expr->expr_name(); +} + +std::optional variant_comparison_op(std::string_view name) { + if (name == "eq") { + return VariantComparisonOp::EQ; + } + if (name == "ne") { + return VariantComparisonOp::NE; + } + if (name == "lt") { + return VariantComparisonOp::LT; + } + if (name == "le") { + return VariantComparisonOp::LE; + } + if (name == "gt") { + return VariantComparisonOp::GT; + } + if (name == "ge") { + return VariantComparisonOp::GE; + } + return std::nullopt; +} + +VariantComparisonOp reverse_variant_comparison(VariantComparisonOp op) { + switch (op) { + case VariantComparisonOp::EQ: + case VariantComparisonOp::NE: + return op; + case VariantComparisonOp::LT: + return VariantComparisonOp::GT; + case VariantComparisonOp::LE: + return VariantComparisonOp::GE; + case VariantComparisonOp::GT: + return VariantComparisonOp::LT; + case VariantComparisonOp::GE: + return VariantComparisonOp::LE; + } + __builtin_unreachable(); +} + +std::optional> variant_literal(const VExprSPtr& expr) { + const auto literal = std::dynamic_pointer_cast(expr); + if (literal == nullptr || !literal->get_column_ptr() || literal->get_column_ptr()->empty()) { + return std::nullopt; + } + Field value; + literal->get_column_ptr()->get(0, value); + if (value.is_null()) { + return std::nullopt; + } + return std::make_pair(std::move(value), literal->get_data_type()); +} + +std::optional extract_variant_shredded_predicate( + const VExprContextSPtr& conjunct) { + if (conjunct == nullptr || conjunct->root() == nullptr || + conjunct->root()->get_num_children() != 2) { + return std::nullopt; + } + auto op = variant_comparison_op(callable_name(conjunct->root())); + if (!op.has_value()) { + return std::nullopt; + } + + VExprSPtr value_expr; + std::optional> literal; + if ((literal = variant_literal(conjunct->root()->get_child(1))).has_value()) { + value_expr = conjunct->root()->get_child(0); + } else if ((literal = variant_literal(conjunct->root()->get_child(0))).has_value()) { + value_expr = conjunct->root()->get_child(1); + op = reverse_variant_comparison(*op); + } else { + return std::nullopt; + } + + const auto comparison_type = value_expr->data_type(); + while (value_expr->node_type() == TExprNodeType::CAST_EXPR && + value_expr->get_num_children() == 1) { + if (!expr_zonemap::data_types_compatible(value_expr->data_type(), comparison_type)) { + // Every removed cast must preserve the comparison domain. Otherwise bounds for the + // raw typed leaf could skip rows whose value changes in an intermediate narrowing cast. + return std::nullopt; + } + value_expr = value_expr->get_child(0); + } + + std::vector reverse_path; + while (callable_name(value_expr) == "element_at" && value_expr->get_num_children() == 2) { + const auto key = variant_literal(value_expr->get_child(1)); + if (!key.has_value() || key->first.get_type() != TYPE_STRING) { + // Repeated array shredding has no single scalar page range, so only object keys are + // eligible for this file-level optimization. + return std::nullopt; + } + reverse_path.push_back(key->first.get()); + value_expr = value_expr->get_child(0); + } + const auto slot = std::dynamic_pointer_cast(value_expr); + if (slot == nullptr || reverse_path.empty() || comparison_type == nullptr || + remove_nullable(slot->data_type())->get_primitive_type() != TYPE_VARIANT || + !expr_zonemap::data_types_compatible(comparison_type, literal->second)) { + return std::nullopt; + } + std::ranges::reverse(reverse_path); + return VariantShreddedPredicate {.slot_index = slot->column_id(), + .path = std::move(reverse_path), + .comparison_type = comparison_type, + .literal_type = literal->second, + .literal = std::move(literal->first), + .op = *op}; +} + +bool has_variant_shredded_filter(const format::FileScanRequest& request) { + return std::ranges::any_of(request.conjuncts, [](const auto& conjunct) { + return extract_variant_shredded_predicate(conjunct).has_value(); + }); +} + +const ParquetColumnSchema* child_named(const ParquetColumnSchema& parent, std::string_view name) { + const auto it = std::ranges::find_if(parent.children, [&](const auto& child) { + return child != nullptr && child->name == name; + }); + return it == parent.children.end() ? nullptr : it->get(); +} + +struct ResolvedVariantShredding { + const ParquetColumnSchema* fallback_value = nullptr; + const ParquetColumnSchema* typed_value = nullptr; +}; + +bool metadata_cast_is_order_preserving(const DataTypePtr& source, const DataTypePtr& target) { + if (expr_zonemap::data_types_compatible(source, target)) { + return true; + } + const auto source_type = remove_nullable(source); + const auto target_type = remove_nullable(target); + const auto source_primitive = source_type->get_primitive_type(); + const auto target_primitive = target_type->get_primitive_type(); + // Metadata bounds may cross only exact widening domains. This mirrors the residual CAST while + // excluding rounding, overflow, and narrowing cases that could reverse a pruning decision. + if (source_primitive == TYPE_FLOAT && target_primitive == TYPE_DOUBLE) { + return true; + } + if (is_int(source_primitive) && source_primitive != TYPE_LARGEINT && + is_decimalv3(target_primitive)) { + const uint32_t required_integer_digits = source_primitive == TYPE_TINYINT ? 3 + : source_primitive == TYPE_SMALLINT ? 5 + : source_primitive == TYPE_INT ? 10 + : 19; + return target_type->get_precision() >= target_type->get_scale() && + target_type->get_precision() - target_type->get_scale() >= required_integer_digits; + } + if (is_decimalv3(source_primitive) && is_decimalv3(target_primitive)) { + const uint32_t source_integer_digits = + source_type->get_precision() - source_type->get_scale(); + const uint32_t target_integer_digits = + target_type->get_precision() - target_type->get_scale(); + return target_integer_digits >= source_integer_digits && + target_type->get_scale() >= source_type->get_scale(); + } + return false; +} + +std::optional cast_metadata_field(const Field& value, const DataTypePtr& source, + const DataTypePtr& target) { + if (expr_zonemap::data_types_compatible(source, target)) { + return value; + } + const auto source_type = remove_nullable(source); + const auto target_type = remove_nullable(target); + if (source_type->get_primitive_type() == TYPE_FLOAT && + target_type->get_primitive_type() == TYPE_DOUBLE) { + return Field::create_field(static_cast(value.get())); + } + try { + auto source_column = source_type->create_column(); + source_column->insert(value); + DataTypeSerDe::FormatOptions options = DataTypeSerDe::get_default_format_options(); + options.converted_from_string = true; + std::string text = source_type->to_string(*source_column, 0, options); + StringRef input(text.data(), text.size()); + auto target_column = target_type->create_column(); + if (!target_type->get_serde() + ->from_string_strict_mode(input, *target_column, options) + .ok() || + target_column->size() != 1) { + return std::nullopt; + } + Field result; + target_column->get(0, result); + return result; + } catch (...) { + return std::nullopt; + } +} + +std::optional normalize_variant_statistics( + const VariantShreddedPredicate& predicate, const ParquetColumnSchema& typed_value, + const ParquetColumnStatistics& statistics) { + if (!statistics.has_min_max || + expr_zonemap::data_types_compatible(typed_value.type, predicate.comparison_type)) { + return statistics; + } + auto min_value = + cast_metadata_field(statistics.min_value, typed_value.type, predicate.comparison_type); + auto max_value = + cast_metadata_field(statistics.max_value, typed_value.type, predicate.comparison_type); + if (!min_value.has_value() || !max_value.has_value()) { + return std::nullopt; + } + auto normalized = statistics; + normalized.min_value = std::move(*min_value); + normalized.max_value = std::move(*max_value); + return normalized; +} + +std::optional resolve_variant_shredding( + const std::vector>& file_schema, + const format::FileScanRequest& request, const VariantShreddedPredicate& predicate) { + const auto local_id = file_column_id_by_block_position(request, predicate.slot_index); + if (!local_id.has_value() || local_id->value() < 0 || + local_id->value() >= static_cast(file_schema.size())) { + return std::nullopt; + } + const ParquetColumnSchema* wrapper = file_schema[local_id->value()].get(); + if (wrapper == nullptr || wrapper->kind != ParquetColumnSchemaKind::VARIANT) { + return std::nullopt; + } + for (const auto& component : predicate.path) { + const auto* typed_object = child_named(*wrapper, "typed_value"); + if (typed_object == nullptr || typed_object->kind != ParquetColumnSchemaKind::STRUCT) { + return std::nullopt; + } + wrapper = child_named(*typed_object, component); + if (wrapper == nullptr || wrapper->kind != ParquetColumnSchemaKind::STRUCT) { + return std::nullopt; + } + } + const auto* fallback = child_named(*wrapper, "value"); + const auto* typed = child_named(*wrapper, "typed_value"); + const auto typed_primitive = typed == nullptr || typed->type == nullptr + ? INVALID_TYPE + : remove_nullable(typed->type)->get_primitive_type(); + if (fallback == nullptr || typed == nullptr || + fallback->kind != ParquetColumnSchemaKind::PRIMITIVE || + typed->kind != ParquetColumnSchemaKind::PRIMITIVE || typed->max_repetition_level != 0 || + // Parquet float statistics do not prove that a page contains no NaN. Min/max pruning in + // the presence of NaN is not order preserving, so keep those pages until such proof exists. + typed_primitive == TYPE_FLOAT || typed_primitive == TYPE_DOUBLE || + !metadata_cast_is_order_preserving(typed->type, predicate.comparison_type) || + !expr_zonemap::data_types_compatible(predicate.comparison_type, predicate.literal_type)) { + return std::nullopt; + } + return ResolvedVariantShredding {.fallback_value = fallback, .typed_value = typed}; +} + +bool fallback_is_all_null(const tparquet::RowGroup& row_group, + const ParquetColumnSchema& fallback) { + if (fallback.max_repetition_level != 0 || fallback.leaf_column_id < 0 || + fallback.leaf_column_id >= static_cast(row_group.columns.size())) { + return false; + } + const auto& chunk = row_group.columns[fallback.leaf_column_id]; + return row_group.num_rows >= 0 && chunk.__isset.meta_data && + chunk.meta_data.num_values == row_group.num_rows && chunk.meta_data.__isset.statistics && + chunk.meta_data.statistics.__isset.null_count && + chunk.meta_data.statistics.null_count == chunk.meta_data.num_values; +} + +bool variant_statistics_exclude(const VariantShreddedPredicate& predicate, + const ParquetColumnStatistics& statistics) { + if (!statistics.has_any_statistics()) { + return false; + } + if (!statistics.has_not_null) { + return true; + } + if (!statistics.has_min_max) { + return false; + } + const auto& literal = predicate.literal; + switch (predicate.op) { + case VariantComparisonOp::EQ: + return literal < statistics.min_value || statistics.max_value < literal; + case VariantComparisonOp::NE: + return statistics.min_value == literal && statistics.max_value == literal; + case VariantComparisonOp::LT: + return statistics.min_value >= literal; + case VariantComparisonOp::LE: + return statistics.min_value > literal; + case VariantComparisonOp::GT: + return statistics.max_value <= literal; + case VariantComparisonOp::GE: + return statistics.max_value < literal; + } + __builtin_unreachable(); +} + bool has_expr_zonemap_filter(const format::FileScanRequest& request, const RuntimeState*) { // FileScannerV2 metadata pruning is a fixed part of its scan pipeline and must not inherit // the legacy scanner's expression ZoneMap session gate. @@ -463,7 +782,7 @@ bool has_expr_zonemap_filter(const format::FileScanRequest& request, const Runti return true; } } - return false; + return has_variant_shredded_filter(request); } std::set collect_expr_zonemap_slot_indexes(const VExprContextSPtrs& conjuncts) { @@ -617,9 +936,11 @@ void collect_filtered_leaf_ids(const ParquetColumnSchema& column_schema, if (!format::is_child_projected(projection, child_schema->local_id)) { continue; } - collect_filtered_leaf_ids(*child_schema, - format::find_child_projection(projection, child_schema->local_id), - leaf_column_ids); + // The leaf set must match the physical projection. A complete Variant projection naturally + // reaches every sibling; a validated typed-leaf projection reads only retained children. + const auto* child_projection = + format::find_child_projection(projection, child_schema->local_id); + collect_filtered_leaf_ids(*child_schema, child_projection, leaf_column_ids); } } @@ -627,7 +948,21 @@ bool native_metadata_predicate_is_type_safe(const ParquetColumnSchema& column_sc DORIS_CHECK(column_schema.type != nullptr); // Raw VARBINARY file slots may feed table-side STRING casts. Footer/page metadata is still in // the pre-cast domain, so using it for a rewritten table predicate can cause false negatives. - return remove_nullable(column_schema.type)->get_primitive_type() != TYPE_VARBINARY; + if (remove_nullable(column_schema.type)->get_primitive_type() == TYPE_VARBINARY) { + return false; + } + // UUID readers render canonical text, so their physical 16-byte bounds are not STRING bounds. + return !column_schema.type_descriptor.is_uuid; +} + +bool variant_metadata_predicate_is_type_safe(const ParquetColumnSchema& column_schema) { + if (!native_metadata_predicate_is_type_safe(column_schema)) { + return false; + } + const auto& descriptor = column_schema.type_descriptor; + // An ordinary raw-binary STRING slot preserves its bytes, but Variant reconstruction renders + // the binary identity before the residual STRING cast and therefore changes the domain. + return !descriptor.is_string_like || descriptor.is_string_annotation; } bool check_native_statistics(const tparquet::FileMetaData& metadata, @@ -675,6 +1010,49 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, return result == ZoneMapFilterResult::kNoMatch; } +bool check_shredded_variant_statistics( + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, + const std::vector>& file_schema, + const format::FileScanRequest& request, const cctz::time_zone* timezone) { + for (const auto& conjunct : request.conjuncts) { + const auto predicate = extract_variant_shredded_predicate(conjunct); + if (!predicate.has_value()) { + continue; + } + const auto shredding = resolve_variant_shredding(file_schema, request, *predicate); + if (!shredding.has_value() || shredding->typed_value->leaf_column_id < 0 || + shredding->typed_value->leaf_column_id >= static_cast(row_group.columns.size()) || + !fallback_is_all_null(row_group, *shredding->fallback_value) || + !variant_metadata_predicate_is_type_safe(*shredding->typed_value) || + !detail::has_supported_type_defined_order(metadata, + shredding->typed_value->leaf_column_id)) { + continue; + } + const auto& chunk = row_group.columns[shredding->typed_value->leaf_column_id]; + if (!chunk.__isset.meta_data) { + continue; + } + const auto& column_metadata = chunk.meta_data; + if (column_metadata.num_values != row_group.num_rows) { + continue; + } + std::optional safe_statistics; + if (column_metadata.__isset.statistics) { + safe_statistics = detail::sanitize_native_footer_statistics( + shredding->typed_value->type_descriptor, column_metadata.statistics, true); + } + const auto statistics = ParquetStatisticsUtils::TransformColumnStatistics( + *shredding->typed_value, safe_statistics.has_value() ? &*safe_statistics : nullptr, + column_metadata.num_values, timezone); + const auto normalized = + normalize_variant_statistics(*predicate, *shredding->typed_value, statistics); + if (normalized.has_value() && variant_statistics_exclude(*predicate, *normalized)) { + return true; + } + } + return false; +} + bool is_native_dictionary_data_encoding(tparquet::Encoding::type encoding) { return encoding == tparquet::Encoding::PLAIN_DICTIONARY || encoding == tparquet::Encoding::RLE_DICTIONARY; @@ -923,8 +1301,10 @@ Status select_row_groups_by_metadata( ParquetRowGroupPruneReason prune_reason = ParquetRowGroupPruneReason::NONE; if (probe_mode != ParquetMetadataProbeMode::EXPENSIVE_ONLY && has_expr_zonemap_filter(request, runtime_state) && - check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, - timezone)) { + (check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, + timezone) || + check_shredded_variant_statistics(metadata, row_group, file_schema, request, + timezone))) { prune_reason = ParquetRowGroupPruneReason::STATISTICS; } if (probe_mode != ParquetMetadataProbeMode::FOOTER_ONLY && @@ -1029,11 +1409,16 @@ void collect_leaf_schemas(const ParquetColumnSchema& column_schema, return; } for (const auto& child_schema : column_schema.children) { - if (!format::is_child_projected(projection, child_schema->local_id)) { + if (column_schema.kind != ParquetColumnSchemaKind::VARIANT && + !format::is_child_projected(projection, child_schema->local_id)) { continue; } + // A logical Variant projection materializes every physical sibling; build skip plans for + // that identical leaf set so shredded columns cannot drift to different row positions. const auto* child_projection = - format::find_child_projection(projection, child_schema->local_id); + column_schema.kind == ParquetColumnSchemaKind::VARIANT + ? nullptr + : format::find_child_projection(projection, child_schema->local_id); collect_leaf_schemas(*child_schema, child_projection, leaf_schemas); } } @@ -1238,7 +1623,7 @@ RowRange native_page_row_range(const tparquet::OffsetIndex& offset_index, size_t } // namespace Status select_row_group_ranges_by_native_page_index( - const tparquet::FileMetaData& metadata, + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::unordered_map& page_indexes, const std::vector>& file_schema, const format::FileScanRequest& request, int64_t row_group_rows, @@ -1324,6 +1709,59 @@ Status select_row_group_ranges_by_native_page_index( } } + for (const auto& conjunct : request.conjuncts) { + const auto predicate = extract_variant_shredded_predicate(conjunct); + if (!predicate.has_value()) { + continue; + } + const auto shredding = resolve_variant_shredding(file_schema, request, *predicate); + if (!shredding.has_value() || shredding->typed_value->leaf_column_id < 0 || + !fallback_is_all_null(row_group, *shredding->fallback_value) || + !variant_metadata_predicate_is_type_safe(*shredding->typed_value) || + !detail::has_supported_type_defined_order(metadata, + shredding->typed_value->leaf_column_id)) { + continue; + } + const auto index_it = page_indexes.find(shredding->typed_value->leaf_column_id); + if (index_it == page_indexes.end()) { + continue; + } + const auto& indexes = index_it->second; + std::vector filter_ranges; + bool usable = true; + for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size(); + ++page_idx) { + const auto page_range = + native_page_row_range(indexes.offset_index, page_idx, row_group_rows); + ParquetColumnStatistics statistics; + if (!build_native_page_statistics(indexes.column_index, *shredding->typed_value, + page_idx, page_range.length, &statistics, timezone)) { + usable = false; + break; + } + const auto normalized = + normalize_variant_statistics(*predicate, *shredding->typed_value, statistics); + if (!normalized.has_value()) { + usable = false; + break; + } + if (!variant_statistics_exclude(*predicate, *normalized)) { + append_row_range(page_range, &filter_ranges); + } + } + if (!usable) { + continue; + } + *selected_ranges = intersect_ranges(*selected_ranges, filter_ranges); + if (selected_ranges->empty()) { + if (pruning_stats != nullptr) { + pruning_stats->filtered_page_rows += row_group_rows; + ++pruning_stats->filtered_row_groups_by_page_index; + } + return Status::OK(); + } + } + if (page_skip_plans != nullptr) { std::vector leaves; collect_request_leaf_schemas(file_schema, request, &leaves); diff --git a/be/src/format_v2/parquet/parquet_statistics.h b/be/src/format_v2/parquet/parquet_statistics.h index 72381548656f9d..611ec5abe97ac2 100644 --- a/be/src/format_v2/parquet/parquet_statistics.h +++ b/be/src/format_v2/parquet/parquet_statistics.h @@ -145,7 +145,7 @@ Status select_row_groups_by_metadata( ParquetMetadataProbeMode probe_mode = ParquetMetadataProbeMode::ALL); Status select_row_group_ranges_by_native_page_index( - const tparquet::FileMetaData& metadata, + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::unordered_map& page_indexes, const std::vector>& file_schema, const format::FileScanRequest& request, int64_t row_group_rows, diff --git a/be/src/format_v2/parquet/parquet_type.h b/be/src/format_v2/parquet/parquet_type.h index 7ab3d2b3b39d8d..f06f4c25dd8ba2 100644 --- a/be/src/format_v2/parquet/parquet_type.h +++ b/be/src/format_v2/parquet/parquet_type.h @@ -67,6 +67,8 @@ struct ParquetTypeDescriptor { bool is_timestamp = false; // whether this is a timestamp type bool timestamp_is_adjusted_to_utc = false; // whether the timestamp is UTC-normalized bool is_string_like = false; // binary type that is neither decimal nor FLOAT16 + bool is_string_annotation = false; // STRING logical type, distinct from raw BINARY + bool is_uuid = false; // UUID logical type over FIXED_LEN_BYTE_ARRAY(16) std::string unsupported_reason; // non-empty when this Parquet logical type is unsupported }; diff --git a/be/src/format_v2/parquet/reader/count_column_reader.cpp b/be/src/format_v2/parquet/reader/count_column_reader.cpp index 171107107f1a38..8be684d7235957 100644 --- a/be/src/format_v2/parquet/reader/count_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/count_column_reader.cpp @@ -69,6 +69,17 @@ Status find_count_leaf(const ParquetColumnSchema& schema, // reading a potentially huge value BYTE_ARRAY for COUNT(map_col). DORIS_CHECK(!schema.children.empty()); return find_count_leaf(*schema.children.front(), nullptr, leaf); + case ParquetColumnSchemaKind::VARIANT: { + // The required metadata leaf is present exactly when the enclosing Variant group is + // present, so its levels preserve COUNT(variant_col) SQL-null semantics without decoding + // any Variant payload. + const auto metadata = std::ranges::find_if( + schema.children, [](const auto& child) { return child->name == "metadata"; }); + if (metadata == schema.children.end()) { + return Status::Corruption("Parquet Variant {} has no metadata column", schema.name); + } + return find_count_leaf(**metadata, nullptr, leaf); + } } return Status::InternalError("Unknown Parquet schema kind for column {}", schema.name); } diff --git a/be/src/format_v2/parquet/reader/native_column_reader.cpp b/be/src/format_v2/parquet/reader/native_column_reader.cpp index 6b686597fa600e..618eca6f6a50ce 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native_column_reader.cpp @@ -42,6 +42,7 @@ #include "format_v2/column_data.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_file_context.h" +#include "format_v2/parquet/reader/variant_column_reader.h" #include "runtime/runtime_state.h" namespace doris::format::parquet { @@ -50,25 +51,57 @@ namespace { constexpr size_t MAX_RETAINED_BATCH_SCRATCH_BYTES = 4UL << 20; DataTypePtr projected_type(const ParquetColumnSchema& schema, - const format::LocalColumnIndex* projection) { - if (!format::is_partial_projection(projection)) { - return schema.type; - } + const format::LocalColumnIndex* projection, bool physical_variant) { switch (schema.kind) { case ParquetColumnSchemaKind::PRIMITIVE: return schema.type; + case ParquetColumnSchemaKind::VARIANT: + DORIS_CHECK(schema.variant_physical_type != nullptr); + if (!physical_variant || !format::is_partial_projection(projection)) { + return physical_variant ? schema.variant_physical_type : schema.type; + } + { + DataTypes child_types; + Strings child_names; + child_types.reserve(projection->children.size()); + child_names.reserve(projection->children.size()); + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + child_types.push_back(make_nullable( + projected_type(**child_it, &child_projection, physical_variant))); + child_names.push_back((*child_it)->name); + } + DataTypePtr type = std::make_shared(std::move(child_types), + std::move(child_names)); + return schema.variant_physical_type->is_nullable() ? make_nullable(std::move(type)) + : std::move(type); + } case ParquetColumnSchemaKind::STRUCT: { DataTypes child_types; Strings child_names; - child_types.reserve(projection->children.size()); - child_names.reserve(projection->children.size()); - for (const auto& child_projection : projection->children) { - const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { - return child->local_id == child_projection.local_id(); - }); - DORIS_CHECK(child_it != schema.children.end()); - child_types.push_back(make_nullable(projected_type(**child_it, &child_projection))); - child_names.push_back((*child_it)->name); + if (format::is_partial_projection(projection)) { + child_types.reserve(projection->children.size()); + child_names.reserve(projection->children.size()); + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + child_types.push_back(make_nullable( + projected_type(**child_it, &child_projection, physical_variant))); + child_names.push_back((*child_it)->name); + } + } else { + child_types.reserve(schema.children.size()); + child_names.reserve(schema.children.size()); + for (const auto& child : schema.children) { + child_types.push_back( + make_nullable(projected_type(*child, nullptr, physical_variant))); + child_names.push_back(child->name); + } } DataTypePtr type = std::make_shared(child_types, child_names); return schema.type->is_nullable() ? make_nullable(type) : type; @@ -76,20 +109,25 @@ DataTypePtr projected_type(const ParquetColumnSchema& schema, case ParquetColumnSchemaKind::LIST: { DORIS_CHECK(schema.children.size() == 1); const auto* child_projection = - format::find_child_projection(projection, schema.children[0]->local_id); - DORIS_CHECK(child_projection != nullptr); + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[0]->local_id) + : nullptr; + DORIS_CHECK(!format::is_partial_projection(projection) || child_projection != nullptr); DataTypePtr type = std::make_shared( - projected_type(*schema.children[0], child_projection)); + projected_type(*schema.children[0], child_projection, physical_variant)); return schema.type->is_nullable() ? make_nullable(type) : type; } case ParquetColumnSchemaKind::MAP: { DORIS_CHECK(schema.children.size() == 2); const auto* value_projection = - format::find_child_projection(projection, schema.children[1]->local_id); - DORIS_CHECK(value_projection != nullptr); + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[1]->local_id) + : nullptr; + DORIS_CHECK(!format::is_partial_projection(projection) || value_projection != nullptr); DataTypePtr type = std::make_shared( - make_nullable(schema.children[0]->type), - make_nullable(projected_type(*schema.children[1], value_projection))); + make_nullable(projected_type(*schema.children[0], nullptr, physical_variant)), + make_nullable( + projected_type(*schema.children[1], value_projection, physical_variant))); return schema.type->is_nullable() ? make_nullable(type) : type; } } @@ -97,6 +135,64 @@ DataTypePtr projected_type(const ParquetColumnSchema& schema, return nullptr; } +std::unique_ptr build_variant_plan( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection) { + auto plan = std::make_unique(); + plan->schema = &schema; + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + plan->contains_variant = true; + if (projection != nullptr) { + plan->variant_projection = *projection; + } + plan->variant_state_schema = create_variant_state_schema(schema, projection); + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::PRIMITIVE) { + return plan; + } + + auto append_child = [&](const ParquetColumnSchema& child, + const format::LocalColumnIndex* child_projection) { + auto child_plan = build_variant_plan(child, child_projection); + plan->contains_variant = plan->contains_variant || child_plan->contains_variant; + plan->children.push_back(std::move(child_plan)); + }; + if (schema.kind == ParquetColumnSchemaKind::STRUCT && + format::is_partial_projection(projection)) { + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + append_child(**child_it, &child_projection); + } + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::LIST) { + DORIS_CHECK(schema.children.size() == 1); + const auto* child_projection = + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[0]->local_id) + : nullptr; + append_child(*schema.children[0], child_projection); + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::MAP) { + DORIS_CHECK(schema.children.size() == 2); + append_child(*schema.children[0], nullptr); + const auto* value_projection = + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[1]->local_id) + : nullptr; + append_child(*schema.children[1], value_projection); + return plan; + } + for (const auto& child : schema.children) { + append_child(*child, nullptr); + } + return plan; +} + const NativeFieldSchema* find_child_field(const NativeFieldSchema& parent, const ParquetColumnSchema& child) { auto field_it = std::ranges::find_if(parent.children, [&](const NativeFieldSchema& field) { @@ -149,10 +245,13 @@ void collect_projected_ids(const ParquetColumnSchema& schema, } // namespace -NativeColumnReader::NativeColumnReader(const ParquetColumnSchema& schema, - DataTypePtr projected_type, +NativeColumnReader::NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr logical_type, + DataTypePtr native_type, + std::unique_ptr variant_plan, ParquetColumnReaderProfile profile) - : ParquetColumnReader(schema, std::move(projected_type), profile), + : ParquetColumnReader(schema, std::move(logical_type), profile), + _native_type(std::move(native_type)), + _variant_plan(std::move(variant_plan)), _nested(schema.kind != ParquetColumnSchemaKind::PRIMITIVE) {} NativeColumnReader::~NativeColumnReader() { @@ -192,14 +291,17 @@ Status NativeColumnReader::create( column_schema.local_id, field->name, column_schema.name); } - auto type = projected_type(column_schema, projection); + auto logical_type = projected_type(column_schema, projection, false); + auto native_type = projected_type(column_schema, projection, true); std::shared_ptr schema_node; - RETURN_IF_ERROR(build_native_schema_node(type, column_schema, &schema_node)); + RETURN_IF_ERROR(build_native_schema_node(native_type, column_schema, &schema_node)); std::set projected_ids; collect_projected_ids(column_schema, projection, *field, &projected_ids); + auto variant_plan = build_variant_plan(column_schema, projection); auto native_reader = std::unique_ptr( - new NativeColumnReader(column_schema, std::move(type), profile)); + new NativeColumnReader(column_schema, std::move(logical_type), std::move(native_type), + std::move(variant_plan), profile)); RETURN_IF_ERROR(native_reader->init( std::move(file), metadata, row_group_id, field, std::move(schema_node), std::move(projected_ids), selected_ranges, offset_indexes, timezone, io_ctx, @@ -261,7 +363,10 @@ Status NativeColumnReader::init( _filter_column_ids, page_cache_file_key, compat, runtime_state != nullptr && runtime_state->enable_strict_mode())); DORIS_CHECK(_native_reader != nullptr); - _skip_column = _type->create_column(); + _skip_column = _native_type->create_column(); + if (_variant_plan->contains_variant) { + _variant_physical_column = _native_type->create_column(); + } return Status::OK(); } @@ -281,15 +386,22 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ native::FilterMap filter; RETURN_IF_ERROR(filter.init(filter_data, static_cast(rows), filter_all)); _native_reader->reset_filter_map_index(); - ColumnPtr native_column(std::move(column)); + const bool materialize_variant = + !dictionary_ids && _variant_plan->contains_variant && output_type->equals(*_type); + if (materialize_variant) { + _variant_physical_column->clear(); + } + ColumnPtr native_column = materialize_variant ? ColumnPtr(std::move(_variant_physical_column)) + : ColumnPtr(std::move(column)); bool eof = false; int64_t native_calls = 0; int64_t consecutive_empty_calls = 0; while (*rows_read < rows && !eof) { ++native_calls; size_t loop_rows = 0; + const DataTypePtr& decoder_type = materialize_variant ? _native_type : output_type; RETURN_IF_ERROR(_native_reader->read_column_data( - native_column, output_type, _schema_node, filter, + native_column, decoder_type, _schema_node, filter, static_cast(rows - *rows_read), &loop_rows, &eof, dictionary_ids)); if (loop_rows == 0 && !eof) { // A selected RowRanges plan may reject the current data page completely. V1 advances @@ -297,7 +409,11 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ // next page. Bound consecutive empty transitions by the Row Group row count to retain // a deterministic corruption exit if a decoder ever stops advancing. if (++consecutive_empty_calls > _row_group_rows + 1) { - column = IColumn::mutate(std::move(native_column)); + if (materialize_variant) { + _variant_physical_column = IColumn::mutate(std::move(native_column)); + } else { + column = IColumn::mutate(std::move(native_column)); + } return Status::Corruption("Native parquet reader made no progress for column {}", _name); } @@ -306,7 +422,20 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ consecutive_empty_calls = 0; *rows_read += static_cast(loop_rows); } - column = IColumn::mutate(std::move(native_column)); + if (materialize_variant) { + if (*rows_read != rows) { + _variant_physical_column = IColumn::mutate(std::move(native_column)); + return Status::Corruption("Native parquet reader returned {} rows, expected {} for {}", + *rows_read, rows, _name); + } + // The shredded state owns this decoded batch. Replace scanner scratch before handing the + // pointer off so typed path expressions can retain its physical leaves without a copy. + _variant_physical_column = _native_type->create_column(); + RETURN_IF_ERROR(materialize_variant_columns(*_variant_plan, std::move(native_column), + column, _profile)); + } else { + column = IColumn::mutate(std::move(native_column)); + } if (_profile.native_read_calls != nullptr) { COUNTER_UPDATE(_profile.native_read_calls, native_calls); } @@ -545,7 +674,7 @@ Status NativeColumnReader::skip(int64_t rows) { _filter_scratch.assign(static_cast(selected_rows), 0); int64_t rows_read = 0; RETURN_IF_ERROR(read_with_filter(selected_rows, _filter_scratch.data(), true, _skip_column, - _type, false, &rows_read)); + _native_type, false, &rows_read)); DORIS_CHECK(_skip_column->empty()); DORIS_CHECK(rows_read == selected_rows); _logical_row_position += rows_read; @@ -586,6 +715,11 @@ Status NativeColumnReader::select_with_dictionary_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(survivor_count != nullptr); DORIS_CHECK(used_filter != nullptr); + if (_variant_plan->contains_variant) { + row_filter->clear(); + *used_filter = false; + return Status::OK(); + } RETURN_IF_ERROR(validate_selected_span(batch_rows)); *used_filter = false; *survivor_count = 0; @@ -737,6 +871,13 @@ Status NativeColumnReader::select_with_fixed_width_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(used_filter != nullptr); DORIS_CHECK(execution_kind != nullptr); + if (_variant_plan->contains_variant) { + // Direct fixed-width evaluation cannot preserve a Variant physical subtree's row shape. + row_filter->clear(); + *used_filter = false; + *execution_kind = DirectPredicateExecutionKind::NONE; + return Status::OK(); + } RETURN_IF_ERROR(validate_selected_span(batch_rows)); const uint8_t* filter_data = nullptr; RETURN_IF_ERROR(selection.materialize_filter(selected_rows, batch_rows, &filter_data)); @@ -867,6 +1008,10 @@ bool NativeColumnReader::crossed_page_since_last_batch() { Result NativeColumnReader::dictionary_values() { DORIS_CHECK(_native_reader != nullptr); + if (_variant_plan->contains_variant) { + return ResultError( + Status::NotSupported("Parquet Variant columns do not expose dictionary values")); + } return _native_reader->dictionary_values(_type); } diff --git a/be/src/format_v2/parquet/reader/native_column_reader.h b/be/src/format_v2/parquet/reader/native_column_reader.h index f7dfcb3e696be8..150ab4352d00c6 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.h +++ b/be/src/format_v2/parquet/reader/native_column_reader.h @@ -43,6 +43,7 @@ struct IOContext; namespace doris::format::parquet { class NativeParquetMetadata; +struct VariantMaterializationNode; namespace detail { inline constexpr int64_t MAX_NATIVE_LAZY_SKIP_ROWS = std::numeric_limits::max(); @@ -102,7 +103,9 @@ class NativeColumnReader final : public ParquetColumnReader { Result dictionary_values() override; private: - NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr projected_type, + NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr logical_type, + DataTypePtr native_type, + std::unique_ptr variant_plan, ParquetColumnReaderProfile profile); Status init(io::FileReaderSPtr file, const NativeParquetMetadata* metadata, int row_group_id, @@ -142,6 +145,9 @@ class NativeColumnReader final : public ParquetColumnReader { const std::unordered_map* _offset_indexes = nullptr; std::shared_ptr _schema_node; std::unique_ptr _native_reader; + DataTypePtr _native_type; + std::unique_ptr _variant_plan; + MutableColumnPtr _variant_physical_column; std::unique_ptr _page_cache_runtime_state; std::vector _selected_ranges; size_t _selected_range_idx = 0; diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp new file mode 100644 index 00000000000000..4827f951c94fe5 --- /dev/null +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -0,0 +1,1112 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/parquet/reader/variant_column_reader.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common/exception.h" +#include "core/assert_cast.h" +#include "core/column/column_array.h" +#include "core/column/column_decimal.h" +#include "core/column/column_map.h" +#include "core/column/column_nullable.h" +#include "core/column/column_struct.h" +#include "core/column/column_vector.h" +#include "core/column/variant_v2/column_variant_v2.h" +#include "core/column/variant_v2/column_variant_v2_typed_column.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_variant_v2.h" +#include "core/value/variant/variant_batch_builder.h" +#include "core/value/variant/variant_metadata.h" +#include "format_v2/parquet/parquet_column_schema.h" + +namespace doris::format::parquet { +namespace { + +struct Cell { + const IColumn* column = nullptr; + bool is_null = false; +}; + +Cell cell_at(const IColumn& column, size_t row) { + if (row >= column.size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant row {} exceeds column size {}", row, + column.size()); + } + if (const auto* nullable = check_and_get_column(column)) { + return {.column = &nullable->get_nested_column(), + .is_null = nullable->get_null_map_data()[row] != 0}; + } + return {.column = &column, .is_null = false}; +} + +const ParquetColumnSchema* find_child(const ParquetColumnSchema& schema, std::string_view name, + size_t* index) { + for (size_t i = 0; i < schema.children.size(); ++i) { + if (schema.children[i]->name == name) { + if (index != nullptr) { + *index = i; + } + return schema.children[i].get(); + } + } + return nullptr; +} + +Cell struct_child_at(const ParquetColumnSchema& schema, const IColumn& physical, size_t row, + std::string_view name, const ParquetColumnSchema** child_schema) { + const auto& structure = assert_cast(physical); + size_t index = 0; + const auto* child = find_child(schema, name, &index); + if (child == nullptr || index >= structure.tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} has no physical child {}", + schema.name, name); + } + if (child_schema != nullptr) { + *child_schema = child; + } + return cell_at(structure.get_column(index), row); +} + +uint8_t decimal_width(int precision) { + if (precision <= 0 || precision > 38) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant decimal precision {} is outside [1, 38]", precision); + } + return precision <= 9 ? 4 : (precision <= 18 ? 8 : 16); +} + +uint8_t integer_width(const ParquetColumnSchema& schema, PrimitiveType type) { + if (schema.type_descriptor.is_unsigned_integer) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Unsigned integers are not valid Parquet Variant typed values"); + } + if (schema.type_descriptor.integer_bit_width > 0) { + switch (schema.type_descriptor.integer_bit_width) { + case 8: + return 1; + case 16: + return 2; + case 32: + return 4; + case 64: + return 8; + default: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant integer width {}", + schema.type_descriptor.integer_bit_width); + } + } + switch (type) { + case TYPE_TINYINT: + return 1; + case TYPE_SMALLINT: + return 2; + case TYPE_INT: + return 4; + case TYPE_BIGINT: + return 8; + default: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant integer type {}", type); + } +} + +void append_typed_scalar(const ParquetColumnSchema& schema, const IColumn& column, size_t row, + VariantBatchBuilder::Row& builder) { + const PrimitiveType type = remove_nullable(schema.type)->get_primitive_type(); + switch (type) { + case TYPE_BOOLEAN: + builder.add_bool(assert_cast(column).get_data()[row] != 0); + return; + case TYPE_TINYINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_SMALLINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_INT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_BIGINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_FLOAT: + builder.add_float(assert_cast(column).get_data()[row]); + return; + case TYPE_DOUBLE: + builder.add_double(assert_cast(column).get_data()[row]); + return; + case TYPE_DECIMAL128I: { + const auto value = assert_cast(column).get_data()[row].value; + builder.add_decimal(value, static_cast(schema.type_descriptor.decimal_scale), + decimal_width(schema.type_descriptor.decimal_precision)); + return; + } + case TYPE_TIMEV2: { + const double seconds = assert_cast(column).get_data()[row]; + if (!std::isfinite(seconds) || + std::abs(seconds) > static_cast(std::numeric_limits::max()) / 1e6) { + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant TIME value"); + } + builder.add_time_ntz_micros(static_cast(std::llround(seconds * 1e6))); + return; + } + case TYPE_DATETIMEV2: { + if (schema.type_descriptor.time_unit == ParquetTimeUnit::NANOS) { + // Native DATETIMEV2 is microsecond based. Reject before returning a silently truncated + // value; a raw INT64 nanos decoder can be added independently. + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + const auto& value = assert_cast(column).get_data()[row]; + builder.add_timestamp_micros( + variant_timestamp_micros(value, row, "Parquet Variant TIMESTAMP"), + schema.type_descriptor.timestamp_is_adjusted_to_utc); + return; + } + case TYPE_TIMESTAMPTZ: { + if (schema.type_descriptor.time_unit == ParquetTimeUnit::NANOS) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + const auto& value = assert_cast(column).get_data()[row]; + builder.add_timestamp_micros( + variant_timestamp_micros(value, row, "Parquet Variant TIMESTAMP"), true); + return; + } + case TYPE_VARBINARY: { + const StringRef value = column.get_data_at(row); + if (!schema.type_descriptor.is_uuid) { + builder.add_binary(value); + return; + } + if (value.size != 16) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant UUID has {} bytes instead of 16", value.size); + } + std::array uuid {}; + std::memcpy(uuid.data(), value.data, uuid.size()); + builder.add_uuid(uuid); + return; + } + case TYPE_STRING: { + const StringRef value = column.get_data_at(row); + if (schema.type_descriptor.is_uuid) { + if (value.size != 16) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant UUID has {} bytes instead of 16", value.size); + } + std::array uuid {}; + std::memcpy(uuid.data(), value.data, uuid.size()); + builder.add_uuid(uuid); + } else if (schema.type_descriptor.is_string_annotation) { + builder.add_string(value); + } else { + builder.add_binary(value); + } + return; + } + default: + if (!is_supported_variant_typed_identity(type)) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant typed value {} is not supported", + remove_nullable(schema.type)->get_name()); + } + dispatch_variant_typed_column( + column, type, [&](const auto& typed_column) { + with_variant_typed_scalar( + typed_column, row, + static_cast(remove_nullable(schema.type)->get_scale()), + [&](const VariantScalarRef& scalar) { builder.add_scalar(scalar); }); + }); + } +} + +enum class WrapperContext { ROOT, ARRAY_ELEMENT, OBJECT_FIELD }; + +bool append_wrapper(const ParquetColumnSchema& schema, const IColumn& wrapper, size_t row, + VariantMetadataRef metadata, VariantBatchBuilder::Row& builder, + WrapperContext context); + +void append_typed_value(const ParquetColumnSchema& schema, const IColumn& column, size_t row, + VariantMetadataRef metadata, const VariantRef* residual, + VariantBatchBuilder::Row& builder) { + switch (schema.kind) { + case ParquetColumnSchemaKind::PRIMITIVE: + if (static_cast(residual)) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant scalar typed_value cannot have residual value bytes"); + } + append_typed_scalar(schema, column, row, builder); + return; + case ParquetColumnSchemaKind::STRUCT: { + if (static_cast(residual) && residual->basic_type() != VariantBasicType::OBJECT) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant object typed_value has non-object residual value"); + } + const auto& structure = assert_cast(column); + if (structure.tuple_size() != schema.children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant object {} physical field count mismatch", schema.name); + } + auto object = builder.start_object(); + if (static_cast(residual)) { + for (uint32_t i = 0; i < residual->num_elements(); ++i) { + uint32_t field_id = 0; + const VariantRef child = residual->object_value_at(i, &field_id); + object.add_key(residual->metadata.key_at(field_id)); + builder.add_value(child); + } + } + for (size_t i = 0; i < schema.children.size(); ++i) { + const auto& child_schema = *schema.children[i]; + const Cell child = cell_at(structure.get_column(i), row); + if (child.is_null) { + // Shredded object fields are optional wrapper groups. A missing group means the + // key is absent, which differs from a present wrapper encoding a Variant null. + continue; + } + // A null/null wrapper means this object field is absent. Delay add_key until its + // presence is known so absent shredded fields do not turn into Variant nulls. + size_t value_index = 0; + const auto* value_schema = find_child(child_schema, "value", &value_index); + const auto& child_struct = assert_cast(*child.column); + const bool value_present = value_schema != nullptr && + !cell_at(child_struct.get_column(value_index), row).is_null; + size_t typed_index = 0; + const auto* typed_schema = find_child(child_schema, "typed_value", &typed_index); + const bool typed_present = typed_schema != nullptr && + !cell_at(child_struct.get_column(typed_index), row).is_null; + if (!value_present && !typed_present) { + continue; + } + object.add_key(StringRef(child_schema.name)); + (void)append_wrapper(child_schema, *child.column, row, metadata, builder, + WrapperContext::OBJECT_FIELD); + } + object.finish(); + return; + } + case ParquetColumnSchemaKind::LIST: { + if (static_cast(residual)) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant array typed_value cannot have residual value bytes"); + } + if (schema.children.size() != 1) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant array {} has invalid element schema", schema.name); + } + const auto& array = assert_cast(column); + const size_t begin = array.offset_at(static_cast(row)); + const size_t end = array.get_offsets()[row]; + auto scope = builder.start_array(); + for (size_t element = begin; element < end; ++element) { + const Cell cell = cell_at(array.get_data(), element); + if (cell.is_null) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant shredded array element wrapper is null"); + } + (void)append_wrapper(*schema.children[0], *cell.column, element, metadata, builder, + WrapperContext::ARRAY_ELEMENT); + } + scope.finish(); + return; + } + case ParquetColumnSchemaKind::MAP: + case ParquetColumnSchemaKind::VARIANT: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant typed_value schema {}", + schema.name); + } +} + +bool append_wrapper(const ParquetColumnSchema& schema, const IColumn& wrapper, size_t row, + VariantMetadataRef metadata, VariantBatchBuilder::Row& builder, + WrapperContext context) { + Cell value; + if (find_child(schema, "value", nullptr) != nullptr) { + value = struct_child_at(schema, wrapper, row, "value", nullptr); + } else { + value.is_null = true; + } + const ParquetColumnSchema* typed_schema = nullptr; + Cell typed; + if (find_child(schema, "typed_value", nullptr) != nullptr) { + typed = struct_child_at(schema, wrapper, row, "typed_value", &typed_schema); + } else { + typed.is_null = true; + } + + if (find_child(schema, "value", nullptr) == nullptr && typed_schema == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant wrapper {} has neither value nor typed_value", + schema.name); + } + if (value.is_null && typed.is_null) { + if (context == WrapperContext::OBJECT_FIELD) { + return false; + } + if (context == WrapperContext::ARRAY_ELEMENT) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant array element is missing"); + } + builder.add_null(); + return true; + } + + VariantRef residual {.metadata = metadata, .value = {}}; + if (!value.is_null) { + residual.value = value.column->get_data_at(row); + } + if (typed.is_null) { + builder.add_value(residual); + return true; + } + append_typed_value(*typed_schema, *typed.column, row, metadata, + value.is_null ? nullptr : &residual, builder); + return true; +} + +void encode_variant_range(const ParquetColumnSchema& schema, const IColumn& wrapper, + const ColumnNullable* outer_nullable, size_t begin, size_t end, + ColumnVariantV2& variants) { + try { + VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = end - begin}); + for (size_t row = begin; row < end; ++row) { + auto output_row = builder.begin_row(); + if (outer_nullable != nullptr && outer_nullable->get_null_map_data()[row] != 0) { + output_row.add_null(); + output_row.finish(); + continue; + } + const Cell metadata_cell = struct_child_at(schema, wrapper, row, "metadata", nullptr); + if (metadata_cell.is_null) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant {} has null metadata at row {}", schema.name, row); + } + const StringRef metadata_bytes = metadata_cell.column->get_data_at(row); + const VariantMetadataRef metadata {metadata_bytes.data, metadata_bytes.size}; + metadata.validate(); + (void)append_wrapper(schema, wrapper, row, metadata, output_row, WrapperContext::ROOT); + output_row.finish(); + } + VariantBatchBuilder batch = builder.finish_batch(); + variants.insert_encoded_batch(batch); + } catch (...) { + if (end - begin <= 1) { + throw; + } + // A single builder has one metadata dictionary. If heterogeneous file rows cannot fit in + // that dictionary, split without changing the destination column's already-valid batches. + // Corrupt input still reaches a one-row range and propagates its original exception. + const size_t middle = begin + (end - begin) / 2; + encode_variant_range(schema, wrapper, outer_nullable, begin, middle, variants); + encode_variant_range(schema, wrapper, outer_nullable, middle, end, variants); + } +} + +ColumnVariantV2::MutablePtr encode_variant_column(const ParquetColumnSchema& schema, + const IColumn& physical) { + if (schema.kind != ParquetColumnSchemaKind::VARIANT) { + throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant", + schema.name); + } + const auto* outer_nullable = check_and_get_column(physical); + const IColumn& wrapper = + outer_nullable == nullptr ? physical : outer_nullable->get_nested_column(); + const auto& structure = assert_cast(wrapper); + if (structure.tuple_size() != schema.children.size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} physical field count mismatch", + schema.name); + } + + auto variants = ColumnVariantV2::create(); + constexpr size_t MAX_RECONSTRUCTION_BATCH_ROWS = 4096; + for (size_t begin = 0; begin < physical.size(); begin += MAX_RECONSTRUCTION_BATCH_ROWS) { + encode_variant_range(schema, wrapper, outer_nullable, begin, + std::min(physical.size(), begin + MAX_RECONSTRUCTION_BATCH_ROWS), + *variants); + } + return variants; +} + +std::unique_ptr clone_schema( + const ParquetColumnSchema& source, const format::LocalColumnIndex* projection = nullptr) { + auto result = std::make_unique(); + result->local_id = source.local_id; + result->parquet_field_id = source.parquet_field_id; + result->name = source.name; + result->type = source.type; + result->variant_physical_type = source.variant_physical_type; + result->leaf_column_id = source.leaf_column_id; + result->type_descriptor = source.type_descriptor; + result->kind = source.kind; + result->max_definition_level = source.max_definition_level; + result->max_repetition_level = source.max_repetition_level; + result->nullable_definition_level = source.nullable_definition_level; + result->definition_level = source.definition_level; + result->repetition_level = source.repetition_level; + result->repeated_ancestor_definition_level = source.repeated_ancestor_definition_level; + result->repeated_repetition_level = source.repeated_repetition_level; + const bool partial = format::is_partial_projection(projection); + result->children.reserve(partial ? projection->children.size() : source.children.size()); + if (partial) { + // NativeColumnReader emits a partial STRUCT in projection order, so the retained schema + // must use that same order or field names will address the wrong physical tuple element. + for (const auto& child_projection : projection->children) { + const auto child = std::ranges::find_if(source.children, [&](const auto& candidate) { + return candidate->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child != source.children.end()); + result->children.push_back(clone_schema(**child, &child_projection)); + } + } else { + for (const auto& child : source.children) { + result->children.push_back(clone_schema(*child)); + } + } + return result; +} + +ColumnPtr unwrap_nullable(ColumnPtr column) { + if (const auto* nullable = check_and_get_column(*column)) { + return nullable->get_nested_column_ptr(); + } + return column; +} + +ColumnPtr struct_child(const ParquetColumnSchema& schema, ColumnPtr column, std::string_view name, + const ParquetColumnSchema** child_schema) { + column = unwrap_nullable(std::move(column)); + const auto* structure = check_and_get_column(*column); + if (structure == nullptr) { + return nullptr; + } + size_t index = 0; + const auto* child = find_child(schema, name, &index); + if (child == nullptr || index >= structure->tuple_size()) { + return nullptr; + } + if (child_schema != nullptr) { + *child_schema = child; + } + return structure->get_column_ptr(index); +} + +bool has_present_value(const ColumnPtr& column) { + if (const auto* nullable = check_and_get_column(*column)) { + return std::ranges::any_of(nullable->get_null_map_data(), + [](uint8_t is_null) { return is_null == 0; }); + } + return !column->empty(); +} + +bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { + if (schema.type == nullptr || schema.kind != ParquetColumnSchemaKind::PRIMITIVE) { + return false; + } + // ColumnVariantV2 typed state carries only a Doris type. Binary/UUID annotations, temporal + // units, and other Parquet-only identity must therefore reconstruct canonical Variant bytes. + switch (remove_nullable(schema.type)->get_primitive_type()) { + case TYPE_BOOLEAN: + case TYPE_TINYINT: + case TYPE_SMALLINT: + case TYPE_INT: + case TYPE_BIGINT: + case TYPE_FLOAT: + case TYPE_DOUBLE: + case TYPE_DECIMAL128I: + case TYPE_DATEV2: + return true; + default: + return false; + } +} + +bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { + return (!left && !right) || (left && right && left->equals(*right)); +} + +bool same_type_descriptor(const ParquetTypeDescriptor& left, const ParquetTypeDescriptor& right) { + return same_data_type(left.doris_type, right.doris_type) && + same_data_type(left.physical_doris_type, right.physical_doris_type) && + left.extra_type_info == right.extra_type_info && left.time_unit == right.time_unit && + left.physical_type == right.physical_type && + left.integer_bit_width == right.integer_bit_width && + left.decimal_precision == right.decimal_precision && + left.decimal_scale == right.decimal_scale && left.fixed_length == right.fixed_length && + left.is_unsigned_integer == right.is_unsigned_integer && + left.is_decimal == right.is_decimal && left.is_timestamp == right.is_timestamp && + left.timestamp_is_adjusted_to_utc == right.timestamp_is_adjusted_to_utc && + left.is_string_like == right.is_string_like && + left.is_string_annotation == right.is_string_annotation && + left.is_uuid == right.is_uuid && left.unsupported_reason == right.unsupported_reason; +} + +bool same_shredded_schema(const ParquetColumnSchema& left, const ParquetColumnSchema& right) { + if (left.name != right.name || left.kind != right.kind || + !same_data_type(left.type, right.type) || + !same_type_descriptor(left.type_descriptor, right.type_descriptor) || + left.children.size() != right.children.size()) { + return false; + } + for (size_t i = 0; i < left.children.size(); ++i) { + if (!same_shredded_schema(*left.children[i], *right.children[i])) { + return false; + } + } + return true; +} + +void append_compatible_column(IColumn& output, const IColumn& converted); +void validate_compatible_column(const IColumn& output, const IColumn& converted); + +class ParquetVariantShreddedState final : public VariantShreddedState { +public: + ParquetVariantShreddedState(std::shared_ptr schema, + ColumnPtr physical, bool complete, + ParquetColumnReaderProfile profile = {}) + : _schema(std::move(schema)), + _physical(std::move(physical)), + _complete(complete), + _profile(profile) { + DORIS_CHECK(_schema != nullptr && static_cast(_physical)); + const ColumnPtr wrapper = unwrap_nullable(_physical); + const auto* structure = check_and_get_column(*wrapper); + if (structure == nullptr || structure->tuple_size() != _schema->children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant {} physical field count mismatch", _schema->name); + } + } + + size_t size() const override { return _physical->size(); } + size_t byte_size() const override { + std::lock_guard lock(_materialization_lock); + return _physical->byte_size() + (_materialized ? _materialized->byte_size() : 0); + } + size_t allocated_bytes() const override { + std::lock_guard lock(_materialization_lock); + return _physical->allocated_bytes() + + (_materialized ? _materialized->allocated_bytes() : 0); + } + void sanity_check() const override { _physical->sanity_check(); } + + void for_each_subcolumn(IColumn::ColumnCallback callback) const override { + callback(*_physical); + } + + std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t result_size_hint) const override { + // Compact the decoded physical tree directly. In particular, a leaf-only projection has + // no metadata/value columns from which a canonical Variant could be reconstructed. + // The projection schema is immutable and reader-scoped, so derived selections share it + // instead of cloning the whole shredded tree for every filter operation. + return std::make_shared( + _schema, _physical->filter(filter, result_size_hint), _complete, _profile); + } + + std::shared_ptr select_range(size_t start, size_t length) const override { + return std::make_shared(_schema, _physical->cut(start, length), + _complete, _profile); + } + + std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const override { + MutableColumnPtr selected = _physical->clone_empty(); + selected->insert_indices_from(*_physical, indices_begin, indices_end); + return std::make_shared(_schema, std::move(selected), + _complete, _profile); + } + + bool try_append(const VariantShreddedState& source) override { + const auto* parquet_source = dynamic_cast(&source); + if (parquet_source == nullptr || _complete != parquet_source->_complete || + !same_shredded_schema(*_schema, *parquet_source->_schema)) { + return false; + } + validate_compatible_column(*_physical, *parquet_source->_physical); + auto mutable_physical = IColumn::mutate(std::move(_physical)); + append_compatible_column(*mutable_physical, *parquet_source->_physical); + _physical = std::move(mutable_physical); + std::lock_guard lock(_materialization_lock); + _materialized.reset(); + return true; + } + + std::optional find_typed_value( + std::span path) const override { + auto path_miss = [&]() -> std::optional { + update_counter(_profile.variant_direct_leaf_path_misses, 1); + return std::nullopt; + }; + if (path.empty()) { + return path_miss(); + } + + const ParquetColumnSchema* typed_schema = nullptr; + ColumnPtr typed = struct_child(*_schema, _physical, "typed_value", &typed_schema); + if (!typed || typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { + return path_miss(); + } + + for (size_t position = 0; position < path.size(); ++position) { + if (path[position].kind != VariantShreddedPathSegment::Kind::OBJECT_KEY) { + return path_miss(); + } + + const std::string_view key(path[position].key.data, path[position].key.size); + const ParquetColumnSchema* wrapper_schema = nullptr; + ColumnPtr wrapper = struct_child(*typed_schema, typed, key, &wrapper_schema); + if (!wrapper) { + return path_miss(); + } + + if (ColumnPtr residual = struct_child(*wrapper_schema, wrapper, "value", nullptr); + static_cast(residual) && has_present_value(residual)) { + // A residual value can contribute data to the same logical object. Reconstructing + // is required in that case; returning only the typed leaf would drop information. + update_counter(_profile.variant_direct_leaf_residual_fallbacks, 1); + return std::nullopt; + } + + typed = struct_child(*wrapper_schema, wrapper, "typed_value", &typed_schema); + if (!typed) { + return path_miss(); + } + if (position + 1 == path.size()) { + if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || + check_and_get_column(*typed) == nullptr || + !supports_direct_typed_variant_state(*typed_schema)) { + update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); + return std::nullopt; + } + update_counter(_profile.variant_direct_leaf_rows, + static_cast(typed->size())); + return VariantShreddedTypedValue {.column = std::move(typed), + .type = remove_nullable(typed_schema->type)}; + } + if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { + return path_miss(); + } + } + return std::nullopt; + } + + const ColumnVariantV2& materialized_column() const override { + std::lock_guard lock(_materialization_lock); + if (!_complete) { + throw Exception( + ErrorCode::INTERNAL_ERROR, + "A projected Parquet Variant can only serve its validated shredded leaves"); + } + if (!_materialized) { + SCOPED_TIMER(_profile.variant_reconstruction_time); + _materialized = encode_variant_column(*_schema, *_physical); + update_counter(_profile.variant_reconstructed_rows, + static_cast(_physical->size())); + } + return *_materialized; + } + +private: + static void update_counter(RuntimeProfile::Counter* counter, int64_t value) { + if (counter != nullptr) { + COUNTER_UPDATE(counter, value); + } + } + + std::shared_ptr _schema; + ColumnPtr _physical; + bool _complete = true; + ParquetColumnReaderProfile _profile; + mutable std::mutex _materialization_lock; + mutable ColumnVariantV2::MutablePtr _materialized; +}; + +MutableColumnPtr build_variant_column(std::shared_ptr schema, + ColumnPtr physical, bool complete, + const ParquetColumnReaderProfile& profile) { + DORIS_CHECK(schema != nullptr); + if (schema->kind != ParquetColumnSchemaKind::VARIANT) { + throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant", + schema->name); + } + + const auto* outer_nullable = check_and_get_column(*physical); + MutableColumnPtr variants = + ColumnVariantV2::create_shredded(std::make_shared( + std::move(schema), physical, complete, profile)); + if (outer_nullable == nullptr) { + return variants; + } + auto nulls = outer_nullable->get_null_map_column().clone_resized(physical->size()); + return ColumnNullable::create(std::move(variants), std::move(nulls)); +} + +ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile); + +ColumnPtr transform_non_nullable(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile) { + const auto& schema = *plan.schema; + switch (schema.kind) { + case ParquetColumnSchemaKind::PRIMITIVE: + return physical; + case ParquetColumnSchemaKind::VARIANT: + return build_variant_column( + plan.variant_state_schema + ? plan.variant_state_schema + : create_variant_state_schema(schema, plan.variant_projection + ? &*plan.variant_projection + : nullptr), + std::move(physical), + !format::is_partial_projection(plan.variant_projection ? &*plan.variant_projection + : nullptr), + profile); + case ParquetColumnSchemaKind::STRUCT: { + const auto& structure = assert_cast(*physical); + if (structure.tuple_size() != plan.children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Projected Parquet STRUCT {} field count mismatch", schema.name); + } + Columns fields; + fields.reserve(plan.children.size()); + for (size_t i = 0; i < plan.children.size(); ++i) { + fields.push_back( + transform_node(*plan.children[i], structure.get_column_ptr(i), profile)); + } + return ColumnStruct::create(std::move(fields)); + } + case ParquetColumnSchemaKind::LIST: { + const auto& array = assert_cast(*physical); + if (plan.children.size() != 1) { + throw Exception(ErrorCode::CORRUPTION, "Projected Parquet ARRAY plan is invalid"); + } + auto values = transform_node(*plan.children[0], array.get_data_ptr(), profile); + return ColumnArray::create(std::move(values), array.get_offsets_ptr()); + } + case ParquetColumnSchemaKind::MAP: { + const auto& map = assert_cast(*physical); + if (plan.children.size() != 2) { + throw Exception(ErrorCode::CORRUPTION, "Projected Parquet MAP plan is invalid"); + } + auto keys = transform_node(*plan.children[0], map.get_keys_ptr(), profile); + auto values = transform_node(*plan.children[1], map.get_values_ptr(), profile); + return ColumnMap::create(std::move(keys), std::move(values), map.get_offsets_ptr()); + } + } + throw Exception(ErrorCode::INTERNAL_ERROR, "Unknown Parquet schema kind"); +} + +ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile) { + if (plan.schema == nullptr) { + throw Exception(ErrorCode::INTERNAL_ERROR, "Parquet Variant materialization plan is null"); + } + if (plan.schema->kind == ParquetColumnSchemaKind::VARIANT) { + return build_variant_column( + plan.variant_state_schema + ? plan.variant_state_schema + : create_variant_state_schema( + *plan.schema, + plan.variant_projection ? &*plan.variant_projection : nullptr), + std::move(physical), + !format::is_partial_projection(plan.variant_projection ? &*plan.variant_projection + : nullptr), + profile); + } + if (const auto* nullable = check_and_get_column(*physical)) { + auto nested = transform_non_nullable(plan, nullable->get_nested_column_ptr(), profile); + return ColumnNullable::create(std::move(nested), nullable->get_null_map_column_ptr()); + } + return transform_non_nullable(plan, std::move(physical), profile); +} + +void append_compatible_column(IColumn& output, const IColumn& converted) { + if (auto* output_nullable = check_and_get_column(output)) { + if (const auto* converted_nullable = check_and_get_column(converted)) { + append_compatible_column(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + output_nullable->get_null_map_column().insert_range_from( + converted_nullable->get_null_map_column(), 0, converted.size()); + } else { + append_compatible_column(output_nullable->get_nested_column(), converted); + // External slots and nested Iceberg fields may remain nullable even when one file's + // physical node is required. Preserve that destination invariant with non-null bits. + output_nullable->push_false_to_nullmap(converted.size()); + } + return; + } + + if (const auto* converted_nullable = check_and_get_column(converted)) { + // Parquet writers may encode an Iceberg required field as optional. It can populate a + // non-nullable destination only when this batch proves that every value is present. + if (converted_nullable->has_null()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced null data for a " + "non-nullable destination"); + } + append_compatible_column(output, converted_nullable->get_nested_column()); + return; + } + + if (auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible STRUCT"); + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + append_compatible_column(output_struct->get_column(i), converted_struct->get_column(i)); + } + return; + } + + if (auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + if (converted_array == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible ARRAY"); + } + const size_t element_base = output_array->get_data().size(); + append_compatible_column(output_array->get_data(), converted_array->get_data()); + auto& output_offsets = output_array->get_offsets(); + output_offsets.reserve(output_offsets.size() + converted_array->size()); + for (const auto offset : converted_array->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + return; + } + + if (auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + if (converted_map == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible MAP"); + } + const size_t element_base = output_map->get_keys().size(); + append_compatible_column(output_map->get_keys(), converted_map->get_keys()); + append_compatible_column(output_map->get_values(), converted_map->get_values()); + auto& output_offsets = output_map->get_offsets(); + output_offsets.reserve(output_offsets.size() + converted_map->size()); + for (const auto offset : converted_map->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + return; + } + + if (auto* output_variant = check_and_get_column(output)) { + const auto* converted_variant = check_and_get_column(converted); + if (converted_variant == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible column"); + } + output_variant->insert_range_from(*converted_variant, 0, converted_variant->size()); + return; + } + + if (output.get_name() != converted.get_name()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced {} for {} destination", + converted.get_name(), output.get_name()); + } + output.insert_range_from(converted, 0, converted.size()); +} + +void validate_compatible_column(const IColumn& output, const IColumn& converted) { + if (const auto* output_nullable = check_and_get_column(output)) { + if (const auto* converted_nullable = check_and_get_column(converted)) { + validate_compatible_column(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + } else { + validate_compatible_column(output_nullable->get_nested_column(), converted); + } + return; + } + if (const auto* converted_nullable = check_and_get_column(converted)) { + if (converted_nullable->has_null()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced null data for a " + "non-nullable destination"); + } + validate_compatible_column(output, converted_nullable->get_nested_column()); + return; + } + if (const auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible STRUCT"); + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + validate_compatible_column(output_struct->get_column(i), + converted_struct->get_column(i)); + } + return; + } + if (const auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + if (converted_array == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible ARRAY"); + } + validate_compatible_column(output_array->get_data(), converted_array->get_data()); + return; + } + if (const auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + if (converted_map == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible MAP"); + } + validate_compatible_column(output_map->get_keys(), converted_map->get_keys()); + validate_compatible_column(output_map->get_values(), converted_map->get_values()); + return; + } + if (check_and_get_column(output) != nullptr) { + if (check_and_get_column(converted) == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible column"); + } + return; + } + if (output.get_name() != converted.get_name()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced {} for {} destination", + converted.get_name(), output.get_name()); + } +} + +bool has_exact_column_shape(const IColumn& output, const IColumn& converted) { + const auto* output_nullable = check_and_get_column(output); + const auto* converted_nullable = check_and_get_column(converted); + if (output_nullable != nullptr || converted_nullable != nullptr) { + return output_nullable != nullptr && converted_nullable != nullptr && + has_exact_column_shape(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + } + if (const auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + return false; + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + if (!has_exact_column_shape(output_struct->get_column(i), + converted_struct->get_column(i))) { + return false; + } + } + return true; + } + if (const auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + return converted_array != nullptr && + has_exact_column_shape(output_array->get_data(), converted_array->get_data()); + } + if (const auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + return converted_map != nullptr && + has_exact_column_shape(output_map->get_keys(), converted_map->get_keys()) && + has_exact_column_shape(output_map->get_values(), converted_map->get_values()); + } + if (check_and_get_column(output) != nullptr) { + return check_and_get_column(converted) != nullptr; + } + return output.get_name() == converted.get_name(); +} + +void append_materialized_column(MutableColumnPtr& output, ColumnPtr converted) { + // Validate the complete destination shape before mutation. This preserves atomic failures + // without copying a full scratch batch, while an empty exact-shape output can adopt the tree. + validate_compatible_column(*output, *converted); + if (output->empty() && has_exact_column_shape(*output, *converted)) { + if (converted->is_exclusive()) { + // The transformed tree consumed the decoder tree and is recursively exclusive. Keep + // primitive siblings and their buffers intact instead of recursively COW-cloning them. + output = converted->assert_mutable(); + return; + } + output = IColumn::mutate(std::move(converted)); + return; + } + append_compatible_column(*output, *converted); +} + +} // namespace + +std::shared_ptr create_variant_state_schema( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection) { + return std::shared_ptr(clone_schema(schema, projection)); +} + +Status materialize_variant_rows(const ParquetColumnSchema& schema, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + return materialize_variant_rows(schema, physical.get_ptr(), output, profile); +} + +Status materialize_variant_rows(const ParquetColumnSchema& schema, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + if (!output) { + return Status::InvalidArgument("Parquet Variant output column is null"); + } + RETURN_IF_CATCH_EXCEPTION({ + auto converted = build_variant_column(create_variant_state_schema(schema), + std::move(physical), true, profile); + append_materialized_column(output, std::move(converted)); + }); + return Status::OK(); +} + +Status materialize_variant_columns(const VariantMaterializationNode& plan, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + return materialize_variant_columns(plan, physical.get_ptr(), output, profile); +} + +Status materialize_variant_columns(const VariantMaterializationNode& plan, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + if (!output) { + return Status::InvalidArgument("Parquet Variant output column is null"); + } + RETURN_IF_CATCH_EXCEPTION({ + auto converted = transform_node(plan, std::move(physical), profile); + append_materialized_column(output, std::move(converted)); + }); + return Status::OK(); +} + +} // namespace doris::format::parquet diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.h b/be/src/format_v2/parquet/reader/variant_column_reader.h new file mode 100644 index 00000000000000..53cc53983181c8 --- /dev/null +++ b/be/src/format_v2/parquet/reader/variant_column_reader.h @@ -0,0 +1,67 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include + +#include "common/status.h" +#include "core/column/column.h" +#include "format_v2/column_data.h" +#include "format_v2/parquet/parquet_profile.h" + +namespace doris::format::parquet { + +struct ParquetColumnSchema; + +// Projection-aligned view of the file schema. Complex nodes contain only the children decoded by +// NativeColumnReader. A VARIANT node may own a validated fully-shredded physical leaf projection. +struct VariantMaterializationNode { + const ParquetColumnSchema* schema = nullptr; + std::vector> children; + bool contains_variant = false; + std::optional variant_projection; + std::shared_ptr variant_state_schema; +}; + +// Builds the immutable schema retained by a shredded state in the exact order of the decoded +// physical projection. +std::shared_ptr create_variant_state_schema( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection = nullptr); + +// Converts one physical Parquet Variant wrapper column to ColumnVariantV2 and appends it to output. +// SQL NULL is represented by the wrapper's outer null map; a present wrapper with neither value nor +// typed_value is the Variant null value. +Status materialize_variant_rows(const ParquetColumnSchema& schema, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); +Status materialize_variant_rows(const ParquetColumnSchema& schema, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); + +// Recursively replaces projected VARIANT nodes inside STRUCT/LIST/MAP columns while preserving the +// surrounding column shape, offsets, and null maps. The destination is unchanged on decode errors. +Status materialize_variant_columns(const VariantMaterializationNode& plan, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); +Status materialize_variant_columns(const VariantMaterializationNode& plan, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); + +} // namespace doris::format::parquet diff --git a/be/src/format_v2/schema_projection.cpp b/be/src/format_v2/schema_projection.cpp index 342f4c91898c92..d7b2a359131ab7 100644 --- a/be/src/format_v2/schema_projection.cpp +++ b/be/src/format_v2/schema_projection.cpp @@ -88,6 +88,11 @@ Status rebuild_semantic_projected_type(const DataTypePtr& original_type, nested_projected_type = std::make_shared(key_type, value_type); break; } + case TYPE_VARIANT: + // Variant children describe a format-specific physical shredding carrier, not the public + // logical type. Pruning those children must keep the file block exposed as Variant. + *projected_type = original_type; + return Status::OK(); default: return Status::InvalidArgument("Cannot project children from non-complex type {}", original_type->get_name()); diff --git a/be/src/format_v2/table/iceberg_reader.cpp b/be/src/format_v2/table/iceberg_reader.cpp index 097e6bab111091..1fc36a57c1ab69 100644 --- a/be/src/format_v2/table/iceberg_reader.cpp +++ b/be/src/format_v2/table/iceberg_reader.cpp @@ -63,6 +63,94 @@ namespace doris::format::iceberg { static constexpr const char* ROW_LINEAGE_ROW_ID = "_row_id"; static constexpr int32_t ROW_LINEAGE_ROW_ID_FIELD_ID = 2147483540; +namespace { + +bool contains_variant_type(const DataTypePtr& input) { + if (input == nullptr) { + return false; + } + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + +bool mapping_reads_variant(const format::ColumnMapping& mapping) { + if (!mapping.file_local_id.has_value()) { + return false; + } + if (contains_variant_type(mapping.original_file_type)) { + return true; + } + if (mapping.table_type != nullptr && + remove_nullable(mapping.table_type)->get_primitive_type() == TYPE_VARIANT) { + return true; + } + return std::ranges::any_of(mapping.child_mappings, mapping_reads_variant); +} + +const char* file_format_name(FileFormat format) { + switch (format) { + case FileFormat::PARQUET: + return "PARQUET"; + case FileFormat::ORC: + return "ORC"; + case FileFormat::CSV: + return "CSV"; + case FileFormat::JSON: + return "JSON"; + case FileFormat::TEXT: + return "TEXT"; + case FileFormat::JNI: + return "JNI"; + case FileFormat::NATIVE: + return "NATIVE"; + case FileFormat::ARROW: + return "ARROW"; + case FileFormat::WAL: + return "WAL"; + } + return "UNKNOWN"; +} + +} // namespace + +Status IcebergTableReader::validate_variant_file_mappings( + FileFormat format, const std::vector& mappings) { + if (format == FileFormat::PARQUET || !std::ranges::any_of(mappings, mapping_reads_variant)) { + return Status::OK(); + } + // Gate on a physical mapping, not the table schema: an older ORC/Avro file may legitimately + // omit a Variant field added by schema evolution, in which case the mapper synthesizes NULL. + return Status::NotSupported( + "Iceberg Variant is supported only for Parquet files in FileScannerV2; file format {} " + "(including ORC/Avro readers) is not supported", + file_format_name(format)); +} + +Status IcebergTableReader::validate_file_mapping(const format::TableColumnMapper& mapper) const { + if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && + _push_down_count_columns->empty()) { + // COUNT(*) may retain an arbitrary minimum-width slot, but that carrier is never a + // semantic physical read and must not trigger the Variant file-format capability gate. + return Status::OK(); + } + return validate_variant_file_mappings(_format, mapper.mappings()); +} + template static std::string join_values_for_debug(const std::vector& values) { std::ostringstream out; diff --git a/be/src/format_v2/table/iceberg_reader.h b/be/src/format_v2/table/iceberg_reader.h index 2768e4cd3e8c73..4118655dca46f0 100644 --- a/be/src/format_v2/table/iceberg_reader.h +++ b/be/src/format_v2/table/iceberg_reader.h @@ -49,6 +49,8 @@ namespace doris::format::iceberg { class IcebergTableReader : public format::TableReader { public: ~IcebergTableReader() override = default; + static Status validate_variant_file_mappings( + FileFormat format, const std::vector& mappings); Status init(format::TableReadOptions&& options) override { RETURN_IF_ERROR(format::TableReader::init(std::move(options))); _mapper_options.mode = format::TableColumnMappingMode::BY_FIELD_ID; @@ -73,6 +75,8 @@ class IcebergTableReader : public format::TableReader { } protected: + Status validate_file_mapping(const format::TableColumnMapper& mapper) const override; + void configure_mapper_options(format::TableColumnMapperOptions* options) const override { options->enable_row_lineage_virtual_columns = true; options->allow_idless_complex_wrapper_projection = diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index 07d2ed652a4306..d09bf9e469c190 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -414,6 +414,7 @@ class TableReader { DORIS_CHECK(file_schema != nullptr); return Status::OK(); } + virtual Status validate_file_mapping(const TableColumnMapper&) const { return Status::OK(); } // Open the concrete reader for the current split/task and build the file-local scan request. virtual Status open_reader() { @@ -457,6 +458,7 @@ class TableReader { RETURN_IF_ERROR(close_current_reader()); return Status::OK(); } + RETURN_IF_ERROR(validate_file_mapping(*_data_reader.column_mapper)); // COUNT(*) has no semantic column argument, but Nereids retains a minimum-width scan slot // so the scan node still has an output tuple. Record only the current non-predicate file // columns before table-format hooks add row-position or equality-delete dependencies. This @@ -479,40 +481,34 @@ class TableReader { _data_reader.file_block_layout.clear(); _data_reader.block_template.clear(); _file_scan_request.reset(); - _data_reader.file_block_layout.resize(file_request->local_positions.size()); + _data_reader.file_block_layout.resize(file_request->block_column_count()); // 4. Build file block layout from file schema and column mapping. The layout describes // the block returned by file reader before table-column materialization. - for (const auto& [file_column_id, block_position] : file_request->local_positions) { + auto add_file_block_column = [&](const LocalColumnIndex& projection, + LocalIndex block_position) -> Status { DORIS_CHECK(block_position.value() < _data_reader.file_block_layout.size()); + const auto file_column_id = projection.column_id(); const auto* field = _find_column_definition(_data_reader.file_schema, file_column_id); DORIS_CHECK(field != nullptr); ColumnDefinition projected_field; - { - auto it = std::find_if( - file_request->non_predicate_columns.begin(), - file_request->non_predicate_columns.end(), - [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); - if (it != file_request->non_predicate_columns.end()) { - RETURN_IF_ERROR(project_column_definition(*field, *it, &projected_field)); - } - } - { - auto it = std::find_if( - file_request->predicate_columns.begin(), - file_request->predicate_columns.end(), - [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); - if (it != file_request->predicate_columns.end()) { - RETURN_IF_ERROR(project_column_definition(*field, *it, &projected_field)); - } - } + RETURN_IF_ERROR(project_column_definition(*field, projection, &projected_field)); _data_reader.file_block_layout[block_position.value()] = { .file_column_id = file_column_id, .name = projected_field.name, .type = projected_field.type, }; DORIS_CHECK(_data_reader.file_block_layout[block_position.value()].type != nullptr); + return Status::OK(); + }; + for (const auto& projection : file_request->predicate_columns) { + RETURN_IF_ERROR(add_file_block_column( + projection, file_request->local_positions.at(projection.column_id()))); + } + for (const auto& projection : file_request->non_predicate_columns) { + RETURN_IF_ERROR(add_file_block_column( + projection, file_request->non_predicate_position(projection.column_id()))); } // 5. Prepare block template from file block layout. The block template stores the block diff --git a/be/test/core/column/column_variant_v2_test.cpp b/be/test/core/column/column_variant_v2_test.cpp index 2665304cfa0601..9937555b96d7c8 100644 --- a/be/test/core/column/column_variant_v2_test.cpp +++ b/be/test/core/column/column_variant_v2_test.cpp @@ -34,6 +34,7 @@ #include "common/exception.h" #include "core/arena.h" #include "core/assert_cast.h" +#include "core/block/block.h" #include "core/column/column_const.h" #include "core/column/column_decimal.h" #include "core/column/column_nullable.h" @@ -51,6 +52,7 @@ #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_time.h" #include "core/data_type/data_type_timestamptz.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/value/decimalv2_value.h" #include "core/value/ipv4_value.h" #include "core/value/ipv6_value.h" @@ -1788,6 +1790,31 @@ TEST(ColumnVariantV2Test, CowDetachAndClear) { EXPECT_EQ(a.num_elements(), 2); } +TEST(ColumnVariantV2Test, BlockClearDetachesSharedEncodedSubcolumns) { + for (bool clear_selected_only : {false, true}) { + SCOPED_TRACE(clear_selected_only); + auto column = ColumnVariantV2::create(); + insert_encoded_field(*column, encode_json(R"({"a":[1,2]})")); + const std::vector shared_subcolumns = subcolumns(*column); + + Block block; + block.insert({std::move(column), std::make_shared(), "v"}); + ASSERT_TRUE(block.get_by_position(0).column->is_exclusive()); + + if (clear_selected_only) { + block.clear_column_data(std::vector {0}); + } else { + block.clear_column_data(); + } + + EXPECT_EQ(block.get_by_position(0).column->size(), 0); + ASSERT_EQ(shared_subcolumns.size(), 3); + EXPECT_EQ(shared_subcolumns[0]->size(), 1); + EXPECT_EQ(shared_subcolumns[1]->size(), 1); + EXPECT_EQ(shared_subcolumns[2]->size(), 1); + } +} + TEST(ColumnVariantV2Test, EncodedRowCountInvariant) { EXPECT_DEATH( { diff --git a/be/test/exec/scan/access_path_parser_test.cpp b/be/test/exec/scan/access_path_parser_test.cpp index f72da2a69b0b56..23b95d54c0d7b6 100644 --- a/be/test/exec/scan/access_path_parser_test.cpp +++ b/be/test/exec/scan/access_path_parser_test.cpp @@ -32,6 +32,7 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/vexpr_context.h" #include "exprs/vliteral.h" @@ -130,6 +131,92 @@ TEST(AccessPathParserTest, IgnoresPrimitiveColumnsAndScannerVirtualColumns) { EXPECT_TRUE(rowid.children.empty()); } +TEST(AccessPathParserTest, PreservesVariantObjectKeysForPhysicalShreddingProjection) { + auto variant = root_column(100, "v", std::make_shared()); + auto status = AccessPathParser::build_nested_children( + &variant, + std::vector {data_access_path({"100", "typed_col"}), + data_access_path({"100", "nested", "leaf"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(variant.variant_access_paths, + (std::vector> {{"nested", "leaf"}, {"typed_col"}})); + + status = AccessPathParser::build_nested_children( + &variant, std::vector {data_access_path({"100"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); + + status = AccessPathParser::build_nested_children( + &variant, + std::vector {data_access_path({"100", "typed_col"}), + data_access_path({})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); +} + +TEST(AccessPathParserTest, SeparatesFinalAndPredicateComplexAccessPaths) { + auto variant = root_column(100, "v", std::make_shared()); + auto status = AccessPathParser::build_nested_children( + &variant, std::vector {data_access_path({"v"})}, + std::vector {data_access_path({"v", "n"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); + ASSERT_TRUE(variant.has_predicate_access_paths); + EXPECT_EQ(variant.predicate_variant_access_paths, + (std::vector> {{"n"}})); + + auto int_type = std::make_shared(); + auto struct_type = + std::make_shared(DataTypes {int_type, int_type}, Strings {"a", "b"}); + auto structure = root_column(101, "s", struct_type); + status = AccessPathParser::build_nested_children( + &structure, std::vector {data_access_path({"s"})}, + std::vector {data_access_path({"s", "b"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(structure.children.size(), 2); + ASSERT_TRUE(structure.has_predicate_access_paths); + ASSERT_EQ(structure.predicate_children.size(), 1); + EXPECT_EQ(structure.predicate_children[0].name, "b"); +} + +TEST(AccessPathParserTest, PreservesVariantPathsNestedInComplexColumns) { + auto variant_type = std::make_shared(); + + auto struct_type = + std::make_shared(DataTypes {variant_type}, Strings {"payload"}); + auto structure = root_column(100, "s", struct_type); + auto status = AccessPathParser::build_nested_children( + &structure, + std::vector {data_access_path({"s", "payload", "typed_col"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(structure.children.size(), 1); + EXPECT_EQ(structure.children[0].variant_access_paths, + (std::vector> {{"typed_col"}})); + + auto array = root_column(101, "items", std::make_shared(variant_type)); + status = AccessPathParser::build_nested_children( + &array, std::vector {data_access_path({"items", "*", "kind"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(array.children.size(), 1); + EXPECT_EQ(array.children[0].variant_access_paths, + (std::vector> {{"kind"}})); + + auto map = root_column( + 102, "attrs", + std::make_shared(std::make_shared(), variant_type)); + status = AccessPathParser::build_nested_children( + &map, std::vector {data_access_path({"attrs", "*", "enabled"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + const auto* value = find_child_by_name(map, "value"); + ASSERT_NE(value, nullptr); + EXPECT_EQ(value->variant_access_paths, (std::vector> {{"enabled"}})); +} + // Scenario: reject unsupported top-level inputs before recursive type parsing, including META // paths, missing DATA payloads, and access paths whose root does not match the projected slot. TEST(AccessPathParserTest, RejectsUnsupportedTopLevelAccessPathInputs) { diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp b/be/test/exec/scan/file_scanner_v2_test.cpp index 3506e5db27ae8a..fa167d2c7dce4a 100644 --- a/be/test/exec/scan/file_scanner_v2_test.cpp +++ b/be/test/exec/scan/file_scanner_v2_test.cpp @@ -117,6 +117,16 @@ TEST(FileScannerTest, V1CountPushdownRequiresExplicitCountStarArguments) { TPushAggOp::type::MINMAX, std::nullopt)); } +TEST(FileScannerTest, CountStarPlaceholderIsNotASemanticProjection) { + EXPECT_TRUE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, + std::vector {})); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, + std::vector {7})); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, std::nullopt)); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::MINMAX, + std::vector {})); +} + TEST(FileScannerV2Test, AdaptiveBatchSizeRunsForCountFallbackOnly) { EXPECT_TRUE(FileScannerV2::TEST_should_run_adaptive_batch_size(true, false)); EXPECT_FALSE(FileScannerV2::TEST_should_run_adaptive_batch_size(true, true)); @@ -456,6 +466,30 @@ TEST(FileScannerV2Test, FileScanLocalStateSelectsV2ForSupportedQueriesOnly) { EXPECT_FALSE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); } +TEST(FileScannerV2Test, LegacyCountExemptionRequiresMetadataCountOnEveryRange) { + auto scan_range = [](std::optional row_count) { + TScanRangeParams params; + auto& file_range = params.scan_range.ext_scan_range.file_scan_range; + TFileRangeDesc range; + if (row_count.has_value()) { + TTableFormatFileDesc table_format; + table_format.__set_table_level_row_count(*row_count); + range.__set_table_format_params(table_format); + } + file_range.ranges.push_back(std::move(range)); + return params; + }; + + LocalSplitSourceConnector proven({scan_range(4), scan_range(0)}, 2); + EXPECT_TRUE(proven.all_ranges_have_table_level_row_count()); + + LocalSplitSourceConnector missing({scan_range(4), scan_range(std::nullopt)}, 2); + EXPECT_FALSE(missing.all_ranges_have_table_level_row_count()); + + LocalSplitSourceConnector invalid({scan_range(4), scan_range(-1)}, 2); + EXPECT_FALSE(invalid.all_ranges_have_table_level_row_count()); +} + TEST(FileScannerV2Test, JniCompatibilityShapesUseV2Scanner) { TQueryOptions query_options; query_options.__set_enable_file_scanner_v2(true); diff --git a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp index 960eba0f9c1bd9..23b7cd815c5f54 100644 --- a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp +++ b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp @@ -44,12 +44,16 @@ namespace doris { class TestScanner final : public Scanner { public: TestScanner(RuntimeState* state, ScanLocalStateBase* local_state, int64_t limit, - RuntimeProfile* profile) - : Scanner(state, local_state, limit, profile) {} + RuntimeProfile* profile, bool allow_padding = true) + : Scanner(state, local_state, limit, profile), _allow_padding(allow_padding) {} void add_block(Block block) { _blocks.push_back(std::move(block)); } protected: + bool _can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const override { + return _allow_padding; + } + Status _get_block_impl(RuntimeState* /*state*/, Block* block, bool* eof) override { if (_blocks.empty()) { *eof = true; @@ -62,6 +66,7 @@ class TestScanner final : public Scanner { } private: + bool _allow_padding = true; std::list _blocks; }; @@ -210,4 +215,44 @@ TEST(ScannerProjectionTest, publishes_shared_column_and_reuses_output_block) { EXPECT_EQ(output.get_by_position(0).column->get_int(1), 4); } +TEST(ScannerProjectionTest, projects_incompatible_blocks_before_reading_the_next_block) { + ObjectPool pool; + auto data_type = std::make_shared(); + auto row_descriptor = MockRowDescriptor({data_type}, &pool); + + MockRuntimeState state; + state._batch_size = 8; + + auto op = std::make_shared(); + op->_row_descriptor = row_descriptor; + op->_output_row_descriptor = + std::make_unique(std::vector {data_type}, &pool); + op->_output_tuple_desc = op->_output_row_descriptor->tuple_descriptors()[0]; + + auto local_state = std::make_shared(&state, op.get()); + local_state->_projections = MockSlotRef::create_mock_contexts(0, data_type); + + RuntimeProfile profile("scanner"); + TestScanner scanner(&state, local_state.get(), -1, &profile, false); + ASSERT_TRUE(scanner.init(&state, {}).ok()); + scanner.add_block(ColumnHelper::create_block({0, 1})); + scanner.add_block(ColumnHelper::create_block({2, 3, 4})); + + Block first_output; + bool eos = false; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &first_output, &eos).ok()); + EXPECT_FALSE(eos); + EXPECT_EQ(first_output.rows(), 2); + + Block second_output; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &second_output, &eos).ok()); + EXPECT_FALSE(eos); + EXPECT_EQ(second_output.rows(), 3); + + Block final_output; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &final_output, &eos).ok()); + EXPECT_TRUE(eos); + EXPECT_EQ(final_output.rows(), 0); +} + } // namespace doris diff --git a/be/test/exec/sink/viceberg_merge_sink_test.cpp b/be/test/exec/sink/viceberg_merge_sink_test.cpp index eb7c0159d5fd38..d1da8fb7bfe1d5 100644 --- a/be/test/exec/sink/viceberg_merge_sink_test.cpp +++ b/be/test/exec/sink/viceberg_merge_sink_test.cpp @@ -33,6 +33,7 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "exec/sink/sink_common.h" #include "exec/sink/viceberg_delete_sink.h" #include "exec/sink/writer/iceberg/viceberg_table_writer.h" #include "exprs/vexpr_context.h" @@ -200,6 +201,43 @@ TEST_F(VIcebergMergeSinkTest, TestUpdateProducesDeleteAndInsert) { ASSERT_TRUE(sink->close(Status::OK()).ok()); } +TEST_F(VIcebergMergeSinkTest, TestDeleteOnlySkipsVariantDataWriter) { + ObjectPool pool; + MockRuntimeState state; + + DataTypes types {std::make_shared(), + std::make_shared(DataTypes {std::make_shared(), + std::make_shared()}, + Strings {"file_path", "row_position"}), + std::make_shared(), std::make_shared()}; + MockRowDescriptor row_desc(types, &pool); + + auto output_exprs = build_output_exprs(&pool, &state, row_desc); + TDataSink t_sink = build_sink(); + t_sink.iceberg_merge_sink.__set_writes_data_files(false); + t_sink.iceberg_merge_sink.__set_schema_json( + "{\"type\":\"struct\",\"schema-id\":0,\"fields\":[" + "{\"id\":1,\"name\":\"payload\",\"required\":false,\"type\":\"variant\"}" + "]}"); + + auto sink = std::make_shared(t_sink, output_exprs, nullptr, nullptr); + sink->set_skip_io(true); + + ASSERT_TRUE(sink->init_properties(&pool, row_desc).ok()); + EXPECT_EQ(nullptr, sink->_table_writer); + RuntimeProfile profile("iceberg_merge_sink"); + ASSERT_TRUE(sink->open(&state, &profile).ok()); + + // Delete-only plans must never use the insert opcode, which intentionally requires a data writer. + Block block = build_block_with_ops({kDeleteOperation}); + Status status = sink->write(&state, block); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(1, sink->_delete_row_count); + EXPECT_EQ(0, sink->_insert_row_count); + + ASSERT_TRUE(sink->close(Status::OK()).ok()); +} + TEST_F(VIcebergMergeSinkTest, TestMissingOperationColumn) { ObjectPool pool; MockRuntimeState state; diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index dd669f84a061f2..ab644372333081 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -39,6 +39,7 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_varbinary.h" +#include "core/data_type/data_type_variant_v2.h" #include "exprs/vectorized_fn_call.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" @@ -65,18 +66,10 @@ DataTypePtr i64() { return std::make_shared(); } -DataTypePtr f32() { - return std::make_shared(); -} - DataTypePtr f64() { return std::make_shared(); } -DataTypePtr dec32(uint32_t precision, uint32_t scale) { - return std::make_shared(precision, scale); -} - DataTypePtr str() { return std::make_shared(); } @@ -93,6 +86,10 @@ DataTypePtr u8() { return std::make_shared(); } +DataTypePtr variant_v2() { + return std::make_shared(); +} + ColumnDefinition field_id_col(const std::string& name, int32_t field_id, DataTypePtr type, int32_t local_id = -1) { ColumnDefinition column; @@ -518,14 +515,6 @@ VExprSPtr like_expr(const VExprSPtr& left, const std::string& pattern) { return expr; } -VExprSPtr struct_element_by_selector(const VExprSPtr& parent, DataTypePtr child_type, - const VExprSPtr& selector) { - auto expr = std::make_shared("struct_element", std::move(child_type)); - expr->add_child(parent); - expr->add_child(selector); - return expr; -} - VExprSPtr int_gt(const VExprSPtr& left, int32_t value) { auto expr = std::make_shared("gt", u8(), TExprNodeType::BINARY_PRED, TExprOpcode::GT); @@ -543,58 +532,12 @@ VExprSPtr binary_predicate(TExprOpcode::type opcode, const VExprSPtr& left, return expr; } -VExprSPtr in_predicate(const VExprSPtr& probe, const DataTypePtr& literal_type, - const std::vector& values) { - auto expr = std::make_shared("in", u8(), TExprNodeType::IN_PRED); - expr->add_child(probe); - for (const auto& value : values) { - expr->add_child(literal(literal_type, value)); - } - return expr; -} - -VExprSPtr null_predicate(const VExprSPtr& child, bool is_null) { - auto expr = - std::make_shared(is_null ? "is_null_pred" : "is_not_null_pred", u8()); - expr->add_child(child); - return expr; -} - VExprSPtr cast_expr(const VExprSPtr& child, DataTypePtr target_type) { auto expr = Cast::create_shared(std::move(target_type)); expr->add_child(child); return expr; } -VExprSPtr compound_predicate(TExprOpcode::type opcode, const VExprSPtr& left, - const VExprSPtr& right) { - auto expr = std::make_shared("compound", u8(), TExprNodeType::COMPOUND_PRED, - opcode); - expr->add_child(left); - expr->add_child(right); - return expr; -} - -std::vector collect_paths(const VExprSPtr& expr) { - std::vector paths; - collect_nested_struct_paths(expr, &paths); - return paths; -} - -void expect_name_selector(const StructChildSelector& selector, const std::string& name) { - EXPECT_TRUE(selector.by_name); - EXPECT_EQ(selector.name, name); -} - -void expect_ordinal_selector(const StructChildSelector& selector, size_t ordinal) { - EXPECT_FALSE(selector.by_name); - EXPECT_EQ(selector.ordinal, ordinal); -} - -void expect_path_root(const NestedStructPath& path, size_t global_index) { - EXPECT_EQ(path.root_global_index, GlobalIndex(global_index)); -} - class ColumnMapperCastTest : public testing::Test { protected: void SetUp() override { state.set_enable_strict_cast(true); } @@ -890,225 +833,6 @@ TEST(ColumnMapperNestedHelperTest, BuildsProjectionByNameAndOrdinalSelectors) { EXPECT_EQ(ordinal_projection.children[0].local_id(), 0); } -// ---------------------------------------------------------------------- -// collect_nested_struct_paths() helper tests. -// These tests assert the entry helper for nested scan projection: it only discovers -// table-side struct paths. Later localization decides how to add scan projections. -// ---------------------------------------------------------------------- - -TEST(ColumnMapperCollectNestedStructPathsTest, CollectsNameOrdinalAndBooleanSelectors) { - const auto leaf_type = i32(); - const auto inner_type = - std::make_shared(DataTypes {leaf_type, leaf_type}, Strings {"x", "y"}); - const auto root_type = std::make_shared(DataTypes {inner_type, leaf_type}, - Strings {"nested", "missing"}); - const auto root = table_slot(0, 3, root_type, "s"); - - const auto nested_by_ordinal = struct_element_by_selector( - struct_element_by_selector(root, inner_type, - literal(i32(), Field::create_field(1))), - leaf_type, literal(i32(), Field::create_field(2))); - auto paths = collect_paths(nested_by_ordinal); - ASSERT_EQ(paths.size(), 1); - expect_path_root(paths[0], 3); - ASSERT_EQ(paths[0].selectors.size(), 2); - expect_ordinal_selector(paths[0].selectors[0], 1); - expect_ordinal_selector(paths[0].selectors[1], 2); - - const std::vector positive_ordinal_selectors = { - literal(std::make_shared(), - Field::create_field(static_cast(1))), - literal(std::make_shared(), - Field::create_field(static_cast(2))), - literal(i32(), Field::create_field(3)), - literal(i64(), Field::create_field(4)), - literal(u8(), Field::create_field(true)), - }; - for (size_t idx = 0; idx < positive_ordinal_selectors.size(); ++idx) { - const auto selected = - struct_element_by_selector(root, leaf_type, positive_ordinal_selectors[idx]); - paths = collect_paths(selected); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 1); - expect_ordinal_selector(paths[0].selectors[0], idx == 4 ? 1 : idx + 1); - } - - paths = collect_paths(struct_element(root, leaf_type, "missing")); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 1); - expect_name_selector(paths[0].selectors[0], "missing"); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, IgnoresInvalidSelectorsAndNonPathRoots) { - const auto leaf_type = i32(); - const auto root_type = std::make_shared(DataTypes {leaf_type}, Strings {"a"}); - const auto root = table_slot(0, 0, root_type, "s"); - - const std::vector invalid_selectors = { - literal(i32(), Field::create_field(0)), - literal(i32(), Field::create_field(-1)), - literal(u8(), Field::create_field(false)), - literal(f32(), Field::create_field(1.0F)), - literal(f64(), Field::create_field(1.0)), - table_slot(1, 1, i32(), "selector"), - }; - for (const auto& selector : invalid_selectors) { - EXPECT_TRUE(collect_paths(struct_element_by_selector(root, leaf_type, selector)).empty()); - } - - auto wrong_arity = std::make_shared("struct_element", leaf_type); - wrong_arity->add_child(root); - EXPECT_TRUE(collect_paths(wrong_arity).empty()); - - auto not_struct_element = std::make_shared("other_function", leaf_type); - not_struct_element->add_child(root); - not_struct_element->add_child(literal(str(), Field::create_field("a"))); - EXPECT_TRUE(collect_paths(not_struct_element).empty()); - - EXPECT_TRUE(collect_paths(struct_element(literal(str(), Field::create_field("x")), - leaf_type, "a")) - .empty()); - EXPECT_TRUE(collect_paths(nullptr).empty()); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, RecursesThroughExpressionsAndKeepsCompletePath) { - const auto leaf_type = i32(); - const auto inner_type = std::make_shared(DataTypes {leaf_type}, Strings {"b"}); - const auto root_type = - std::make_shared(DataTypes {inner_type, leaf_type}, Strings {"a", "c"}); - const auto root = table_slot(0, 2, root_type, "s"); - const auto path_a = struct_element_by_selector( - root, inner_type, literal(str(), Field::create_field("a"))); - const auto path_ab = struct_element_by_selector( - path_a, leaf_type, literal(str(), Field::create_field("b"))); - const auto path_c = struct_element_by_selector( - root, leaf_type, literal(str(), Field::create_field("c"))); - - auto paths = collect_paths(binary_predicate( - TExprOpcode::GT, path_ab, literal(leaf_type, Field::create_field(1)))); - ASSERT_EQ(paths.size(), 1); - expect_path_root(paths[0], 2); - ASSERT_EQ(paths[0].selectors.size(), 2); - expect_name_selector(paths[0].selectors[0], "a"); - expect_name_selector(paths[0].selectors[1], "b"); - - paths = collect_paths(compound_predicate( - TExprOpcode::COMPOUND_OR, - binary_predicate(TExprOpcode::GT, path_ab, - literal(leaf_type, Field::create_field(1))), - binary_predicate(TExprOpcode::LT, path_c, - literal(leaf_type, Field::create_field(2))))); - ASSERT_EQ(paths.size(), 2); - ASSERT_EQ(paths[0].selectors.size(), 2); - ASSERT_EQ(paths[1].selectors.size(), 1); - expect_name_selector(paths[0].selectors[0], "a"); - expect_name_selector(paths[0].selectors[1], "b"); - expect_name_selector(paths[1].selectors[0], "c"); - - auto fn = std::make_shared("fn", leaf_type); - fn->add_child(path_ab); - fn->add_child(table_slot(3, 4, leaf_type, "other")); - paths = collect_paths(fn); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 2); - - auto if_expr = std::make_shared("if", leaf_type); - if_expr->add_child(literal(u8(), Field::create_field(true))); - if_expr->add_child(path_ab); - if_expr->add_child(path_c); - paths = collect_paths(if_expr); - ASSERT_EQ(paths.size(), 2); - - paths = collect_paths(compound_predicate(TExprOpcode::COMPOUND_AND, path_ab, path_ab)); - ASSERT_EQ(paths.size(), 2); - - paths = collect_paths(path_ab); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 2); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, CastBehaviorSeparatesProjectionAndPruningRules) { - const auto int_type = i32(); - const auto bigint_type = i64(); - const auto float_type = f32(); - const auto double_type = f64(); - const auto decimal_small = dec32(8, 2); - const auto decimal_wide = dec32(9, 2); - const auto decimal_changed_scale = dec32(9, 3); - - const auto root_type = std::make_shared( - DataTypes {int_type, float_type, decimal_small}, Strings {"i", "f", "d"}); - const auto root = table_slot(0, 0, root_type, "s"); - const auto int_path = struct_element(root, int_type, "i"); - const auto float_path = struct_element(root, float_type, "f"); - const auto decimal_path = struct_element(root, decimal_small, "d"); - - auto paths = collect_paths(cast_expr(int_path, bigint_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - paths = collect_paths(cast_expr(float_path, double_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "f"); - - paths = collect_paths(cast_expr(decimal_path, decimal_wide)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "d"); - - paths = collect_paths( - cast_expr(struct_element(root, make_nullable(int_type), "i"), make_nullable(int_type))); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - // Unsafe casts are not accepted as pruning paths, but collect_nested_struct_paths() still - // recurses into children so scan projection can read the column needed by row-level filters. - paths = collect_paths(cast_expr(struct_element(root, bigint_type, "i"), int_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - paths = collect_paths(cast_expr(decimal_path, decimal_changed_scale)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "d"); - - EXPECT_TRUE(collect_paths(cast_expr(table_slot(1, 1, int_type, "plain"), bigint_type)).empty()); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, ProjectionMergeKeepsFilterOnlyPathAndDeduplicates) { - const auto int_type = i32(); - const auto string_type = str(); - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", int_type); - auto table_output = struct_name_col("s", {table_a}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", int_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b, name_col("c", string_type, 2)}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_output}, {}, {file_struct}).ok()); - - const auto path_b = - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "b"); - auto filter_expr = compound_predicate( - TExprOpcode::COMPOUND_AND, - binary_predicate(TExprOpcode::GT, path_b, - literal(int_type, Field::create_field(1))), - binary_predicate(TExprOpcode::LT, path_b, - literal(int_type, Field::create_field(10)))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_output}, &request).ok()); - - EXPECT_TRUE(request.non_predicate_columns.empty()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - ASSERT_FALSE(request.predicate_columns[0].project_all_children); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); -} - // Scenario: row-oriented readers such as CSV/Text cannot lazy-read predicate columns separately. // For a complex root that is both projected and referenced by a filter, the materialized mapper // keeps one non-predicate scan entry and asks the reader to read the full top-level struct. @@ -2409,41 +2133,6 @@ TEST(ColumnMapperLocalizeFiltersTest, ConstantFilterBuildsEntryWithoutFileScanCo mapper.mappings()[0].constant_index); } -TEST(ColumnMapperLocalizeFiltersTest, NestedFilterOnlyChildMergesIntoPredicateProjection) { - const auto int_type = i32(); - const auto string_type = str(); - - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"), 10); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.localize_filters({filter}, &request).ok()); - - EXPECT_TRUE(request.non_predicate_columns.empty()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - ASSERT_FALSE(request.predicate_columns[0].project_all_children); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); - ASSERT_EQ(request.local_positions.size(), 1); - EXPECT_EQ(request.local_positions.at(LocalColumnId(5)), LocalIndex(0)); - ASSERT_TRUE(mapper.filter_entries().at(GlobalIndex(0)).is_local()); - EXPECT_EQ(mapper.filter_entries().at(GlobalIndex(0)).local_index(), LocalIndex(0)); -} - TEST(ColumnMapperLocalizeFiltersTest, PreservesExistingScanStateWhenAddingPredicateColumn) { const auto int_type = i32(); const std::vector table_schema = { @@ -2542,24 +2231,24 @@ TEST(ColumnMapperScanRequestTest, OrdinaryPredicateSlotRetainsPayloadForScannerB EXPECT_TRUE(request.predicate_only_columns.empty()); } -TEST(ColumnMapperScanRequestTest, StructOutputAndFilterOnlyChildAreMerged) { +TEST(ColumnMapperScanRequestTest, StructAllAccessPathsAreEagerWithoutPredicateMapping) { const auto int_type = i32(); const auto string_type = str(); auto table_a = name_col("a", int_type); auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); + auto table_struct = struct_name_col("s", {table_a, table_b}); auto file_a = name_col("a", int_type, 0); auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); + auto file_c = name_col("c", int_type, 2); + auto file_struct = struct_name_col("s", {file_a, file_b, file_c}, 5); TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"), 10); + auto filter_expr = + int_gt(struct_element(table_slot(0, 0, table_struct.type, "s"), int_type, "a"), 10); TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), .global_indices = {GlobalIndex(0)}}; @@ -2573,159 +2262,6 @@ TEST(ColumnMapperScanRequestTest, StructOutputAndFilterOnlyChildAreMerged) { EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); } -TEST(ColumnMapperScanRequestTest, RenamedNestedPredicateTargetsMappedFileChild) { - const auto int_type = i32(); - - auto table_a = field_id_col("a", 1, int_type); - auto table_renamed_b = field_id_col("renamed_b", 2, int_type); - auto table_struct = struct_col("s", 10, {table_a, table_renamed_b}); - auto file_a = field_id_col("a", 1, int_type, 0); - auto file_b = field_id_col("b", 2, int_type, 1); - auto file_struct = struct_col("s", 10, {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, table_struct.type, "s"), int_type, "renamed_b"), 10); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, NestedInNullAndReverseComparisonFiltersAreMerged) { - const auto int_type = i32(); - const auto string_type = str(); - - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"); - auto in_filter = - in_predicate(nested_a, int_type, - {Field::create_field(5), Field::create_field(7)}); - auto reverse_filter = binary_predicate( - TExprOpcode::LT, literal(int_type, Field::create_field(3)), nested_a); - auto null_filter = null_predicate(nested_a, true); - auto not_null_filter = null_predicate(nested_a, false); - auto filter_expr = compound_predicate( - TExprOpcode::COMPOUND_AND, - compound_predicate(TExprOpcode::COMPOUND_AND, in_filter, reverse_filter), - compound_predicate(TExprOpcode::COMPOUND_AND, null_filter, not_null_filter)); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, NestedPredicateFilterThroughSafeCast) { - const auto file_int_type = i32(); - const auto table_bigint_type = i64(); - const auto string_type = str(); - - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = std::make_shared( - DataTypes {table_bigint_type, string_type}, Strings {"a", "b"}); - - auto file_a = name_col("a", file_int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct, "s"), file_int_type, "a"); - auto filter_expr = - binary_predicate(TExprOpcode::GT, cast_expr(nested_a, table_bigint_type), - literal(table_bigint_type, Field::create_field(5))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, UnsafeCastDoesNotBuildNestedPredicateFilter) { - const auto file_bigint_type = i64(); - const auto table_int_type = i32(); - const auto string_type = str(); - - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = std::make_shared( - DataTypes {table_int_type, string_type}, Strings {"a", "b"}); - - auto file_a = name_col("a", file_bigint_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct, "s"), file_bigint_type, "a"); - auto filter_expr = binary_predicate(TExprOpcode::GT, cast_expr(nested_a, table_int_type), - literal(table_int_type, Field::create_field(5))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); -} - -TEST(ColumnMapperScanRequestTest, DeepNestedPredicateTargetsLeafPath) { - const auto id_type = i32(); - const auto name_type = str(); - const auto string_type = str(); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - - auto full_table_inner_type = - std::make_shared(DataTypes {id_type, name_type}, Strings {"id", "n"}); - auto full_table_struct_type = std::make_shared( - DataTypes {full_table_inner_type, string_type}, Strings {"a", "b"}); - - auto file_id = name_col("id", id_type, 0); - auto file_name = name_col("n", name_type, 1); - auto file_a = struct_name_col("a", {file_id, file_name}, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_id = - struct_element(struct_element(table_slot(0, 0, full_table_struct_type, "s"), - full_table_inner_type, "a"), - id_type, "id"); - auto filter_expr = - in_predicate(nested_id, id_type, - {Field::create_field(5), Field::create_field(7)}); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - TEST(ColumnMapperScanRequestTest, ArrayStructProjectionPrunesElementChildren) { const auto int_type = i32(); const auto string_type = str(); @@ -2878,51 +2414,6 @@ TEST(ColumnMapperScanRequestTest, ArrayWrapperDoesNotBuildNestedPredicateFilter) EXPECT_TRUE(request.non_predicate_columns[0].children.empty()); } -// Scenario: a map value struct projects child `b`, while a row filter reads value child `a`. -// The filter is too complex to become a file-local nested predicate. Lazy demotion must move the -// merged projection to the non-predicate set without dropping either physical value child. -TEST(ColumnMapperScanRequestTest, MapFilterOnlyValueChildMergesWithOutputProjection) { - const auto key_type = i32(); - const auto int_type = i32(); - const auto string_type = str(); - - auto table_value_b = name_col("b", string_type); - auto table_value = struct_name_col("value", {table_value_b}); - auto table_map = map_col("m", -1, {table_value}, key_type, table_value.type); - set_name_identifiers(&table_map, 0); - - auto file_key = name_col("key", key_type, 0); - auto file_value_a = name_col("a", int_type, 0); - auto file_value_b = name_col("b", string_type, 1); - auto file_value = struct_name_col("value", {file_value_a, file_value_b}, 1); - auto file_map = map_col("m", -1, {file_key, file_value}, key_type, file_value.type, 0); - set_name_identifiers(&file_map, 0); - - auto full_value_type = - std::make_shared(DataTypes {int_type, string_type}, Strings {"a", "b"}); - auto full_map_type = std::make_shared(key_type, full_value_type); - auto value_expr = - struct_element(table_slot(0, 0, full_map_type, "m"), full_value_type, "value"); - auto filter_expr = int_gt(struct_element(value_expr, int_type, "a"), 5); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_map}, {}, {file_map}).ok()); - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_map}, &request).ok()); - - EXPECT_TRUE(request.predicate_columns.empty()); - ASSERT_EQ(request.non_predicate_columns.size(), 1); - const auto& projection = request.non_predicate_columns[0]; - EXPECT_EQ(projection.column_id(), LocalColumnId(0)); - ASSERT_FALSE(projection.project_all_children); - ASSERT_EQ(projection.children.size(), 1); - EXPECT_EQ(projection.children[0].local_id(), 1); - EXPECT_EQ(projection_ids(projection.children[0].children), std::vector({0, 1})); -} - // Scenario: when projected struct children are an in-order prefix of the file struct, the mapper can // read those physical children directly without rebuilding the file-side complex type. TEST(ColumnMapperScanRequestTest, MatchingProjectedStructDoesNotNeedComplexRematerialize) { @@ -2987,9 +2478,10 @@ TEST(ColumnMapperScanRequestTest, RenameOnlyProjectedStructDoesNotRebuildFilePro EXPECT_TRUE(mapper.mappings()[0].is_trivial); } -// Scenario: a row filter references an unprojected struct child, so the predicate projection is -// merged with the output projection and the mapper rebuilds the projected file struct type. -TEST(ColumnMapperScanRequestTest, PredicateProjectionRebuildsProjectedStructFileType) { +// Scenario: FE access paths are the sole contract for nested predicate projection. If a filter +// references a Struct child absent from all_access_paths and no predicate_access_paths were sent, +// File Scanner V2 must not infer and append that child from the expression. +TEST(ColumnMapperScanRequestTest, MissingPredicateAccessPathsDoNotInferStructProjection) { const auto int_type = i32(); const auto string_type = str(); @@ -3015,19 +2507,18 @@ TEST(ColumnMapperScanRequestTest, PredicateProjectionRebuildsProjectedStructFile FileScanRequest request; ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_TRUE(request.non_predicate_columns.empty()); - const auto& projection = request.predicate_columns[0]; + EXPECT_TRUE(request.predicate_columns.empty()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& projection = request.non_predicate_columns[0]; EXPECT_FALSE(projection.project_all_children); - EXPECT_EQ(projection_ids(projection.children), std::vector({0, 1, 2})); + EXPECT_EQ(projection_ids(projection.children), std::vector({0, 1})); const auto* mapped_type = assert_cast( remove_nullable(mapper.mappings()[0].file_type).get()); - ASSERT_EQ(mapped_type->get_elements().size(), 3); + ASSERT_EQ(mapped_type->get_elements().size(), 2); EXPECT_EQ(mapped_type->get_element_name(0), "a"); EXPECT_EQ(mapped_type->get_element_name(1), "b"); - EXPECT_EQ(mapped_type->get_element_name(2), "c"); - EXPECT_FALSE(mapper.mappings()[0].is_trivial); + EXPECT_TRUE(request.conjuncts.empty()); } // Scenario: a filter references a top-level column that is not projected by the query; the mapper @@ -3435,63 +2926,6 @@ TEST_F(ColumnMapperCastTest, NestedElementAtInPredicateUsesAllOrNothingLiteralRe EXPECT_TRUE(fallback_root->children()[2]->data_type()->equals(*table_bigint_type)); } -// Scenario: output projection reads one struct child while the row filter reads a different nested -// struct child. File-local conjunct rewrite must use the merged scan projection type. In the SQL -// shape below, `SELECT element_at(s, 'c') WHERE element_at(element_at(s, 'b'), 'cc') LIKE ...` -// reads file children `b.cc` and `c`; the localized inner `element_at(s, 'b')` returns -// `Struct(cc)`, not the full old file child `Struct(cc, new_dd)`. -TEST(ColumnMapperScanRequestTest, NestedElementAtConjunctUsesMergedScanProjectionChildType) { - const auto string_type = str(); - const auto int_type = i32(); - - auto table_cc = field_id_col("cc", 23, string_type); - auto table_new_dd = field_id_col("new_dd", 24, int_type); - auto table_b = struct_col("b", 20, {table_cc, table_new_dd}); - auto table_c = field_id_col("c", 25, string_type); - auto full_table_struct = struct_col("struct_column2", 19, {table_b, table_c}); - auto projected_table_struct = struct_col("struct_column2", 19, {table_c}); - - auto file_cc = field_id_col("cc", 23, string_type, 0); - auto file_new_dd = field_id_col("new_dd", 24, int_type, 1); - auto file_b = struct_col("b", 20, {file_cc, file_new_dd}, 0); - auto file_c = field_id_col("c", 25, string_type, 1); - auto file_struct = struct_col("new_struct_column", 19, {file_b, file_c}, 10); - - const auto table_slot_expr = table_slot(0, 0, full_table_struct.type, "struct_column2"); - const auto table_parent_expr = element_at(table_slot_expr, table_b.type, "b"); - const auto table_leaf_expr = element_at(table_parent_expr, string_type, "cc"); - auto filter_expr = like_expr(table_leaf_expr, "NestedC%"); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); - ASSERT_TRUE(mapper.create_mapping({projected_table_struct}, {}, {file_struct}).ok()); - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {projected_table_struct}, &request).ok()); - ASSERT_EQ(request.conjuncts.size(), 1); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(10)); - - const auto& localized_leaf = request.conjuncts[0]->root()->children()[0]; - ASSERT_EQ(localized_leaf->expr_name(), "element_at"); - const auto& localized_parent = localized_leaf->children()[0]; - ASSERT_EQ(localized_parent->expr_name(), "element_at"); - - const auto* localized_slot = - assert_cast(localized_parent->children()[0].get()); - EXPECT_EQ(localized_slot->column_name(), "new_struct_column"); - // The scan projection keeps the top-level file column id above, while the localized conjunct - // executes on the file-reader Block. The VSlotRef column id is therefore the block position of - // `new_struct_column` in this request, not the file schema id 10. - EXPECT_EQ(localized_slot->column_id(), 0); - - const auto* localized_parent_type = assert_cast( - remove_nullable(localized_parent->data_type()).get()); - ASSERT_EQ(localized_parent_type->get_elements().size(), 1); - EXPECT_EQ(localized_parent_type->get_element_name(0), "cc"); -} - // Scenario: struct child access through a computed map/array parent is not localized as a file // conjunct, because the projected value struct can have a different physical child order. TEST(ColumnMapperScanRequestTest, MapValuesStructChildConjunctStaysTableLevel) { @@ -4591,5 +4025,301 @@ TEST_F(ColumnMapperCastTest, ColumnMapperKeepsTableSlotIdWhenFileBlockPositionCh conjunct->close(); } +TEST(ColumnMapperTest, VariantAccessPathProjectsOnlyPhysicalTypedLeaf) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& root = request.non_predicate_columns[0]; + ASSERT_FALSE(root.project_all_children); + ASSERT_EQ(root.children.size(), 1); + EXPECT_EQ(root.children[0].local_id(), 2); + ASSERT_EQ(root.children[0].children.size(), 1); + EXPECT_EQ(root.children[0].children[0].local_id(), 0); + ASSERT_EQ(root.children[0].children[0].children.size(), 1); + EXPECT_EQ(root.children[0].children[0].children[0].local_id(), 1); + EXPECT_TRUE(root.children[0].children[0].children[0].project_all_children); +} + +TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredStructOutputProjection) { + auto table_a = field_id_col("a", 2, i64()); + auto table_b = field_id_col("b", 3, i64()); + auto table_struct = struct_col("s", 1, {table_a, table_b}); + table_struct.has_predicate_access_paths = true; + table_struct.predicate_children = {table_b}; + + auto file_a = field_id_col("a", 2, i64(), 0); + auto file_b = field_id_col("b", 3, i64(), 1); + auto file_struct = struct_col("s", 1, {file_a, file_b}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + auto b = struct_element(table_slot(0, 0, table_struct.type, "s"), i64(), "b"); + auto predicate = binary_predicate(TExprOpcode::GT, b, + literal(i64(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns.size(), 1) << request.debug_string(); + ASSERT_EQ(request.predicate_columns[0].children.size(), 1); + EXPECT_EQ(request.predicate_columns[0].children[0].local_id(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); + EXPECT_EQ(request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); +} + +TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredVariantRootProjection) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.has_predicate_access_paths = true; + table_variant.predicate_variant_access_paths = {{"typed_col"}}; + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + + auto typed_col = + element_at(table_slot(0, 0, table_variant.type, "v"), variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i64()), + literal(i64(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_variant}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_FALSE(request.predicate_columns[0].project_all_children); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); + EXPECT_EQ(request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); +} + +TEST(ColumnMapperTest, NestedVariantAccessPathProjectsPhysicalTypedLeaf) { + auto table_variant = field_id_col("payload", 2, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + auto table_struct = struct_col("info", 1, {table_variant}); + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("payload", 2, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + auto file_struct = struct_col("info", 1, {std::move(file_variant)}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_struct}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& root = request.non_predicate_columns[0]; + ASSERT_FALSE(root.project_all_children); + ASSERT_EQ(root.children.size(), 1); + const auto& variant = root.children[0]; + EXPECT_EQ(variant.local_id(), 0); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].local_id(), 0); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); +} + +TEST(ColumnMapperTest, NestedVariantAllAccessPathKeepsPhysicalTypedLeaf) { + auto table_variant = field_id_col("payload", 2, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + auto table_struct = struct_col("info", 1, {table_variant}); + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("payload", 2, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + auto file_struct = struct_col("info", 1, {std::move(file_variant)}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + auto payload = + struct_element(table_slot(0, 0, table_struct.type, "info"), variant_v2(), "payload"); + auto typed_col = element_at(payload, variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i32()), + literal(i32(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + const auto& root = request.predicate_columns[0]; + ASSERT_EQ(root.children.size(), 1); + const auto& variant = root.children[0]; + ASSERT_FALSE(variant.project_all_children); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); +} + +TEST(ColumnMapperTest, ArrayAndMapNestedVariantPathsReachPhysicalTypedLeaf) { + auto make_file_variant = [](std::string name, int32_t field_id, int32_t local_id) { + auto wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, + 0); + auto typed = struct_name_col("typed_value", {std::move(wrapper)}, 2); + auto variant = field_id_col(name, field_id, variant_v2(), local_id); + variant.children = {name_col("metadata", varbinary(), 0), name_col("value", varbinary(), 1), + std::move(typed)}; + return variant; + }; + auto assert_variant_leaf = [](const LocalColumnIndex& variant) { + ASSERT_FALSE(variant.project_all_children); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); + }; + + { + auto table_element = field_id_col("element", 2, variant_v2()); + table_element.variant_access_paths = {{"typed_col"}}; + auto table_array = array_col("items", 1, table_element); + auto file_array = array_col("items", 1, make_file_variant("element", 2, 0), 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_array}, {}, {file_array}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_array}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns[0].children.size(), 1); + assert_variant_leaf(request.non_predicate_columns[0].children[0]); + } + + { + auto table_key = field_id_col("key", 2, str()); + auto table_value = field_id_col("value", 3, variant_v2()); + table_value.variant_access_paths = {{"typed_col"}}; + auto table_map = map_col("attributes", 1, {table_key, table_value}, str(), variant_v2()); + auto file_key = field_id_col("key", 2, str(), 0); + auto file_value = make_file_variant("value", 3, 1); + auto file_map = map_col("attributes", 1, {file_key, file_value}, str(), variant_v2(), 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_map}, {}, {file_map}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_map}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& children = request.non_predicate_columns[0].children; + const auto value_it = std::ranges::find_if( + children, [](const LocalColumnIndex& child) { return child.local_id() == 1; }); + ASSERT_NE(value_it, children.end()); + assert_variant_leaf(*value_it); + } +} + +TEST(ColumnMapperTest, VariantLeafProjectionRequiresLosslessObjectPath) { + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto dotted_wrapper = struct_name_col( + "a.b", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 1); + auto numeric_wrapper = struct_name_col( + "1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 2); + auto negative_numeric_wrapper = struct_name_col( + "-1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 3); + auto positive_numeric_wrapper = struct_name_col( + "+1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 4); + auto null_wrapper = struct_name_col( + "NULL", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 5); + auto typed_value = + struct_name_col("typed_value", + {std::move(field_wrapper), std::move(dotted_wrapper), + std::move(numeric_wrapper), std::move(negative_numeric_wrapper), + std::move(positive_numeric_wrapper), std::move(null_wrapper)}, + 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + for (const std::vector& path : + {std::vector {"a.b"}, std::vector {"a", "b"}, + std::vector {"1"}, std::vector {"-1"}, + std::vector {"+1"}, std::vector {"NULL"}}) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {path}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children) + << "unsafe Variant path must fall back to the complete physical subtree"; + } +} + +TEST(ColumnMapperTest, VariantLeafProjectionRequiresObjectTypedValue) { + auto element_wrapper = struct_name_col( + "element", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto array_typed_value = array_col("typed_value", -1, std::move(element_wrapper), 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(array_typed_value)}; + + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"element"}}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); +} + +TEST(ColumnMapperTest, VariantLeafProjectionDeclinesAmbiguousPrimitiveIdentity) { + auto field_wrapper = struct_name_col( + "binary_col", + {name_col("value", varbinary(), 0), name_col("typed_value", varbinary(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"binary_col"}}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); +} + } // namespace } // namespace doris::format diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp index bc73055ba88c5a..abc67839d2dc14 100644 --- a/be/test/format_v2/parquet/parquet_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_test.cpp @@ -48,17 +48,20 @@ #include "core/column/column_string.h" #include "core/column/column_struct.h" #include "core/column/column_vector.h" +#include "core/column/variant_v2/column_variant_v2.h" #include "core/data_type/data_type_array.h" #include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/data_type/primitive_type.h" #include "core/field.h" #include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" #include "exprs/vslot_ref.h" #include "format_v2/column_mapper.h" #include "format_v2/expr/delete_predicate.h" @@ -152,6 +155,132 @@ class Int32GreaterThanExpr final : public VExpr { const std::string _expr_name = "Int32GreaterThanExpr"; }; +class VariantPathMetadataExpr : public VExpr { +public: + VariantPathMetadataExpr(std::string name, DataTypePtr type, + TExprNodeType::type node_type = TExprNodeType::FUNCTION_CALL) + : VExpr(std::move(type), false), _name(std::move(name)) { + set_node_type(node_type); + } + + const std::string& expr_name() const override { return _name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("VariantPathMetadataExpr is not executable"); + } + +private: + std::string _name; +}; + +class VariantInt32PathGreaterThanExpr final : public VariantPathMetadataExpr { +public: + VariantInt32PathGreaterThanExpr(int column_id, std::string key, int32_t value) + : VariantPathMetadataExpr("gt", std::make_shared(), + TExprNodeType::BINARY_PRED), + _column_id(column_id), + _key(std::move(key)), + _value(value) {} + + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, + size_t count, ColumnPtr& result_column) const override { + const auto& nullable = + assert_cast(*block->get_by_position(_column_id).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef(_key)}}; + const auto typed = variants.find_shredded_typed_value(path); + if (!typed.has_value()) { + return Status::InternalError("Expected the projected Variant typed leaf"); + } + const auto& typed_nullable = assert_cast(*typed->column); + const auto& values = + assert_cast(typed_nullable.get_nested_column()).get_data(); + auto result = ColumnUInt8::create(); + auto& output = result->get_data(); + output.resize(count); + for (size_t row = 0; row < count; ++row) { + const size_t input_row = selector == nullptr ? row : (*selector)[row]; + output[row] = !nullable.is_null_at(input_row) && + !typed_nullable.is_null_at(input_row) && values[input_row] > _value; + } + result_column = std::move(result); + return Status::OK(); + } + +private: + int _column_id; + std::string _key; + int32_t _value; +}; + +VExprContextSPtr create_variant_int32_path_greater_than_conjunct(int column_id, std::string key, + int32_t value) { + auto slot = VSlotRef::create_shared(0, column_id, -1, + make_nullable(std::make_shared()), "v"); + auto key_literal = VLiteral::create_shared(std::make_shared(), + Field::create_field(key)); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key_literal); + auto cast = std::make_shared( + "CAST", make_nullable(std::make_shared()), TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared(std::make_shared(), + Field::create_field(value)); + auto gt = std::make_shared(column_id, std::move(key), value); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +class StructInt32ChildGreaterThanExpr final : public VExpr { +public: + StructInt32ChildGreaterThanExpr(int column_id, int32_t value) + : VExpr(std::make_shared(), false), + _column_id(column_id), + _value(value) {} + + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, + size_t count, ColumnPtr& result_column) const override { + const auto& nullable = + assert_cast(*block->get_by_position(_column_id).column); + const auto& structure = assert_cast(nullable.get_nested_column()); + const auto& child = assert_cast(structure.get_column(0)); + const auto& values = assert_cast(child.get_nested_column()).get_data(); + auto result = ColumnUInt8::create(); + auto& output = result->get_data(); + output.resize(count); + for (size_t row = 0; row < count; ++row) { + const size_t input_row = selector == nullptr ? row : (*selector)[row]; + output[row] = !nullable.is_null_at(input_row) && !child.is_null_at(input_row) && + values[input_row] > _value; + } + result_column = std::move(result); + return Status::OK(); + } + + const std::string& expr_name() const override { return _expr_name; } + + void collect_slot_column_ids(std::set& column_ids) const override { + column_ids.insert(_column_id); + } + +private: + int _column_id; + int32_t _value; + const std::string _expr_name = "StructInt32ChildGreaterThanExpr"; +}; + +VExprContextSPtr create_struct_int32_child_greater_than_conjunct(int column_id, int32_t value) { + auto context = VExprContext::create_shared( + std::make_shared(column_id, value)); + context->_prepared = true; + context->_opened = true; + return context; +} + class Int32DictionaryEqualsExpr final : public VExpr { public: Int32DictionaryEqualsExpr(int column_id, int32_t value) @@ -1578,6 +1707,447 @@ TEST_F(NewParquetReaderTest, CreatesParquetColumnMapper) { ASSERT_NE(dynamic_cast(mapper.get()), nullptr); } +TEST(ParquetVariantProjectionTest, ResidualStatisticsGuardPhysicalLeafProjection) { + using format::parquet::ParquetColumnSchema; + using format::parquet::ParquetColumnSchemaKind; + auto node = [](std::string name, int32_t local_id, ParquetColumnSchemaKind kind, + int leaf_id = -1) { + auto result = std::make_unique(); + result->name = std::move(name); + result->local_id = local_id; + result->kind = kind; + result->leaf_column_id = leaf_id; + return result; + }; + auto root = node("v", 0, ParquetColumnSchemaKind::VARIANT); + root->children.push_back(node("metadata", 0, ParquetColumnSchemaKind::PRIMITIVE, 0)); + root->children.push_back(node("value", 1, ParquetColumnSchemaKind::PRIMITIVE, 1)); + auto root_typed = node("typed_value", 2, ParquetColumnSchemaKind::STRUCT); + auto wrapper = node("n", 0, ParquetColumnSchemaKind::STRUCT); + wrapper->children.push_back(node("value", 0, ParquetColumnSchemaKind::PRIMITIVE, 2)); + wrapper->children.push_back(node("typed_value", 1, ParquetColumnSchemaKind::PRIMITIVE, 3)); + root_typed->children.push_back(std::move(wrapper)); + root->children.push_back(std::move(root_typed)); + + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(1)); + + tparquet::RowGroup row_group; + row_group.__set_num_rows(10); + for (int leaf = 0; leaf < 4; ++leaf) { + tparquet::Statistics statistics; + statistics.__set_null_count(leaf == 1 || leaf == 2 ? 10 : 0); + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(std::move(column_metadata)); + row_group.columns.push_back(std::move(chunk)); + } + tparquet::FileMetaData metadata; + metadata.row_groups.push_back(row_group); + EXPECT_TRUE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); + + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(9); + EXPECT_FALSE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); + metadata.row_groups[0].columns[2].meta_data.__isset.statistics = false; + EXPECT_FALSE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); +} + +TEST(ParquetVariantProjectionTest, FinalizesNestedVariantProjectionRecursively) { + using format::parquet::ParquetColumnSchema; + using format::parquet::ParquetColumnSchemaKind; + auto node = [](std::string name, int32_t local_id, ParquetColumnSchemaKind kind, + int leaf_id = -1) { + auto result = std::make_unique(); + result->name = std::move(name); + result->local_id = local_id; + result->kind = kind; + result->leaf_column_id = leaf_id; + return result; + }; + auto root = node("info", 0, ParquetColumnSchemaKind::STRUCT); + auto variant = node("payload", 0, ParquetColumnSchemaKind::VARIANT); + variant->children.push_back(node("metadata", 0, ParquetColumnSchemaKind::PRIMITIVE, 0)); + variant->children.push_back(node("value", 1, ParquetColumnSchemaKind::PRIMITIVE, 1)); + auto typed_object = node("typed_value", 2, ParquetColumnSchemaKind::STRUCT); + auto wrapper = node("n", 0, ParquetColumnSchemaKind::STRUCT); + wrapper->children.push_back(node("value", 0, ParquetColumnSchemaKind::PRIMITIVE, 2)); + wrapper->children.push_back(node("typed_value", 1, ParquetColumnSchemaKind::PRIMITIVE, 3)); + typed_object->children.push_back(std::move(wrapper)); + variant->children.push_back(std::move(typed_object)); + root->children.push_back(std::move(variant)); + + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.back().children.push_back( + format::LocalColumnIndex::local(1)); + + tparquet::RowGroup row_group; + row_group.__set_num_rows(10); + for (int leaf = 0; leaf < 4; ++leaf) { + tparquet::Statistics statistics; + statistics.__set_null_count(leaf == 1 || leaf == 2 ? 10 : 0); + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(std::move(column_metadata)); + row_group.columns.push_back(std::move(chunk)); + } + tparquet::FileMetaData metadata; + metadata.row_groups.push_back(row_group); + + EXPECT_EQ( + format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &projection), + 1); + EXPECT_FALSE(projection.children[0].project_all_children); + + auto fallback = projection; + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(9); + EXPECT_EQ(format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &fallback), + 0); + EXPECT_TRUE(fallback.children[0].project_all_children); + + auto repeated = projection; + root->children[0]->max_repetition_level = 1; + EXPECT_EQ(format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &repeated), + 0); + EXPECT_TRUE(repeated.children[0].project_all_children); +} + +TEST_F(NewParquetReaderTest, ReadsFullyShreddedVariantTypedLeafProjection) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_typed_leaf_projection"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + ASSERT_EQ(remove_nullable(schema[1].type)->get_primitive_type(), TYPE_VARIANT); + + auto find_child = [](const std::vector& children, + std::string_view name) -> const format::ColumnDefinition* { + const auto it = std::ranges::find_if( + children, [name](const auto& child) { return child.name == name; }); + return it == children.end() ? nullptr : &*it; + }; + const auto* root_typed = find_child(schema[1].children, "typed_value"); + ASSERT_NE(root_typed, nullptr); + const auto* n_wrapper = find_child(root_typed->children, "n"); + ASSERT_NE(n_wrapper, nullptr); + const auto* n_typed = find_child(n_wrapper->children, "typed_value"); + ASSERT_NE(n_typed, nullptr); + + auto projection = format::LocalColumnIndex::partial_local(schema[1].local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(root_typed->local_id)); + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(n_wrapper->local_id)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(n_typed->local_id)); + auto request = std::make_shared(); + request->non_predicate_columns.push_back(std::move(projection)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(0)); + ASSERT_TRUE(reader->open(request).ok()); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 1); + + Block block; + block.insert({schema[1].type->create_column(), schema[1].type, "v"}); + size_t rows = 0; + bool eof = false; + while (!eof) { + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + } + ASSERT_EQ(rows, 4096); + const auto& nullable = assert_cast(*block.get_by_position(0).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("n")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->type->get_primitive_type(), TYPE_INT); + EXPECT_EQ(match->column->size(), rows); + ASSERT_NE(profile.get_counter("VariantDirectLeafRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantDirectLeafRows")->value(), rows); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), 0); + const std::array missing_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("missing")}}; + EXPECT_FALSE(variants.find_shredded_typed_value(missing_path).has_value()); + ASSERT_NE(profile.get_counter("VariantDirectLeafPathMisses"), nullptr); + EXPECT_EQ(profile.get_counter("VariantDirectLeafPathMisses")->value(), 1); + const auto first_value = + assert_cast( + assert_cast(*match->column).get_nested_column()) + .get_data()[0]; + + IColumn::Filter keep(rows, 0); + keep[0] = 1; + const ColumnPtr filtered = variants.filter(keep, 1); + const auto& filtered_variants = assert_cast(*filtered); + ASSERT_TRUE(filtered_variants.is_shredded()); + const auto filtered_match = filtered_variants.find_shredded_typed_value(path); + ASSERT_TRUE(filtered_match.has_value()); + EXPECT_EQ(filtered_match->column->size(), 1); + EXPECT_EQ( + assert_cast( + assert_cast(*filtered_match->column).get_nested_column()) + .get_data()[0], + first_value); + EXPECT_TRUE(variants.clone_resized(0)->empty()); + auto mutable_filtered = variants.clone_resized(variants.size()); + EXPECT_EQ(mutable_filtered->filter(keep), 1); + EXPECT_TRUE(assert_cast(*mutable_filtered).is_shredded()); + + // TableReader detaches mapped output columns before upper expressions run. Detachment must + // preserve an incomplete leaf projection because it has no canonical Variant to materialize. + auto detached = IColumn::mutate(block.get_by_position(0).column); + const auto& detached_variants = assert_cast( + assert_cast(*detached).get_nested_column()); + ASSERT_TRUE(detached_variants.is_shredded()); + ASSERT_TRUE(detached_variants.find_shredded_typed_value(path).has_value()); + + // Adaptive predicate probing cuts retained output columns into proper subsets. Keep that row + // selection in the physical shredded state as well. + const ColumnPtr sliced = variants.cut(1, 2); + const auto& sliced_variants = assert_cast(*sliced); + ASSERT_TRUE(sliced_variants.is_shredded()); + const auto sliced_match = sliced_variants.find_shredded_typed_value(path); + ASSERT_TRUE(sliced_match.has_value()); + ASSERT_EQ(sliced_match->column->size(), 2); + EXPECT_EQ(assert_cast( + assert_cast(*sliced_match->column).get_nested_column()) + .get_data()[0], + first_value + 1); + + const std::array indices {2, 0}; + MutableColumnPtr gathered = variants.clone_empty(); + gathered->insert_indices_from(variants, indices.data(), indices.data() + indices.size()); + const auto& gathered_variants = assert_cast(*gathered); + ASSERT_TRUE(gathered_variants.is_shredded()); + const auto gathered_match = gathered_variants.find_shredded_typed_value(path); + ASSERT_TRUE(gathered_match.has_value()); + ASSERT_EQ(gathered_match->column->size(), indices.size()); + EXPECT_EQ( + assert_cast( + assert_cast(*gathered_match->column).get_nested_column()) + .get_data()[0], + first_value + 2); +} + +TEST_F(NewParquetReaderTest, ShreddedVariantPredicateUsesTypedLeafPageIndexWithRootOutput) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_page_pruning_with_root_output"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[0].local_id))); + // The root output deliberately retains the complete wrapper; its predicate may still use the + // typed leaf's page index without converting the output to a leaf-only Variant projection. + request->predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[1].local_id))); + request->local_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(0)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_variant_int32_path_greater_than_conjunct(1, "n", 3000)); + ASSERT_TRUE(reader->open(request).ok()); + + size_t rows = 0; + bool eof = false; + while (!eof) { + Block block = build_file_block(schema); + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + if (batch_rows > 0) { + const auto& nullable = + assert_cast(*block.get_by_position(1).column); + auto canonical = IColumn::mutate(nullable.get_nested_column_ptr()); + assert_cast(*canonical).ensure_encoded(); + } + } + EXPECT_EQ(rows, 1095); + ASSERT_NE(profile.get_counter("FilteredRowsByPage"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByPage")->value(), 0); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 0); + ASSERT_NE(profile.get_counter("VariantDirectLeafRows"), nullptr); + EXPECT_GT(profile.get_counter("VariantDirectLeafRows")->value(), 0); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), rows); + ASSERT_NE(profile.get_counter("VariantReconstructionTime"), nullptr); + EXPECT_GT(profile.get_counter("VariantReconstructionTime")->value(), 0); +} + +TEST_F(NewParquetReaderTest, ReadsVariantPredicateLeafBeforeDeferredRootOutput) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_predicate_leaf_deferred_root"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto find_child = [](const std::vector& children, + std::string_view name) -> const format::ColumnDefinition* { + const auto it = std::ranges::find_if( + children, [name](const auto& child) { return child.name == name; }); + return it == children.end() ? nullptr : &*it; + }; + const auto* root_typed = find_child(schema[1].children, "typed_value"); + ASSERT_NE(root_typed, nullptr); + const auto* n_wrapper = find_child(root_typed->children, "n"); + ASSERT_NE(n_wrapper, nullptr); + const auto* n_typed = find_child(n_wrapper->children, "typed_value"); + ASSERT_NE(n_typed, nullptr); + + auto predicate_projection = format::LocalColumnIndex::partial_local(schema[1].local_id); + predicate_projection.children.push_back( + format::LocalColumnIndex::partial_local(root_typed->local_id)); + predicate_projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(n_wrapper->local_id)); + predicate_projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(n_typed->local_id)); + + auto request = std::make_shared(); + request->predicate_columns.push_back(std::move(predicate_projection)); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[1].local_id))); + request->predicate_only_columns.push_back(format::LocalColumnId(schema[1].local_id)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(0)); + request->non_predicate_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_variant_int32_path_greater_than_conjunct(0, "n", 3000)); + ASSERT_TRUE(reader->open(request).ok()); + + size_t rows = 0; + bool eof = false; + while (!eof) { + Block block; + block.insert({schema[1].type->create_column(), schema[1].type, "v_predicate"}); + block.insert({schema[1].type->create_column(), schema[1].type, "v_output"}); + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + ASSERT_EQ(block.get_by_position(0).column->size(), batch_rows); + ASSERT_EQ(block.get_by_position(1).column->size(), batch_rows); + if (batch_rows > 0) { + const auto& nullable = + assert_cast(*block.get_by_position(1).column); + auto canonical = IColumn::mutate(nullable.get_nested_column_ptr()); + assert_cast(*canonical).ensure_encoded(); + } + } + EXPECT_EQ(rows, 1095); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 1); + ASSERT_NE(profile.get_counter("FilteredRowsByLazyRead"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByLazyRead")->value(), 0); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), rows); +} + +TEST_F(NewParquetReaderTest, ReadsStructPredicateChildBeforeDeferredRootOutput) { + write_struct_filter_parquet_file(_file_path); + RuntimeProfile profile("struct_predicate_child_deferred_root"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 1); + ASSERT_EQ(schema[0].children.size(), 2); + + auto predicate_projection = format::LocalColumnIndex::partial_local(schema[0].local_id); + predicate_projection.children.push_back( + format::LocalColumnIndex::local(schema[0].children[0].local_id)); + auto request = std::make_shared(); + request->predicate_columns.push_back(predicate_projection); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[0].local_id))); + request->predicate_only_columns.push_back(format::LocalColumnId(schema[0].local_id)); + request->local_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(0)); + request->non_predicate_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_struct_int32_child_greater_than_conjunct(0, 2)); + ASSERT_TRUE(reader->open(request).ok()); + + format::ColumnDefinition predicate_field; + ASSERT_TRUE(format::project_column_definition(schema[0], predicate_projection, &predicate_field) + .ok()); + size_t total_rows = 0; + std::vector names; + bool eof = false; + while (!eof) { + Block block; + block.insert({predicate_field.type->create_column(), predicate_field.type, "s_predicate"}); + block.insert({schema[0].type->create_column(), schema[0].type, "s_output"}); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + total_rows += rows; + ASSERT_EQ(block.get_by_position(0).column->size(), rows); + ASSERT_EQ(block.get_by_position(1).column->size(), rows); + const auto& output_nullable = + assert_cast(*block.get_by_position(1).column); + const auto& output_struct = + assert_cast(output_nullable.get_nested_column()); + ASSERT_EQ(output_struct.tuple_size(), 2); + const auto& name_nullable = assert_cast(output_struct.get_column(1)); + const auto& name_values = + assert_cast(name_nullable.get_nested_column()); + for (size_t row = 0; row < rows; ++row) { + names.push_back(name_values.get_data_at(row).to_string()); + } + } + EXPECT_EQ(total_rows, 2); + EXPECT_EQ(names, (std::vector {"ten", "eleven"})); + ASSERT_NE(profile.get_counter("FilteredRowsByLazyRead"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByLazyRead")->value(), 0); +} + TEST_F(NewParquetReaderTest, CountComplexColumnUsesShapeOnlyPath) { write_nullable_map_parquet_file(_file_path); RuntimeProfile profile("count_map_shape_only_path"); diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp b/be/test/format_v2/parquet/parquet_schema_test.cpp index 74781c0e532b31..9ecc1edf6fc582 100644 --- a/be/test/format_v2/parquet/parquet_schema_test.cpp +++ b/be/test/format_v2/parquet/parquet_schema_test.cpp @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -28,6 +29,7 @@ #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/data_type/primitive_type.h" #include "format_v2/parquet/native_schema_desc.h" #include "format_v2/parquet/native_schema_node.h" @@ -35,6 +37,401 @@ #include "format_v2/parquet/parquet_file_context.h" namespace doris::format::parquet { +namespace { + +std::vector unshredded_variant_schema( + std::optional specification_version = 1) { + tparquet::SchemaElement root; + root.__set_name("schema"); + root.__set_num_children(1); + + tparquet::SchemaElement variant; + variant.__set_name("payload"); + variant.__set_num_children(2); + variant.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + variant.__set_logicalType(tparquet::LogicalType()); + variant.logicalType.__set_VARIANT(tparquet::VariantType()); + if (specification_version.has_value()) { + variant.logicalType.VARIANT.__set_specification_version(*specification_version); + } + + tparquet::SchemaElement metadata; + metadata.__set_name("metadata"); + metadata.__set_type(tparquet::Type::BYTE_ARRAY); + metadata.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + + tparquet::SchemaElement value; + value.__set_name("value"); + value.__set_type(tparquet::Type::BYTE_ARRAY); + value.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + return {root, variant, metadata, value}; +} + +std::vector struct_with_variant_schema() { + auto variant_fields = unshredded_variant_schema(); + variant_fields[0].__set_name("info"); + variant_fields[0].__set_num_children(2); + variant_fields[0].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + variant_fields[1].__set_name("payload"); + + tparquet::SchemaElement root; + root.__set_name("schema"); + root.__set_num_children(1); + tparquet::SchemaElement label; + label.__set_name("label"); + label.__set_type(tparquet::Type::BYTE_ARRAY); + label.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + return {root, + variant_fields[0], + label, + variant_fields[1], + variant_fields[2], + variant_fields[3]}; +} + +std::vector shredded_object_variant_schema(bool signed_integer = true, + bool required_wrapper = true) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_object; + typed_object.__set_name("typed_value"); + typed_object.__set_num_children(1); + typed_object.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement field_wrapper; + field_wrapper.__set_name("field"); + field_wrapper.__set_num_children(2); + field_wrapper.__set_repetition_type(required_wrapper ? tparquet::FieldRepetitionType::REQUIRED + : tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement fallback; + fallback.__set_name("value"); + fallback.__set_type(tparquet::Type::BYTE_ARRAY); + fallback.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_integer; + typed_integer.__set_name("typed_value"); + typed_integer.__set_type(tparquet::Type::INT32); + typed_integer.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + if (!signed_integer) { + typed_integer.__set_logicalType(tparquet::LogicalType()); + typed_integer.logicalType.__set_INTEGER(tparquet::IntType()); + typed_integer.logicalType.INTEGER.__set_bitWidth(32); + typed_integer.logicalType.INTEGER.__set_isSigned(false); + } + schema.insert(schema.end(), {typed_object, field_wrapper, fallback, typed_integer}); + return schema; +} + +std::vector shredded_primitive_variant_schema( + tparquet::SchemaElement typed_value) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_name("typed_value"); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(typed_value)); + return schema; +} + +std::vector shredded_time_variant_schema(bool adjusted_to_utc, + bool millis) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(millis ? tparquet::Type::INT32 : tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIME(tparquet::TimeType()); + typed_value.logicalType.TIME.__set_isAdjustedToUTC(adjusted_to_utc); + typed_value.logicalType.TIME.__set_unit(tparquet::TimeUnit()); + if (millis) { + typed_value.logicalType.TIME.unit.__set_MILLIS(tparquet::MilliSeconds()); + } else { + typed_value.logicalType.TIME.unit.__set_MICROS(tparquet::MicroSeconds()); + } + schema.push_back(std::move(typed_value)); + return schema; +} + +std::vector shredded_array_variant_schema(bool include_value, + bool include_typed_value) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_array; + typed_array.__set_name("typed_value"); + typed_array.__set_num_children(1); + typed_array.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_array.__set_converted_type(tparquet::ConvertedType::LIST); + tparquet::SchemaElement list; + list.__set_name("list"); + list.__set_num_children(1); + list.__set_repetition_type(tparquet::FieldRepetitionType::REPEATED); + tparquet::SchemaElement element; + element.__set_name("element"); + element.__set_num_children(static_cast(include_value) + + static_cast(include_typed_value)); + element.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + schema.insert(schema.end(), {typed_array, list, element}); + if (include_value) { + tparquet::SchemaElement value; + value.__set_name("value"); + value.__set_type(tparquet::Type::BYTE_ARRAY); + value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(value)); + } + if (include_typed_value) { + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT32); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(typed_value)); + } + return schema; +} + +} // namespace + +TEST(ParquetSchemaTest, NativeSchemaRecognizesVariantLogicalGroup) { + for (const auto version : {std::optional {}, std::optional {1}}) { + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(unshredded_variant_schema(version)).ok()); + descriptor.assign_ids(); + + const auto* native_variant = descriptor.get_column(0); + ASSERT_NE(native_variant, nullptr); + // Native readers must keep seeing the physical group. The logical Variant mapping belongs + // to ParquetColumnSchema and must not make this group look like an unindexed scalar leaf. + EXPECT_EQ(remove_nullable(native_variant->data_type)->get_primitive_type(), TYPE_STRUCT); + ASSERT_EQ(native_variant->children.size(), 2); + EXPECT_EQ(native_variant->children[0].physical_column_index, 0); + EXPECT_EQ(native_variant->children[1].physical_column_index, 1); + EXPECT_EQ(descriptor.physical_fields_size(), 2); + + std::vector> fields; + const auto status = build_parquet_column_schema(descriptor, &fields); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(fields.size(), 1); + EXPECT_EQ(fields[0]->kind, ParquetColumnSchemaKind::VARIANT); + EXPECT_EQ(remove_nullable(fields[0]->type)->get_primitive_type(), TYPE_VARIANT); + EXPECT_NE(typeid_cast(remove_nullable(fields[0]->type).get()), + nullptr); + ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_EQ(fields[0]->children[0]->name, "metadata"); + EXPECT_EQ(fields[0]->children[1]->name, "value"); + } +} + +TEST(ParquetSchemaTest, NativeSchemaAcceptsRequiredAndOptionalVariantGroups) { + for (const auto repetition : + {tparquet::FieldRepetitionType::REQUIRED, tparquet::FieldRepetitionType::OPTIONAL}) { + auto schema = unshredded_variant_schema(); + schema[1].__set_repetition_type(repetition); + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + ASSERT_TRUE(status.ok()) << status; + ASSERT_NE(descriptor.get_column(0), nullptr); + } +} + +TEST(ParquetSchemaTest, NestedVariantPropagatesIntoParentLogicalType) { + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(struct_with_variant_schema()).ok()); + descriptor.assign_ids(); + + std::vector> fields; + const auto status = build_parquet_column_schema(descriptor, &fields); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(fields.size(), 1); + const auto* info_type = + assert_cast(remove_nullable(fields[0]->type).get()); + ASSERT_EQ(info_type->get_elements().size(), 2); + EXPECT_EQ(remove_nullable(info_type->get_elements()[1])->get_primitive_type(), TYPE_VARIANT); +} + +TEST(ParquetSchemaTest, NativeSchemaRejectsUnsupportedVariantVersionAndMalformedLayout) { + NativeFieldDescriptor descriptor; + const auto version_status = descriptor.parse_from_thrift(unshredded_variant_schema(2)); + EXPECT_TRUE(version_status.is()) << version_status; + EXPECT_NE(version_status.to_string().find("Variant specification version 2"), + std::string::npos); + + auto missing_metadata = unshredded_variant_schema(); + missing_metadata[2].__set_name("not_metadata"); + const auto layout_status = descriptor.parse_from_thrift(missing_metadata); + EXPECT_TRUE(layout_status.is()) << layout_status; + EXPECT_NE(layout_status.to_string().find("metadata"), std::string::npos); + + auto optional_unshredded_value = unshredded_variant_schema(); + optional_unshredded_value[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + const auto repetition_status = descriptor.parse_from_thrift(optional_unshredded_value); + EXPECT_TRUE(repetition_status.is()) << repetition_status; + EXPECT_NE(repetition_status.to_string().find("required BYTE_ARRAY"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantPreservesUtcTimestampInstant) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIMESTAMP(tparquet::TimestampType()); + typed_value.logicalType.TIMESTAMP.__set_isAdjustedToUTC(true); + typed_value.logicalType.TIMESTAMP.__set_unit(tparquet::TimeUnit()); + typed_value.logicalType.TIMESTAMP.unit.__set_MICROS(tparquet::MicroSeconds()); + schema.push_back(std::move(typed_value)); + + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(schema).ok()); + const auto* variant = descriptor.get_column(0); + ASSERT_EQ(variant->children.size(), 3); + EXPECT_EQ(remove_nullable(variant->children[2].data_type)->get_primitive_type(), + TYPE_TIMESTAMPTZ); +} + +TEST(ParquetSchemaTest, NativeVariantValidatesEveryShreddedWrapperAndScalar) { + NativeFieldDescriptor descriptor; + const auto unsigned_status = + descriptor.parse_from_thrift(shredded_object_variant_schema(false, true)); + EXPECT_TRUE(unsigned_status.is()) << unsigned_status; + EXPECT_NE(unsigned_status.to_string().find("unsigned"), std::string::npos); + + const auto optional_wrapper_status = + descriptor.parse_from_thrift(shredded_object_variant_schema(true, false)); + EXPECT_TRUE(optional_wrapper_status.is()) << optional_wrapper_status; + EXPECT_NE(optional_wrapper_status.to_string().find("wrapper"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsDuplicateObjectFieldNames) { + auto schema = shredded_object_variant_schema(); + schema[4].__set_num_children(2); + schema.insert(schema.end(), {schema[5], schema[6], schema[7]}); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("duplicate"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedPrimitiveTypePairs) { + std::vector invalid_typed_values; + + tparquet::SchemaElement int96; + int96.__set_type(tparquet::Type::INT96); + invalid_typed_values.push_back(int96); + + tparquet::SchemaElement fixed_binary; + fixed_binary.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + fixed_binary.__set_type_length(16); + invalid_typed_values.push_back(fixed_binary); + + tparquet::SchemaElement json; + json.__set_type(tparquet::Type::BYTE_ARRAY); + json.__set_logicalType(tparquet::LogicalType()); + json.logicalType.__set_JSON(tparquet::JsonType()); + invalid_typed_values.push_back(json); + + tparquet::SchemaElement float16; + float16.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + float16.__set_type_length(2); + float16.__set_logicalType(tparquet::LogicalType()); + float16.logicalType.__set_FLOAT16(tparquet::Float16Type()); + invalid_typed_values.push_back(float16); + + tparquet::SchemaElement mismatched_integer; + mismatched_integer.__set_type(tparquet::Type::INT64); + mismatched_integer.__set_logicalType(tparquet::LogicalType()); + mismatched_integer.logicalType.__set_INTEGER(tparquet::IntType()); + mismatched_integer.logicalType.INTEGER.__set_bitWidth(16); + mismatched_integer.logicalType.INTEGER.__set_isSigned(true); + invalid_typed_values.push_back(mismatched_integer); + + tparquet::SchemaElement mismatched_decimal; + mismatched_decimal.__set_type(tparquet::Type::INT32); + mismatched_decimal.__set_logicalType(tparquet::LogicalType()); + mismatched_decimal.logicalType.__set_DECIMAL(tparquet::DecimalType()); + mismatched_decimal.logicalType.DECIMAL.__set_precision(10); + mismatched_decimal.logicalType.DECIMAL.__set_scale(2); + invalid_typed_values.push_back(mismatched_decimal); + + tparquet::SchemaElement bad_uuid; + bad_uuid.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + bad_uuid.__set_type_length(15); + bad_uuid.__set_logicalType(tparquet::LogicalType()); + bad_uuid.logicalType.__set_UUID(tparquet::UUIDType()); + invalid_typed_values.push_back(bad_uuid); + + for (auto& typed_value : invalid_typed_values) { + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift( + shredded_primitive_variant_schema(std::move(typed_value))); + EXPECT_FALSE(status.ok()) << "unsupported Variant typed_value pair was accepted"; + } +} + +TEST(ParquetSchemaTest, NativeVariantRejectsRepeatedOuterGroup) { + auto schema = unshredded_variant_schema(); + schema[1].__set_repetition_type(tparquet::FieldRepetitionType::REPEATED); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("repeated"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantAcceptsOmittedShreddedWrapperChildren) { + NativeFieldDescriptor descriptor; + + auto value_only_object = shredded_object_variant_schema(); + value_only_object[5].__set_num_children(1); + value_only_object.pop_back(); + ASSERT_TRUE(descriptor.parse_from_thrift(value_only_object).ok()); + + ASSERT_TRUE(descriptor.parse_from_thrift(shredded_array_variant_schema(true, false)).ok()); + ASSERT_TRUE(descriptor.parse_from_thrift(shredded_array_variant_schema(false, true)).ok()); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsNanosBeforeProjectionChoice) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIMESTAMP(tparquet::TimestampType()); + typed_value.logicalType.TIMESTAMP.__set_isAdjustedToUTC(false); + typed_value.logicalType.TIMESTAMP.__set_unit(tparquet::TimeUnit()); + typed_value.logicalType.TIMESTAMP.unit.__set_NANOS(tparquet::NanoSeconds()); + schema.push_back(std::move(typed_value)); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("TIMESTAMP(NANOS)"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedTimeAnnotations) { + NativeFieldDescriptor descriptor; + const auto adjusted_status = + descriptor.parse_from_thrift(shredded_time_variant_schema(true, false)); + EXPECT_TRUE(adjusted_status.is()) << adjusted_status; + EXPECT_NE(adjusted_status.to_string().find("isAdjustedToUTC"), std::string::npos); + + const auto millis_status = + descriptor.parse_from_thrift(shredded_time_variant_schema(false, true)); + EXPECT_TRUE(millis_status.is()) << millis_status; + EXPECT_NE(millis_status.to_string().find("TIME(MILLIS)"), std::string::npos); +} + TEST(ParquetSchemaTest, NativeMetadataAcceptsRequiredRootWithoutColumns) { tparquet::SchemaElement root; root.__set_name("schema"); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 740892367bb04e..1a13e8ecea362e 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -37,10 +37,12 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_time.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" #include "exprs/vslot_ref.h" #include "format_v2/file_reader.h" #include "format_v2/parquet/parquet_column_schema.h" @@ -196,6 +198,107 @@ class MetadataBoundsProbeExpr final : public VExpr { bool _require_false_boolean; const std::string _expr_name = "MetadataBoundsProbeExpr"; }; + +class VariantPathTestExpr final : public VExpr { +public: + VariantPathTestExpr(std::string name, DataTypePtr type, + TExprNodeType::type node_type = TExprNodeType::FUNCTION_CALL) + : VExpr(std::move(type), false), _name(std::move(name)) { + set_node_type(node_type); + } + + const std::string& expr_name() const override { return _name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("VariantPathTestExpr is metadata-only"); + } + +private: + std::string _name; +}; + +VExprContextSPtr variant_path_gt_conjunct(int32_t literal_value, + bool add_narrowing_intermediate_cast = false, + bool decimal_comparison = false) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + DataTypePtr comparison_type = decimal_comparison + ? DataTypePtr(std::make_shared(38, 9)) + : DataTypePtr(std::make_shared()); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + if (add_narrowing_intermediate_cast) { + auto narrowing = std::make_shared( + "CAST", make_nullable(std::make_shared()), TExprNodeType::CAST_EXPR); + narrowing->add_child(element_at); + cast->add_child(narrowing); + } else { + cast->add_child(element_at); + } + auto literal = decimal_comparison + ? VLiteral::create_shared( + comparison_type, + Field::create_field(Decimal128V3( + static_cast<__int128>(literal_value) * 1'000'000'000))) + : VLiteral::create_shared(comparison_type, + Field::create_field(literal_value)); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +VExprContextSPtr variant_path_float_gt_conjunct(float literal_value) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + auto comparison_type = std::make_shared(); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared(comparison_type, + Field::create_field(literal_value)); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +VExprContextSPtr variant_path_string_gt_conjunct(std::string literal_value) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + auto comparison_type = std::make_shared(); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared( + comparison_type, Field::create_field(std::move(literal_value))); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + VExprContextSPtrs bloom_conjuncts(DataTypePtr data_type, std::vector values) { return {VExprContext::create_shared( std::make_shared(0, std::move(data_type), std::move(values)))}; @@ -331,8 +434,9 @@ TEST(NativeParquetStatisticsTest, InvalidTimeAndPaddedBooleanPageBoundsCannotPru page_indexes.emplace(0, std::move(page_index)); std::vector selected_ranges; std::map skip_plans; + tparquet::RowGroup row_group; EXPECT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, + metadata, row_group, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, nullptr) .ok()); return selected_ranges; @@ -650,8 +754,8 @@ TEST(NativeParquetStatisticsTest, TypeDefinedBoundsRequireSupportedColumnOrder) std::vector selected_ranges; std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr) .ok()); EXPECT_EQ(selected_ranges.size(), 1); @@ -664,8 +768,8 @@ TEST(NativeParquetStatisticsTest, TypeDefinedBoundsRequireSupportedColumnOrder) .ok()); EXPECT_TRUE(selected_row_groups.empty()); ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr) .ok()); EXPECT_TRUE(selected_ranges.empty()); } @@ -737,8 +841,8 @@ TEST(NativeParquetStatisticsTest, ZonemapPruningIgnoresDisabledSessionSwitch) { std::vector selected_ranges; std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr, nullptr, &state) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr, nullptr, &state) .ok()); EXPECT_TRUE(selected_ranges.empty()); } @@ -780,8 +884,8 @@ TEST(NativeParquetStatisticsTest, ContradictoryAllNullPageCountsDisablePruning) std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 10, &selected_ranges, - &skip_plans, nullptr) + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 10, + &selected_ranges, &skip_plans, nullptr) .ok()); // ColumnIndex is optional. An impossible all-null claim must fall back to reading the // ten-row data page instead of proving that no value can satisfy the predicate. @@ -791,5 +895,320 @@ TEST(NativeParquetStatisticsTest, ContradictoryAllNullPageCountsDisablePruning) } } +TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) { + auto encode_int32 = [](int32_t value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto primitive = [](std::string name, int local_id, int leaf_id) { + auto schema = std::make_unique(); + schema->name = std::move(name); + schema->local_id = local_id; + schema->leaf_column_id = leaf_id; + schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + schema->type = make_nullable(std::make_shared()); + schema->type_descriptor.doris_type = schema->type; + schema->type_descriptor.physical_type = tparquet::Type::INT32; + return schema; + }; + auto bytes = [&](std::string name, int local_id, int leaf_id) { + auto schema = primitive(std::move(name), local_id, leaf_id); + schema->type = make_nullable(std::make_shared()); + schema->type_descriptor.doris_type = schema->type; + schema->type_descriptor.physical_type = tparquet::Type::BYTE_ARRAY; + return schema; + }; + + auto variant = std::make_unique(); + variant->name = "v"; + variant->local_id = 0; + variant->kind = format::parquet::ParquetColumnSchemaKind::VARIANT; + variant->type = make_nullable(std::make_shared()); + variant->children.push_back(bytes("metadata", 0, 0)); + variant->children.push_back(bytes("value", 1, 1)); + auto typed_object = std::make_unique(); + typed_object->name = "typed_value"; + typed_object->local_id = 2; + typed_object->kind = format::parquet::ParquetColumnSchemaKind::STRUCT; + auto field = std::make_unique(); + field->name = "col"; + field->local_id = 0; + field->kind = format::parquet::ParquetColumnSchemaKind::STRUCT; + field->children.push_back(bytes("value", 0, 2)); + field->children.push_back(primitive("typed_value", 1, 3)); + typed_object->children.push_back(std::move(field)); + variant->children.push_back(std::move(typed_object)); + std::vector> schema; + schema.push_back(std::move(variant)); + + auto chunk = [&](tparquet::Type::type type, int64_t num_values, int64_t null_count, + std::optional min_value = std::nullopt, + std::optional max_value = std::nullopt) { + tparquet::Statistics statistics; + statistics.__set_null_count(null_count); + if (min_value.has_value() && max_value.has_value()) { + statistics.__set_min_value(encode_int32(*min_value)); + statistics.__set_max_value(encode_int32(*max_value)); + } + tparquet::ColumnMetaData metadata; + metadata.__set_type(type); + metadata.__set_num_values(num_values); + metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk result; + result.__set_meta_data(std::move(metadata)); + return result; + }; + tparquet::RowGroup row_group; + row_group.__set_num_rows(100); + row_group.__set_columns({chunk(tparquet::Type::BYTE_ARRAY, 100, 0), + chunk(tparquet::Type::BYTE_ARRAY, 100, 100), + chunk(tparquet::Type::BYTE_ARRAY, 100, 100), + chunk(tparquet::Type::INT32, 100, 0, 1, 200)}); + tparquet::ColumnOrder order; + order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder()); + tparquet::FileMetaData metadata; + metadata.__set_row_groups({row_group}); + metadata.__set_column_orders({order, order, order, order}); + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.predicate_columns = {format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + request.conjuncts = {variant_path_gt_conjunct(50)}; + + auto footer_only_metadata = metadata; + footer_only_metadata.row_groups[0].columns[3].meta_data.statistics.__set_max_value( + encode_int32(2)); + std::vector selected_row_groups; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + + auto leaf_projection = format::LocalColumnIndex::partial_local(0); + auto typed_object_projection = format::LocalColumnIndex::partial_local(2); + auto field_projection = format::LocalColumnIndex::partial_local(0); + field_projection.children.push_back(format::LocalColumnIndex::local(1)); + typed_object_projection.children.push_back(std::move(field_projection)); + leaf_projection.children.push_back(std::move(typed_object_projection)); + request.predicate_columns = {std::move(leaf_projection)}; + for (int leaf = 0; leaf < 4; ++leaf) { + footer_only_metadata.row_groups[0].columns[leaf].meta_data.__set_total_compressed_size( + (leaf + 1) * 10); + } + format::parquet::ParquetPruningStats leaf_pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + &leaf_pruning_stats, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + EXPECT_EQ(leaf_pruning_stats.filtered_bytes, 40); + + request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + request.conjuncts = {variant_path_gt_conjunct(50)}; + + // Missing typed statistics provide no proof and must retain the row group. + footer_only_metadata.row_groups[0].columns[3].meta_data.__isset.statistics = false; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + // A populated fallback for the same path invalidates both footer and page pruning. + footer_only_metadata = metadata; + footer_only_metadata.row_groups[0].columns[3].meta_data.statistics.__set_max_value( + encode_int32(2)); + footer_only_metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + // A contradictory non-repeated value count cannot prove that every row lacks fallback bytes. + footer_only_metadata.row_groups[0].columns[2].meta_data.__set_num_values(99); + footer_only_metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex typed_pages; + typed_pages.column_index.__set_min_values({encode_int32(1), encode_int32(100)}); + typed_pages.column_index.__set_max_values({encode_int32(2), encode_int32(200)}); + typed_pages.column_index.__set_null_pages({false, false}); + typed_pages.column_index.__set_null_counts({0, 0}); + tparquet::PageLocation first; + first.__set_offset(0); + first.__set_compressed_page_size(10); + first.__set_first_row_index(0); + tparquet::PageLocation second; + second.__set_offset(10); + second.__set_compressed_page_size(10); + second.__set_first_row_index(50); + typed_pages.offset_index.__set_page_locations({first, second}); + std::unordered_map page_indexes; + page_indexes.emplace(3, std::move(typed_pages)); + + std::vector selected_ranges; + std::map skip_plans; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::can_use_parquet_page_index(request, nullptr)); + TQueryOptions query_options; + query_options.__set_enable_expr_zonemap_filter(false); + RuntimeState generic_zonemap_disabled {query_options, TQueryGlobals()}; + EXPECT_TRUE(format::parquet::can_use_parquet_page_index(request, &generic_zonemap_disabled)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, &pruning_stats, nullptr, + &generic_zonemap_disabled) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 50); + EXPECT_EQ(selected_ranges[0].length, 50); + EXPECT_EQ(pruning_stats.page_index_read_calls, 1); + EXPECT_EQ(pruning_stats.filtered_page_rows, 50); + + // Direct Variant numeric comparisons coerce integral literals to a wide DECIMAL domain. + request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 50); + EXPECT_EQ(selected_ranges[0].length, 50); + + // Bounds for the raw INT32 typed leaf are not valid for CAST(CAST(v['col'] AS TINYINT) AS INT). + request.conjuncts = {variant_path_gt_conjunct(50, true)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + + // Raw binary and UUID bounds are physical bytes, while the residual Variant-to-STRING cast + // compares their rendered values. Those domains differ, so neither footer nor page metadata + // may exclude a row solely from the raw byte interval. + auto assert_binary_identity_does_not_prune = [&](bool is_uuid) { + auto* binary_leaf = schema[0]->children[2]->children[0]->children[1].get(); + binary_leaf->type = make_nullable(std::make_shared()); + binary_leaf->type_descriptor = {}; + binary_leaf->type_descriptor.doris_type = binary_leaf->type; + binary_leaf->type_descriptor.physical_type = + is_uuid ? tparquet::Type::FIXED_LEN_BYTE_ARRAY : tparquet::Type::BYTE_ARRAY; + binary_leaf->type_descriptor.fixed_length = is_uuid ? 16 : -1; + binary_leaf->type_descriptor.is_string_like = true; + binary_leaf->type_descriptor.is_uuid = is_uuid; + + auto& binary_chunk = metadata.row_groups[0].columns[3].meta_data; + binary_chunk.__set_type(binary_leaf->type_descriptor.physical_type); + binary_chunk.statistics.__set_min_value("a"); + binary_chunk.statistics.__set_max_value("b"); + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(100); + request.conjuncts = {variant_path_string_gt_conjunct("z")}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex binary_pages; + binary_pages.column_index.__set_min_values({"a"}); + binary_pages.column_index.__set_max_values({"b"}); + binary_pages.column_index.__set_null_pages({false}); + binary_pages.column_index.__set_null_counts({0}); + tparquet::PageLocation binary_location; + binary_location.__set_offset(0); + binary_location.__set_compressed_page_size(10); + binary_location.__set_first_row_index(0); + binary_pages.offset_index.__set_page_locations({binary_location}); + page_indexes.clear(); + page_indexes.emplace(3, std::move(binary_pages)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + }; + assert_binary_identity_does_not_prune(false); + assert_binary_identity_does_not_prune(true); + + // Parquet floating min/max omits NaN values. Without an explicit no-NaN proof, [0, NaN] + // cannot be represented by max=0 and must not prune a Variant comparison that retains NaN. + auto encode_float = [](float value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto* float_leaf = schema[0]->children[2]->children[0]->children[1].get(); + float_leaf->type = make_nullable(std::make_shared()); + float_leaf->type_descriptor.doris_type = float_leaf->type; + float_leaf->type_descriptor.physical_type = tparquet::Type::FLOAT; + auto& float_chunk = metadata.row_groups[0].columns[3].meta_data; + float_chunk.__set_type(tparquet::Type::FLOAT); + float_chunk.statistics.__set_min_value(encode_float(0.0F)); + float_chunk.statistics.__set_max_value(encode_float(0.0F)); + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(100); + request.conjuncts = {variant_path_float_gt_conjunct(1.0F)}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, false, nullptr, + nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex float_pages; + float_pages.column_index.__set_min_values({encode_float(0.0F)}); + float_pages.column_index.__set_max_values({encode_float(0.0F)}); + float_pages.column_index.__set_null_pages({false}); + float_pages.column_index.__set_null_counts({0}); + tparquet::PageLocation float_location; + float_location.__set_offset(0); + float_location.__set_compressed_page_size(10); + float_location.__set_first_row_index(0); + float_pages.offset_index.__set_page_locations({float_location}); + page_indexes.clear(); + page_indexes.emplace(3, std::move(float_pages)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + + // A fallback value in the same row group may have a different Variant type. In that case the + // typed bounds cannot prove anything about the SQL comparison, so all pages must be read. + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + request.conjuncts = {variant_path_gt_conjunct(50)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); +} + } // namespace } // namespace doris diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp new file mode 100644 index 00000000000000..40c18635aad339 --- /dev/null +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -0,0 +1,927 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/parquet/reader/variant_column_reader.h" + +#include + +#include +#include +#include + +#include "core/assert_cast.h" +#include "core/column/column_array.h" +#include "core/column/column_nullable.h" +#include "core/column/column_string.h" +#include "core/column/column_struct.h" +#include "core/column/variant_v2/column_variant_v2.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_number.h" +#include "core/data_type/data_type_string.h" +#include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" +#include "core/value/variant/variant_batch_builder.h" +#include "core/value/variant/variant_parquet_encoding.h" +#include "exprs/function/function_variant_element_v2.h" +#include "format_v2/parquet/parquet_column_schema.h" + +namespace doris::format::parquet { +namespace { + +MutableColumnPtr nullable_strings(const std::vector& values, + const std::vector& nulls) { + auto data = ColumnString::create(); + auto null_map = ColumnUInt8::create(); + for (size_t row = 0; row < values.size(); ++row) { + data->insert_data(values[row].data, values[row].size); + null_map->get_data().push_back(nulls[row]); + } + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +ParquetColumnSchema unshredded_schema() { + ParquetColumnSchema schema; + schema.name = "payload"; + schema.kind = ParquetColumnSchemaKind::VARIANT; + schema.type = make_nullable(std::make_shared()); + const auto binary = make_nullable(std::make_shared()); + schema.variant_physical_type = make_nullable(std::make_shared( + DataTypes {binary, binary}, Strings {"metadata", "value"})); + + auto metadata = std::make_unique(); + metadata->name = "metadata"; + metadata->kind = ParquetColumnSchemaKind::PRIMITIVE; + metadata->type = binary; + auto value = std::make_unique(); + value->name = "value"; + value->kind = ParquetColumnSchemaKind::PRIMITIVE; + value->type = binary; + schema.children.push_back(std::move(metadata)); + schema.children.push_back(std::move(value)); + return schema; +} + +ParquetColumnSchema shredded_int64_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + typed->type = make_nullable(std::make_shared()); + typed->type_descriptor.integer_bit_width = 64; + schema.children.push_back(std::move(typed)); + const auto binary = make_nullable(std::make_shared()); + schema.variant_physical_type = make_nullable(std::make_shared( + DataTypes {binary, binary, make_nullable(std::make_shared())}, + Strings {"metadata", "value", "typed_value"})); + return schema; +} + +ParquetColumnSchema shredded_object_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + + auto field = std::make_unique(); + field->name = "a"; + field->kind = ParquetColumnSchemaKind::STRUCT; + auto field_typed = std::make_unique(); + field_typed->name = "typed_value"; + field_typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + field_typed->type = make_nullable(std::make_shared()); + field_typed->type_descriptor.integer_bit_width = 64; + field->children.push_back(std::move(field_typed)); + typed->children.push_back(std::move(field)); + schema.children.push_back(std::move(typed)); + return schema; +} + +ParquetColumnSchema shredded_binary_object_schema() { + auto schema = shredded_object_schema(); + auto* leaf = schema.children.back()->children[0]->children[0].get(); + leaf->type = make_nullable(std::make_shared()); + return schema; +} + +ParquetColumnSchema shredded_array_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::LIST; + auto element = std::make_unique(); + element->name = "element"; + element->kind = ParquetColumnSchemaKind::STRUCT; + auto element_typed = std::make_unique(); + element_typed->name = "typed_value"; + element_typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + element_typed->type = make_nullable(std::make_shared()); + element_typed->type_descriptor.integer_bit_width = 64; + element->children.push_back(std::move(element_typed)); + typed->children.push_back(std::move(element)); + schema.children.push_back(std::move(typed)); + return schema; +} + +MutableColumnPtr shredded_int64_physical(const std::vector& values) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + std::vector metadata_rows(values.size(), metadata); + std::vector empty_values(values.size(), {ignored.data(), 0}); + std::vector present(values.size(), 0); + std::vector absent(values.size(), 1); + MutableColumns fields; + fields.push_back(nullable_strings(metadata_rows, present)); + fields.push_back(nullable_strings(empty_values, absent)); + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().resize_fill(values.size(), 0); + fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().resize_fill(values.size(), 0); + return ColumnNullable::create(std::move(structure), std::move(root_nulls)); +} + +MutableColumnPtr projected_shredded_object_physical(const std::vector& values, + const IColumn** decoded_leaf = nullptr) { + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().resize_fill(values.size(), 0); + MutableColumnPtr leaf = ColumnNullable::create(std::move(integers), std::move(integer_nulls)); + if (decoded_leaf != nullptr) { + *decoded_leaf = leaf.get(); + } + + MutableColumns wrapper_fields; + wrapper_fields.push_back(std::move(leaf)); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back( + ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); +} + +MutableColumnPtr projected_two_field_object_physical(const std::vector& first, + const std::vector& second) { + DORIS_CHECK(first.size() == second.size()); + auto wrapper = [](const std::vector& values) { + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto leaf = + ColumnNullable::create(std::move(integers), ColumnUInt8::create(values.size(), 0)); + MutableColumns fields; + fields.push_back(std::move(leaf)); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), + ColumnUInt8::create(values.size(), 0)); + }; + + MutableColumns object_fields; + object_fields.push_back(wrapper(first)); + object_fields.push_back(wrapper(second)); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(first.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(first.size(), 0)); +} + +MutableColumnPtr projected_wide_object_physical(size_t field_count, int64_t value) { + MutableColumns object_fields; + object_fields.reserve(field_count); + for (size_t field = 0; field < field_count; ++field) { + auto integers = ColumnInt64::create(); + integers->insert_value(value + field); + MutableColumns wrapper_fields; + wrapper_fields.push_back( + ColumnNullable::create(std::move(integers), ColumnUInt8::create(1, 0))); + object_fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(wrapper_fields)), ColumnUInt8::create(1, 0))); + } + MutableColumns root_fields; + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return ColumnNullable::create(ColumnStruct::create(std::move(root_fields)), + ColumnUInt8::create(1, 0)); +} + +} // namespace + +TEST(VariantColumnReaderTest, UnshreddedRowsPreserveSqlNullAndVariantNull) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata, metadata, metadata}, {0, 0, 0})); + fields.push_back(nullable_strings( + {{int_seven.data(), int_seven.size()}, {ignored.data(), 0}, {ignored.data(), 0}}, + {0, 1, 1})); + auto physical_struct = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().assign({0, 1, 0}); + auto physical = ColumnNullable::create(std::move(physical_struct), std::move(root_nulls)); + + auto output_type = make_nullable(std::make_shared()); + auto output = output_type->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(output->size(), 3); + + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0, 1, 0})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_TRUE(variants.is_shredded()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); + EXPECT_TRUE(variants.get_value_ref(2).is_null()); +} + +TEST(VariantColumnReaderTest, RequiredPhysicalGroupAppendsToNullableExternalSlot) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical = ColumnStruct::create(std::move(fields)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, ShreddedIntegerKeepsDeclaredPhysicalWidth) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + auto integers = ColumnInt64::create(); + integers->get_data().push_back(42); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(structure), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_int64_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 42); + EXPECT_EQ(variants.get_value_ref(0).primitive_id(), VariantPrimitiveId::INT64); +} + +TEST(VariantColumnReaderTest, DifferentMetadataDictionariesRemainIndependent) { + VariantBatchBuilder first_builder; + auto first_row = first_builder.begin_row(); + auto first_object = first_row.start_object(); + first_object.add_key(StringRef("alpha")); + first_row.add_int(1); + first_object.finish(); + first_row.finish(); + auto first = first_builder.finish_batch(); + + VariantBatchBuilder second_builder; + auto second_row = second_builder.begin_row(); + auto second_object = second_row.start_object(); + second_object.add_key(StringRef("beta")); + second_row.add_int(2); + second_object.finish(); + second_row.finish(); + auto second = second_builder.finish_batch(); + + const VariantRef first_value = first.value_at(0); + const VariantRef second_value = second.value_at(0); + MutableColumns fields; + fields.push_back(nullable_strings({{first_value.metadata.data, first_value.metadata.size}, + {second_value.metadata.data, second_value.metadata.size}}, + {0, 0})); + fields.push_back(nullable_strings({{first_value.value.data, first_value.value.size}, + {second_value.value.data, second_value.value.size}}, + {0, 0})); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().assign({0, 0}); + auto physical = ColumnNullable::create(std::move(structure), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("alpha"), &field)); + EXPECT_EQ(field.get_int(), 1); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("beta"), &field)); + EXPECT_EQ(field.get_int(), 2); +} + +TEST(VariantColumnReaderTest, ShreddedObjectFieldMayOmitResidualValueColumn) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integer = ColumnInt64::create(); + integer->get_data().push_back(9); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integer), std::move(integer_nulls))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(0); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_int(), 9); + EXPECT_EQ(field.primitive_id(), VariantPrimitiveId::INT64); +} + +TEST(VariantColumnReaderTest, ShreddedTypedPathReusesDecodedLeafColumn) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integers = ColumnInt64::create(); + integers->get_data().push_back(9); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + MutableColumnPtr typed_leaf = + ColumnNullable::create(std::move(integers), std::move(integer_nulls)); + const IColumn* const decoded_typed_leaf = typed_leaf.get(); + + MutableColumns wrapper_fields; + wrapper_fields.push_back(std::move(typed_leaf)); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(0); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + ASSERT_TRUE(variants.is_shredded()); + + const std::array shredded_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(shredded_path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->column.get(), decoded_typed_leaf); + + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + ColumnPtr extracted; + ASSERT_TRUE( + extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) + .ok()); + + const auto& extracted_nullable = assert_cast(*extracted); + const auto& extracted_variant = + assert_cast(extracted_nullable.get_nested_column()); + ASSERT_TRUE(extracted_variant.is_typed()); + EXPECT_EQ(&extracted_variant.typed_column(), decoded_typed_leaf); + const auto& extracted_typed = + assert_cast(extracted_variant.typed_column()); + EXPECT_EQ(assert_cast(extracted_typed.get_nested_column()).get_data()[0], + 9); + EXPECT_TRUE(variants.is_shredded()); +} + +TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializing) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back( + format::LocalColumnIndex::partial_local(schema.children[2]->local_id)); + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(schema.children[2]->children[0]->local_id)); + projection.children.back().children.back().children.push_back(format::LocalColumnIndex::local( + schema.children[2]->children[0]->children[0]->local_id)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + ASSERT_EQ(plan.variant_state_schema.use_count(), 1); + + auto output = make_nullable(std::make_shared())->create_column(); + const IColumn* first_decoded_leaf = nullptr; + ASSERT_TRUE( + materialize_variant_columns( + plan, projected_shredded_object_physical({10, 20}, &first_decoded_leaf), output) + .ok()); + EXPECT_EQ(plan.variant_state_schema.use_count(), 2); + const auto append_status = + materialize_variant_columns(plan, projected_shredded_object_physical({30}), output); + ASSERT_TRUE(append_status.ok()) << append_status; + + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_TRUE(variants.is_shredded()); + ASSERT_EQ(variants.size(), 3); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->column.get(), first_decoded_leaf); + const auto& values = assert_cast( + assert_cast(*match->column).get_nested_column()); + EXPECT_EQ(values.get_data(), ColumnInt64::Container({10, 20, 30})); + + IColumn::Filter filter {1, 0, 1}; + ColumnPtr filtered = output->filter(filter, 2); + EXPECT_EQ(filtered->size(), 2); + EXPECT_EQ(plan.variant_state_schema.use_count(), 3); +} + +TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { + constexpr size_t width = 64; + constexpr size_t batch_count = 16; + auto schema = unshredded_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + typed->local_id = 2; + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(typed->local_id)); + for (size_t field = 0; field < width; ++field) { + auto wrapper = std::make_unique(); + wrapper->name = "field_" + std::to_string(field); + wrapper->kind = ParquetColumnSchemaKind::STRUCT; + wrapper->local_id = static_cast(field); + auto leaf = std::make_unique(); + leaf->name = "typed_value"; + leaf->kind = ParquetColumnSchemaKind::PRIMITIVE; + leaf->local_id = 0; + leaf->type = make_nullable(std::make_shared()); + leaf->type_descriptor.integer_bit_width = 64; + wrapper->children.push_back(std::move(leaf)); + typed->children.push_back(std::move(wrapper)); + + auto wrapper_projection = format::LocalColumnIndex::partial_local(static_cast(field)); + wrapper_projection.children.push_back(format::LocalColumnIndex::local(0)); + projection.children.back().children.push_back(std::move(wrapper_projection)); + } + schema.children.push_back(std::move(typed)); + + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto output = make_nullable(std::make_shared())->create_column(); + for (size_t batch = 0; batch < batch_count; ++batch) { + ASSERT_TRUE(materialize_variant_columns( + plan, projected_wide_object_physical(width, batch * width), output) + .ok()); + } + ASSERT_EQ(output->size(), batch_count); + ASSERT_EQ(plan.variant_state_schema.use_count(), 2); + + // Holding derived slices makes schema ownership observable: every state must retain the same + // reader-scoped schema instead of allocating a width-sized clone for each row selection. + std::vector slices; + slices.reserve(batch_count); + for (size_t row = 0; row < batch_count; ++row) { + slices.push_back(output->cut(row, 1)); + } + EXPECT_EQ(plan.variant_state_schema.use_count(), 2 + batch_count); +} + +TEST(VariantColumnReaderTest, RetainedSchemaFollowsDecodedProjectionOrder) { + auto schema = unshredded_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + typed->local_id = 2; + auto make_wrapper = [](std::string name, int local_id) { + auto wrapper = std::make_unique(); + wrapper->name = std::move(name); + wrapper->kind = ParquetColumnSchemaKind::STRUCT; + wrapper->local_id = local_id; + auto leaf = std::make_unique(); + leaf->name = "typed_value"; + leaf->kind = ParquetColumnSchemaKind::PRIMITIVE; + leaf->local_id = 0; + leaf->type = make_nullable(std::make_shared()); + leaf->type_descriptor.integer_bit_width = 64; + wrapper->children.push_back(std::move(leaf)); + return wrapper; + }; + typed->children.push_back(make_wrapper("z", 0)); + typed->children.push_back(make_wrapper("a", 1)); + schema.children.push_back(std::move(typed)); + + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + for (int local_id : {1, 0}) { + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(local_id)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + } + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_two_field_object_physical({11}, {22}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const std::array a_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const std::array z_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("z")}}; + const auto a = variants.find_shredded_typed_value(a_path); + const auto z = variants.find_shredded_typed_value(z_path); + ASSERT_TRUE(a.has_value()); + ASSERT_TRUE(z.has_value()); + EXPECT_EQ(assert_cast( + assert_cast(*a->column).get_nested_column()) + .get_data()[0], + 11); + EXPECT_EQ(assert_cast( + assert_cast(*z->column).get_nested_column()) + .get_data()[0], + 22); +} + +TEST(VariantColumnReaderTest, AmbiguousTypedIdentityRequiresCanonicalMaterialization) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_strings({StringRef("abc")}, {0})); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(1, 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), ColumnUInt8::create(1, 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto physical = ColumnNullable::create(std::move(root), ColumnUInt8::create(1, 0)); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_binary_object_schema(), *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + EXPECT_FALSE(variants.find_shredded_typed_value(path).has_value()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_binary(), StringRef("abc")); +} + +TEST(VariantColumnReaderTest, MaterializedCacheParticipatesInMemoryAccounting) { + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({42, 43}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const size_t physical_bytes = variants.byte_size(); + const size_t physical_allocated = variants.allocated_bytes(); + + EXPECT_EQ(variants.get_value_ref(0).get_int(), 42); + EXPECT_GT(variants.byte_size(), physical_bytes); + EXPECT_GT(variants.allocated_bytes(), physical_allocated); +} + +TEST(VariantColumnReaderTest, MaterializedShreddedCopiesDetachBeforeMutation) { + auto first_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({10, 20}), + first_output) + .ok()); + const auto& first = assert_cast( + assert_cast(*first_output).get_nested_column()); + EXPECT_EQ(first.get_value_ref(0).get_int(), 10); + + auto cloned = first.clone_resized(first.size()); + EXPECT_NO_THROW(cloned->pop_back(1)); + ASSERT_EQ(cloned->size(), 1); + EXPECT_EQ(assert_cast(*cloned).get_value_ref(0).get_int(), 10); + + auto second_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({30}), + second_output) + .ok()); + const auto& second = assert_cast( + assert_cast(*second_output).get_nested_column()); + auto appended = ColumnVariantV2::create(); + appended->insert_range_from(first, 0, first.size()); + EXPECT_NO_THROW(appended->insert_range_from(second, 0, second.size())); + ASSERT_EQ(appended->size(), 3); + EXPECT_EQ(appended->get_value_ref(2).get_int(), 30); +} + +TEST(VariantColumnReaderTest, MissingShreddedObjectWrapperMeansAbsentField) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integer = ColumnInt64::create(); + integer->get_data().push_back(0); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(1); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integer), std::move(integer_nulls))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(1); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).num_elements(), 0); +} + +TEST(VariantColumnReaderTest, MaterializesShreddedArrayElements) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integers = ColumnInt64::create(); + integers->get_data().assign({3, 4}); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().assign({0, 0}); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto wrappers = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().assign({0, 0}); + auto elements = ColumnNullable::create(std::move(wrappers), std::move(wrapper_nulls)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto array = ColumnArray::create(std::move(elements), std::move(offsets)); + auto array_nulls = ColumnUInt8::create(); + array_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(array), std::move(array_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_array_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + const VariantRef value = variants.get_value_ref(0); + ASSERT_EQ(value.num_elements(), 2); + EXPECT_EQ(value.array_at(0).get_int(), 3); + EXPECT_EQ(value.array_at(1).get_int(), 4); +} + +TEST(VariantColumnReaderTest, MaterializesVariantNestedInStruct) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + auto physical = ColumnStruct::create(std::move(root_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + root_schema.children.push_back(std::make_unique(unshredded_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + auto child_plan = std::make_unique(); + child_plan->schema = root_schema.children[0].get(); + child_plan->contains_variant = true; + plan.children.push_back(std::move(child_plan)); + + auto output = std::make_shared( + DataTypes {make_nullable(std::make_shared())}, + Strings {"payload"}) + ->create_column(); + const auto status = materialize_variant_columns(plan, *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& output_struct = assert_cast(*output); + const auto& nullable = assert_cast(output_struct.get_column(0)); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, AlignsNestedPrimitiveNullabilityAroundVariant) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + MutableColumns physical_fields; + physical_fields.push_back(nullable_strings({StringRef("required")}, {0})); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + physical_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + auto physical = ColumnStruct::create(std::move(physical_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + auto label_schema = std::make_unique(); + label_schema->name = "label"; + label_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + label_schema->type = make_nullable(std::make_shared()); + root_schema.children.push_back(std::move(label_schema)); + root_schema.children.push_back(std::make_unique(unshredded_schema())); + + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + for (const auto& child_schema : root_schema.children) { + auto child_plan = std::make_unique(); + child_plan->schema = child_schema.get(); + child_plan->contains_variant = child_schema->kind == ParquetColumnSchemaKind::VARIANT; + plan.children.push_back(std::move(child_plan)); + } + + auto output = std::make_shared( + DataTypes {std::make_shared(), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}) + ->create_column(); + const auto status = materialize_variant_columns(plan, *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& output_struct = assert_cast(*output); + EXPECT_EQ(output_struct.get_column(0).get_data_at(0).to_string(), "required"); + const auto& nullable = assert_cast(output_struct.get_column(1)); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, NestedMaterializationMovesUnaffectedSiblingBuffers) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto label = ColumnString::create(); + label->insert_data("large-sibling", 13); + const IColumn* decoded_label = label.get(); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + MutableColumns root_fields; + root_fields.push_back(std::move(label)); + root_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + ColumnPtr physical = ColumnStruct::create(std::move(root_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + auto label_schema = std::make_unique(); + label_schema->name = "label"; + label_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + label_schema->type = std::make_shared(); + root_schema.children.push_back(std::move(label_schema)); + root_schema.children.push_back(std::make_unique(unshredded_schema())); + + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + for (const auto& child_schema : root_schema.children) { + auto child_plan = std::make_unique(); + child_plan->schema = child_schema.get(); + child_plan->contains_variant = child_schema->kind == ParquetColumnSchemaKind::VARIANT; + plan.children.push_back(std::move(child_plan)); + } + + auto output = std::make_shared( + DataTypes {std::make_shared(), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}) + ->create_column(); + const IColumn* empty_output = output.get(); + ASSERT_TRUE(materialize_variant_columns(plan, std::move(physical), output).ok()); + EXPECT_NE(output.get(), empty_output); + const auto& output_struct = assert_cast(*output); + EXPECT_EQ(&output_struct.get_column(0), decoded_label); +} + +} // namespace doris::format::parquet diff --git a/be/test/format_v2/table/iceberg_reader_test.cpp b/be/test/format_v2/table/iceberg_reader_test.cpp index 806cc7e729a467..b9fe813c9c68f0 100644 --- a/be/test/format_v2/table/iceberg_reader_test.cpp +++ b/be/test/format_v2/table/iceberg_reader_test.cpp @@ -59,6 +59,7 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_varbinary.h" +#include "core/data_type/data_type_variant_v2.h" #include "exec/common/endian.h" #include "exec/scan/access_path_parser.h" #include "exprs/runtime_filter_expr.h" @@ -2627,6 +2628,44 @@ TEST(IcebergV2ReaderTest, IcebergLegacyPlanKeepsAllFieldIdsMappingRule) { TableColumnMappingMode::BY_NAME); } +TEST(IcebergV2ReaderTest, VariantFormatGateUsesPhysicalFileMappings) { + ColumnMapping missing_variant; + missing_variant.table_type = make_nullable(std::make_shared()); + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {missing_variant}) + .ok()); + + ColumnMapping physical_variant = missing_variant; + physical_variant.file_local_id = 0; + const auto orc_status = + doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {physical_variant}); + EXPECT_TRUE(orc_status.is()) << orc_status; + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::PARQUET, {physical_variant}) + .ok()); + + ColumnMapping projected_struct; + projected_struct.table_type = make_nullable(std::make_shared( + DataTypes {make_nullable(std::make_shared()), + make_nullable(std::make_shared())}, + Strings {"label", "payload"})); + projected_struct.file_local_id = 0; + ColumnMapping label; + label.table_type = make_nullable(std::make_shared()); + label.file_local_id = 1; + projected_struct.child_mappings = {label, missing_variant}; + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {projected_struct}) + .ok()); + + projected_struct.child_mappings[1] = physical_variant; + const auto nested_orc_status = + doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {projected_struct}); + EXPECT_TRUE(nested_orc_status.is()) << nested_orc_status; +} + TEST(IcebergV2ReaderTest, IcebergTableReaderDoesNotPushDownAggregateWithPositionDelete) { const auto test_dir = std::filesystem::temp_directory_path() / "doris_iceberg_aggregate_position_delete_test"; diff --git a/be/test/format_v2/table_reader_request_test.cpp b/be/test/format_v2/table_reader_request_test.cpp index 3845e086cea1b1..a58cc82f45d6cb 100644 --- a/be/test/format_v2/table_reader_request_test.cpp +++ b/be/test/format_v2/table_reader_request_test.cpp @@ -71,6 +71,32 @@ TEST(FileScanRequestBuilderTest, PredicateColumnRemovesDuplicateNonPredicateColu EXPECT_EQ(request.non_predicate_columns[0].column_id(), LocalColumnId(2)); } +TEST(FileScanRequestBuilderTest, DeferredComplexOutputSurvivesLaterPredicateMerge) { + FileScanRequest request; + FileScanRequestBuilder builder(&request); + + auto predicate = LocalColumnIndex::partial_local(5); + predicate.children.push_back(LocalColumnIndex::local(0)); + ASSERT_TRUE(builder.add_predicate_column(std::move(predicate)).ok()); + + auto output = LocalColumnIndex::partial_local(5); + output.children.push_back(LocalColumnIndex::local(0)); + output.children.push_back(LocalColumnIndex::local(1)); + ASSERT_TRUE(builder.add_deferred_non_predicate_column(std::move(output)).ok()); + + auto delete_predicate = LocalColumnIndex::partial_local(5); + delete_predicate.children.push_back(LocalColumnIndex::local(2)); + ASSERT_TRUE(builder.add_predicate_column(std::move(delete_predicate)).ok()); + + ASSERT_EQ(request.predicate_columns.size(), 1); + EXPECT_EQ(request.predicate_columns[0].children.size(), 2); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_EQ(request.non_predicate_columns[0].children.size(), 2); + EXPECT_EQ(request.local_positions.at(LocalColumnId(5)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(5)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(5))); +} + // Scenario: TableReader's format-specific customization path delegates to FileScanRequestBuilder // and preserves the same predicate/non-predicate de-duplication rule. TEST(TableReaderRequestTest, AppendPredicateColumnKeepsOtherNonPredicateColumns) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 024a036df11158..50ac62e871c66a 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -470,6 +470,19 @@ public long streamingSplitEstimate(ConnectorSession session, ConnectorTableHandl return fileCount >= threshold ? fileCount : -1; } + @Override + public boolean canServeMetadataOnlyCount(ConnectorSession session, ConnectorTableHandle handle, + Optional filter) { + IcebergTableHandle iceHandle = (IcebergTableHandle) handle; + if (iceHandle.isSystemTable() || filter.isPresent()) { + // Snapshot summaries describe the whole table and cannot prove a filtered row count. + return false; + } + Table table = resolveTable(session, iceHandle); + TableScan scan = buildScan(table, iceHandle, filter, session); + return getCountFromSnapshot(scan, session) >= 0; + } + /** * Lazy streaming split source (FIX-M3), mirroring legacy {@code IcebergScanNode.doStartSplit}: slice files at * a FIXED size ({@code file_split_size} if set, else {@code max_split_size} — NOT the per-table diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java index 2021e967017aff..7cd13afdb85425 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java @@ -93,8 +93,12 @@ public static ConnectorType fromIcebergType(Type icebergType, fieldIds.add(f.fieldId()); } return ConnectorType.structOf(names, types, nullable, comments).withChildrenFieldIds(fieldIds); + case VARIANT: + // Iceberg owns the Parquet Variant physical encoding, so expose an execution-only + // VariantV2 carrier without changing persisted Doris table metadata semantics. + return ConnectorType.of("VARIANT_COMPUTE_V2"); default: - // Any non-primitive iceberg type Doris cannot represent (VARIANT today; future non-primitive + // Any future non-primitive iceberg type Doris cannot represent // typeIds) degrades to UNSUPPORTED: the table still LOADS and only this column is // present-but-unqueryable. This DIVERGES from legacy fe-core (IcebergUtils.icebergTypeToDorisType // threw IllegalArgumentException at schema-load, failing the whole table). Graceful degradation diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java index 6594c2589d85c0..f25cdff9fd4a67 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java @@ -161,6 +161,7 @@ public IcebergWritePlanProvider(Map properties, @Override public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + validateWriteSchema(handle.getColumns(), handle.isWritesDataFiles()); IcebergTableHandle tableHandle = (IcebergTableHandle) handle.getTableHandle(); IcebergConnectorTransaction transaction = currentTransaction(session); @@ -213,7 +214,7 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl case MERGE: { TDataSink dataSink = new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK); dataSink.setIcebergMergeSink(buildMergeSink(table, tableHandle, rewritableDeletes, - handle.isRequireMergeCardinalityCheck(), schemaContext)); + handle.isWritesDataFiles(), handle.isRequireMergeCardinalityCheck(), schemaContext)); return new ConnectorSinkPlan(dataSink); } case REWRITE: { @@ -231,6 +232,27 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl } } + static void validateWriteSchema(List columns, boolean writesDataFiles) { + if (!writesDataFiles) { + return; + } + if (columns.stream().anyMatch(column -> containsVariant(column.getType()))) { + // Reject the whole data-file write: validating only selected columns would let an + // unchanged Variant target flow through a writer that cannot preserve its physical identity. + throw new DorisConnectorException( + "Iceberg VARIANT columns are read-only and cannot be written"); + } + } + + private static boolean containsVariant(ConnectorType type) { + String typeName = type.getTypeName(); + if ("VARIANT".equalsIgnoreCase(typeName) + || "VARIANT_COMPUTE_V2".equalsIgnoreCase(typeName)) { + return true; + } + return type.getChildren().stream().anyMatch(IcebergWritePlanProvider::containsVariant); + } + @Override public Optional> getWriteColumns(ConnectorSession session, ConnectorTableHandle tableHandle, Optional branchName) { @@ -535,7 +557,8 @@ private TIcebergDeleteSink buildDeleteSink(Table table, IcebergTableHandle table */ private TIcebergMergeSink buildMergeSink(Table table, IcebergTableHandle tableHandle, Map> rewritableDeletes, - boolean requireMergeCardinalityCheck, IcebergWriteSchemaContext schemaContext) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck, + IcebergWriteSchemaContext schemaContext) { TIcebergMergeSink tSink = new TIcebergMergeSink(); tSink.setDbName(tableHandle.getDbName()); tSink.setTbName(tableHandle.getTableName()); @@ -549,6 +572,7 @@ private TIcebergMergeSink buildMergeSink(Table table, IcebergTableHandle tableHa IcebergWriterHelper.shouldCollectColumnStats(schemaContext, schema)); // #66112: UPDATE and SQL MERGE share this sink, but only SQL MERGE has the one-source-row invariant. tSink.setRequireMergeCardinalityCheck(requireMergeCardinalityCheck); + tSink.setWritesDataFiles(writesDataFiles); PartitionSpec partitionSpec = schemaContext.getPartitionSpec(); if (partitionSpec.isPartitioned()) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 426f89c7d427b6..43bd3fd0d64c28 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -1459,6 +1459,22 @@ public void countPushdownFollowsTheSnapshotPin() { Assertions.assertEquals(10L, pinned.get(0).getPushDownRowCount()); } + @Test + public void metadataOnlyCountCapabilityUsesSnapshotSummary() { + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(dataFile( + table.spec(), "s3://b/db/t1/f1.parquet", 1000, null, null)).commit(); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider( + Collections.emptyMap(), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", Collections.emptyMap()); + + Assertions.assertTrue(provider.canServeMetadataOnlyCount( + session, new IcebergTableHandle("db1", "t1"), Optional.empty())); + Assertions.assertFalse(provider.canServeMetadataOnlyCount( + session, IcebergTableHandle.forSystemTable( + "db1", "t1", "snapshots", -1L, null, -1L), Optional.empty())); + } + @Test public void getScanNodePropertiesUnderPinEmitsFullPinnedSchemaDict() throws Exception { // T07 Option A: under a time-travel pin the field-id dict is built from the FULL pinned schema (covering diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java index 97f995ecf60eba..41c7e44f2e2a14 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java @@ -87,9 +87,9 @@ public void flagIndependentPrimitivesMatchLegacy() { @Test public void unknownAndV3TypesDegradeToUnsupportedByDesign() { - // WHY (user decision 2026-07-13, DV-051): iceberg types Doris cannot represent — the v3 primitives - // TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN and the non-primitive VARIANT — must map to - // UNSUPPORTED WITHOUT throwing, so the table still loads and only the exotic column is + // WHY (user decision 2026-07-13, DV-051): iceberg primitive types Doris cannot represent — + // TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN — must map to UNSUPPORTED WITHOUT throwing, + // so the table still loads and only the exotic column is // present-but-unqueryable. This deliberately DIVERGES from legacy fe-core, which threw // IllegalArgumentException("Cannot transform unknown type") at schema-load and failed the whole table. // This test PINS the graceful-degradation choice: MUTATION making either default arm throw -> red, @@ -100,9 +100,7 @@ public void unknownAndV3TypesDegradeToUnsupportedByDesign() { Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeometryType.crs84()).getTypeName()); Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeographyType.crs84()).getTypeName()); Assertions.assertEquals("UNSUPPORTED", mapOff(Types.UnknownType.get()).getTypeName()); - // VARIANT is NOT a primitive (falls to the nested-switch default); legacy mapped it to UNSUPPORTED - // too, so this stays parity while the primitives above are the intentional divergence. - Assertions.assertEquals("UNSUPPORTED", mapOff(Types.VariantType.get()).getTypeName()); + Assertions.assertEquals("VARIANT_COMPUTE_V2", mapOff(Types.VariantType.get()).getTypeName()); // The mapping flags do not rescue an unrepresentable type (both arms are flag-independent). Assertions.assertEquals("UNSUPPORTED", mapOn(Types.GeometryType.crs84()).getTypeName()); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java index cc431d2814e8c9..efca95e8e0902a 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java @@ -104,6 +104,19 @@ public class IcebergWritePlanProviderTest { private static final Map NON_REST_PROPS = Collections.singletonMap("iceberg.catalog.type", "hadoop"); + @Test + public void rejectsVariantDataWritesButAllowsDeleteOnlyMerge() { + ConnectorColumn nestedVariant = new ConnectorColumn("payload", + ConnectorType.structOf(Collections.singletonList("nested"), + Collections.singletonList(ConnectorType.of("VARIANT"))), + null, true, null); + Assertions.assertThrows(DorisConnectorException.class, + () -> IcebergWritePlanProvider.validateWriteSchema( + Collections.singletonList(nestedVariant), true)); + Assertions.assertDoesNotThrow(() -> IcebergWritePlanProvider.validateWriteSchema( + Collections.singletonList(nestedVariant), false)); + } + private static InMemoryCatalog freshCatalog() { InMemoryCatalog catalog = new InMemoryCatalog(); catalog.initialize("test", Collections.emptyMap()); diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java index 3987325ae56949..dc7adcfa07d42e 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java @@ -100,6 +100,14 @@ default boolean isRequireMergeCardinalityCheck() { return false; } + /** + * Whether this write can emit data files. A delete-only MERGE returns false so a connector may + * allow position-delete output even when the table has read-only column types. + */ + default boolean isWritesDataFiles() { + return true; + } + /** * The named table branch this write targets ({@code INSERT INTO t@branch(name)}), or * {@link Optional#empty()} when the write goes to the table's default ref. Threaded from the diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java index d1e5569b3fec90..2a5caa2ffcaa33 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java @@ -352,6 +352,17 @@ default long streamingSplitEstimate( return -1; } + /** + * Whether this connector can answer the current table-level COUNT(*) without decoding data files. + * The default is false; connectors may use snapshot metadata to prove the stronger condition. + */ + default boolean canServeMetadataOnlyCount( + ConnectorSession session, + ConnectorTableHandle handle, + Optional filter) { + return false; + } + /** * Builds a lazy {@link ConnectorSplitSource} for streaming split generation. Called once, on a * background task, only when {@link #streamingSplitEstimate} returned a non-negative value. The diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java index a81f27720c10dc..30909a05c14c19 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java @@ -318,6 +318,11 @@ private static Type convertScalarType(String typeName, int precision, int scale) return ScalarType.createVarbinaryType(ScalarType.MAX_VARBINARY_LENGTH); case "JSONB": return ScalarType.createType("JSON"); + case "VARIANT_COMPUTE_V2": + // This carrier is execution-only: connector schemas use it for native external + // Variant encodings, while persisted Doris table metadata keeps regular Variant rules. + return new org.apache.doris.catalog.VariantType( + new ArrayList<>(), 0, false, 10000, 0, false, 0L, 64, false, true); case "UNSUPPORTED": return Type.UNSUPPORTED; default: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index ec13b7e04b8a10..6f51288d5ceed9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -26,9 +26,14 @@ import org.apache.doris.analysis.TableScanParams; import org.apache.doris.analysis.ToSqlParams; import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.catalog.ArrayType; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.MapType; +import org.apache.doris.catalog.StructType; import org.apache.doris.catalog.TableIf; +import org.apache.doris.catalog.Type; +import org.apache.doris.catalog.VariantType; import org.apache.doris.common.UserException; import org.apache.doris.common.profile.RuntimeProfile; import org.apache.doris.common.profile.SummaryProfile; @@ -192,6 +197,64 @@ public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, this.currentHandle = tableHandle; } + @Override + protected void doInitialize() throws UserException { + super.doInitialize(); + checkVariantBackendCompatibilityForCurrentScan(backendPolicy.getBackends()); + } + + void checkVariantBackendCompatibilityForCurrentScan(Iterable backends) + throws UserException { + boolean metadataCountProven = false; + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); + if (isTableLevelCountStarPushdown() && conjuncts.isEmpty() && scanProvider != null) { + metadataCountProven = onPluginClassLoader(scanProvider, + () -> scanProvider.canServeMetadataOnlyCount( + connectorSession, currentHandle, Optional.empty())); + } + checkVariantBackendCompatibility( + !metadataCountProven && projectsComputeVariant(desc), backends); + } + + static boolean projectsComputeVariant(TupleDescriptor tuple) { + // Nested-column pruning updates the effective slot type but deliberately keeps the original + // Column metadata; compatibility must follow the payload this scan actually projects. + return tuple.getSlots().stream().anyMatch(slot -> containsComputeVariant(slot.getType())); + } + + private static boolean containsComputeVariant(Type type) { + if (type instanceof VariantType) { + return ((VariantType) type).isComputeV2(); + } + if (type instanceof ArrayType) { + return containsComputeVariant(((ArrayType) type).getItemType()); + } + if (type instanceof MapType) { + MapType map = (MapType) type; + return containsComputeVariant(map.getKeyType()) || containsComputeVariant(map.getValueType()); + } + if (type instanceof StructType) { + return ((StructType) type).getFields().stream() + .anyMatch(field -> containsComputeVariant(field.getType())); + } + return false; + } + + static void checkVariantBackendCompatibility(boolean projectsVariant, Iterable backends) + throws UserException { + if (!projectsVariant) { + return; + } + for (Backend backend : backends) { + if (backend.isSmoothUpgradeSrc()) { + // Old backends cannot distinguish the logical Variant from its physical carrier, + // so scheduling this projection there could corrupt the result shape. + throw new UserException("Iceberg Variant is unavailable while backend " + + backend.getId() + " is a smooth upgrade source"); + } + } + } + // Lazily resolves this node's ConnectorMetadata through the per-statement funnel and caches it, so the // per-method resolvers below share one instance for the statement instead of rebuilding it each time. private ConnectorMetadata metadata() { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 5af8be9c147bbf..5a9d2dee752b01 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -555,7 +555,7 @@ public PlanFragment visitPhysicalExternalRowLevelDeleteSink( // TIcebergDeleteSink dialect. No output-expr / materialized-name loop is needed: the row id reaches // BE as the __DORIS_ICEBERG_ROWID_COL__ block column (a real hidden column), and viceberg_delete_sink // resolves it by block-name, not by output-expr name. - rootFragment.setSink(buildPluginRowLevelDmlSink(deleteSink, WriteOperation.DELETE, false)); + rootFragment.setSink(buildPluginRowLevelDmlSink(deleteSink, WriteOperation.DELETE, false, false)); return rootFragment; } @@ -592,7 +592,7 @@ public PlanFragment visitPhysicalExternalRowLevelMergeSink( // SQL MERGE INTO carries the cardinality requirement onto the write handle; UPDATE shares this // sink dialect but has no such rule, so it threads false (see PhysicalExternalRowLevelMergeSink). rootFragment.setSink(buildPluginRowLevelDmlSink(mergeSink, WriteOperation.MERGE, - mergeSink.isRequireMergeCardinalityCheck())); + mergeSink.isWritesDataFiles(), mergeSink.isRequireMergeCardinalityCheck())); return rootFragment; } @@ -609,7 +609,7 @@ public PlanFragment visitPhysicalExternalRowLevelMergeSink( */ private PluginDrivenTableSink buildPluginRowLevelDmlSink( PhysicalBaseExternalTableSink sink, WriteOperation writeOperation, - boolean requireMergeCardinalityCheck) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck) { PluginDrivenExternalTable targetTable = (PluginDrivenExternalTable) sink.getTargetTable(); PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) targetTable.getCatalog(); @@ -652,7 +652,8 @@ private PluginDrivenTableSink buildPluginRowLevelDmlSink( // writeSortInfo == null: a row-level DML has no engine-resolved write sort (MERGE's sort lives in the // connector's TIcebergMergeSink.sort_fields, DELETE is unsorted). return new PluginDrivenTableSink(targetTable, writePlanProvider, connSession, - providerTableHandle, connectorColumns, null, writeOperation, requireMergeCardinalityCheck); + providerTableHandle, connectorColumns, null, writeOperation, + writesDataFiles, requireMergeCardinalityCheck); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java index c00d9e7fd619a6..7aa7e357229a0c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java @@ -39,6 +39,7 @@ public Rule build() { sink.getTargetTable(), sink.getCols(), sink.getOutputExprs(), + sink.isWritesDataFiles(), sink.isRequireMergeCardinalityCheck(), Optional.empty(), sink.getLogicalProperties(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java index 0633637b3377aa..915ef5f7eba26f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java @@ -137,6 +137,15 @@ public Void visitSlotReference(SlotReference slotReference, CollectorContext con path, context.bottomFilter, ColumnAccessPathType.DATA)); return null; } + if (dataType instanceof VariantType) { + // A root Variant consumer must dominate any predicate-only leaf path. Otherwise the + // scanner can legally project a shredded leaf that cannot serve the root expression. + int slotId = slotReference.getExprId().asInt(); + slotToAccessPaths.put(slotId, new CollectAccessPathResult( + ImmutableList.of(slotReference.getName()), + context.bottomFilter, ColumnAccessPathType.DATA)); + return null; + } if (dataType instanceof NestedColumnPrunable) { context.accessPathBuilder.addPrefix(slotReference.getName().toLowerCase()); ImmutableList path = Utils.fastToImmutableList(context.accessPathBuilder.accessPath); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java index 7e7674d20586c7..c9cd9d9046fcd4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java @@ -25,6 +25,7 @@ import org.apache.doris.nereids.trees.expressions.Expression; import org.apache.doris.nereids.trees.expressions.NamedExpression; import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SlotReference; import org.apache.doris.nereids.trees.expressions.functions.Function; import org.apache.doris.nereids.trees.expressions.functions.generator.Explode; import org.apache.doris.nereids.trees.expressions.functions.generator.ExplodeMap; @@ -249,6 +250,12 @@ public Void visitLogicalProject(LogicalProject project, Statemen List outerPath = outerSlotAccessPath.getPath(); List replaceSlotNamePath = new ArrayList<>(); replaceSlotNamePath.add(innerSlot.getName()); + if (outerPath.size() == 1 && innerSlot instanceof SlotReference + && ((SlotReference) innerSlot).hasSubColPath()) { + // A whole access to a derived subcolumn slot is whole only relative to that + // slot; preserve its physical leaf path when propagating to the scan slot. + replaceSlotNamePath.addAll(((SlotReference) innerSlot).getSubPath()); + } replaceSlotNamePath.addAll(outerPath.subList(1, outerPath.size())); allSlotToAccessPaths.put( innerSlot.getExprId().asInt(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java index 6ae6ef2b1abf9d..3faf0d581a5c72 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java @@ -724,6 +724,11 @@ public void setAccessByPath(List path, int accessIndex, ColumnAccessPath // Any other sub-path on a string column means full data is needed. accessAll = true; return; + } else if (type.isVariantType()) { + // Variant object keys stay in the serialized access path. Keeping the terminal type + // here lets BE project a shredded leaf without inventing static schema fields. + accessAll = true; + return; } else if (isRoot) { children.get(path.get(accessIndex).toLowerCase()).setAccessByPath(path, accessIndex + 1, pathType); return; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java index 5e3b601df33958..ca8a3b5ba4c7cd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java @@ -64,6 +64,7 @@ import org.apache.doris.nereids.types.MapType; import org.apache.doris.nereids.types.NestedColumnPrunable; import org.apache.doris.nereids.types.StructType; +import org.apache.doris.nereids.types.VariantType; import org.apache.doris.nereids.util.MoreFieldsThread; import com.google.common.collect.ImmutableCollection; @@ -669,6 +670,10 @@ private void replaceAccessPathToFieldId(List originPath, int index, Data break; } } + } else if (type instanceof VariantType) { + // Variant object keys are data, not Iceberg schema field IDs. Replacing them with + // the root ID destroys the physical shredding path before it reaches the scanner. + return; } else { originPath.set(index, String.valueOf(column.getUniqueId())); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java index 853d2275f467cd..fb98fca5cfe7ac 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java @@ -411,6 +411,8 @@ LogicalPlan buildMergePlan(ConnectContext ctx, ExternalTable icebergTable) { icebergTable, ConnectorWriteSchemaUtils.pinAndGet(ctx, icebergTable), outputExprs, + matchedClauses.stream().anyMatch(clause -> !clause.isDelete()) + || !notMatchedClauses.isEmpty(), true, Optional.empty(), Optional.empty(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java index 59686f0012171c..041375512ca7e7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java @@ -138,6 +138,7 @@ LogicalPlan buildMergePlan(ConnectContext ctx, LogicalPlan logicalQuery, icebergTable, writeColumns, outputExprs, + true, false, Optional.empty(), Optional.empty(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java index d2e55779ef5b48..017bfaccdb2666 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java @@ -45,6 +45,8 @@ public class LogicalExternalRowLevelMergeSink extends L implements Sink, PropagateFuncDeps { private final ExternalDatabase database; private final ExternalTable targetTable; + // Delete-only MERGE emits position deletes but never invokes the data-file writer. + private final boolean writesDataFiles; // True for SQL MERGE INTO, false for UPDATE. MERGE must reject a target row matched by more than one // source row (SQL cardinality rule), which the BE sink can only do when the plan keeps the merge // distribution; UPDATE has no such rule. Read by RequestPropertyDeriver (which otherwise drops the @@ -67,12 +69,29 @@ public LogicalExternalRowLevelMergeSink(ExternalDatabase database, Optional groupExpression, Optional logicalProperties, CHILD_TYPE child) { + this(database, targetTable, cols, outputExprs, true, requireMergeCardinalityCheck, + groupExpression, logicalProperties, child); + } + + /** + * Constructor that records whether the merge writes replacement data files. + */ + public LogicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + Optional logicalProperties, + CHILD_TYPE child) { super(PlanType.LOGICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK, outputExprs, groupExpression, logicalProperties, cols, child); this.database = Objects.requireNonNull(database, "database != null in LogicalExternalRowLevelMergeSink"); this.targetTable = Objects.requireNonNull(targetTable, "targetTable != null in LogicalExternalRowLevelMergeSink"); + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } @@ -81,19 +100,19 @@ public Plan withChildAndUpdateOutput(Plan child) { .map(NamedExpression.class::cast) .collect(ImmutableList.toImmutableList()); return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, output, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child); + writesDataFiles, requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child); } @Override public Plan withChildren(List children) { Preconditions.checkArgument(children.size() == 1, "LogicalExternalRowLevelMergeSink only accepts one child"); return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), children.get(0)); + writesDataFiles, requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), children.get(0)); } public LogicalExternalRowLevelMergeSink withOutputExprs(List outputExprs) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child()); + writesDataFiles, requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child()); } public ExternalDatabase getDatabase() { @@ -104,6 +123,10 @@ public ExternalTable getTargetTable() { return targetTable; } + public boolean isWritesDataFiles() { + return writesDataFiles; + } + public boolean isRequireMergeCardinalityCheck() { return requireMergeCardinalityCheck; } @@ -123,12 +146,14 @@ public boolean equals(Object o) { return Objects.equals(database, that.database) && Objects.equals(targetTable, that.targetTable) && Objects.equals(cols, that.cols) + && writesDataFiles == that.writesDataFiles && requireMergeCardinalityCheck == that.requireMergeCardinalityCheck; } @Override public int hashCode() { - return Objects.hash(super.hashCode(), database, targetTable, cols, requireMergeCardinalityCheck); + return Objects.hash(super.hashCode(), database, targetTable, cols, writesDataFiles, + requireMergeCardinalityCheck); } @Override @@ -138,6 +163,7 @@ public String toString() { "database", database.getFullName(), "targetTable", targetTable.getName(), "cols", cols, + "writesDataFiles", writesDataFiles, "requireMergeCardinalityCheck", requireMergeCardinalityCheck); } @@ -149,13 +175,15 @@ public R accept(PlanVisitor visitor, C context) { @Override public Plan withGroupExpression(Optional groupExpression) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, Optional.of(getLogicalProperties()), child()); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, Optional.of(getLogicalProperties()), child()); } @Override public Plan withGroupExprLogicalPropChildren(Optional groupExpression, Optional logicalProperties, List children) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, logicalProperties, children.get(0)); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, logicalProperties, children.get(0)); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java index 9634c33e2458f6..4b27e865c44c06 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java @@ -59,6 +59,7 @@ */ public class PhysicalExternalRowLevelMergeSink extends PhysicalBaseExternalTableSink { + private final boolean writesDataFiles; // True for SQL MERGE INTO, false for UPDATE; see LogicalExternalRowLevelMergeSink. private final boolean requireMergeCardinalityCheck; @@ -73,10 +74,27 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, Optional groupExpression, LogicalProperties logicalProperties, CHILD_TYPE child) { - this(database, targetTable, cols, outputExprs, requireMergeCardinalityCheck, + this(database, targetTable, cols, outputExprs, true, requireMergeCardinalityCheck, groupExpression, logicalProperties, PhysicalProperties.GATHER, null, child); } + /** + * Constructor that records whether the merge writes replacement data files. + */ + public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + LogicalProperties logicalProperties, + CHILD_TYPE child) { + this(database, targetTable, cols, outputExprs, writesDataFiles, + requireMergeCardinalityCheck, groupExpression, logicalProperties, + PhysicalProperties.GATHER, null, child); + } + /** * Constructor */ @@ -90,11 +108,34 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, PhysicalProperties physicalProperties, Statistics statistics, CHILD_TYPE child) { + this(database, targetTable, cols, outputExprs, true, requireMergeCardinalityCheck, + groupExpression, logicalProperties, physicalProperties, statistics, child); + } + + /** + * Constructor with explicit data-file and physical-property settings. + */ + public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + LogicalProperties logicalProperties, + PhysicalProperties physicalProperties, + Statistics statistics, + CHILD_TYPE child) { super(PlanType.PHYSICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK, database, targetTable, cols, outputExprs, groupExpression, logicalProperties, physicalProperties, statistics, child); + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } + public boolean isWritesDataFiles() { + return writesDataFiles; + } + public boolean isRequireMergeCardinalityCheck() { return requireMergeCardinalityCheck; } @@ -103,7 +144,7 @@ public boolean isRequireMergeCardinalityCheck() { public Plan withChildren(List children) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, - cols, outputExprs, requireMergeCardinalityCheck, groupExpression, + cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), physicalProperties, statistics, children.get(0)); } @@ -116,7 +157,8 @@ public R accept(PlanVisitor visitor, C context) { public Plan withGroupExpression(Optional groupExpression) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), child()); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, getLogicalProperties(), child()); } @Override @@ -124,13 +166,14 @@ public Plan withGroupExprLogicalPropChildren(Optional groupExpr Optional logicalProperties, List children) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, logicalProperties.get(), children.get(0)); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, logicalProperties.get(), children.get(0)); } @Override public PhysicalPlan withPhysicalPropertiesAndStats(PhysicalProperties physicalProperties, Statistics statistics) { return new PhysicalExternalRowLevelMergeSink<>( - database, targetTable, cols, outputExprs, requireMergeCardinalityCheck, + database, targetTable, cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), physicalProperties, statistics, child()); } @@ -146,12 +189,13 @@ public boolean equals(Object o) { return false; } PhysicalExternalRowLevelMergeSink that = (PhysicalExternalRowLevelMergeSink) o; - return requireMergeCardinalityCheck == that.requireMergeCardinalityCheck; + return writesDataFiles == that.writesDataFiles + && requireMergeCardinalityCheck == that.requireMergeCardinalityCheck; } @Override public int hashCode() { - return Objects.hash(super.hashCode(), requireMergeCardinalityCheck); + return Objects.hash(super.hashCode(), writesDataFiles, requireMergeCardinalityCheck); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java index e9e539ea018e2f..1ec9e8075a7f32 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java @@ -70,6 +70,7 @@ public class PluginDrivenTableSink extends BaseExternalTableDataSink { // the INSERT TIcebergTableSink. Threaded onto the write handle so planWrite's buildWriteContext // reads it via ConnectorWriteHandle.getWriteOperation(). private final WriteOperation writeOperation; + private final boolean writesDataFiles; // SQL MERGE INTO only: the statement must reject a target row matched by more than one source row. // Carried from PhysicalExternalRowLevelMergeSink onto the write handle so the connector can stamp the // enforcement flag onto its BE sink; false for UPDATE and for every non-row-level write. @@ -121,6 +122,18 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, ConnectorWritePlanProvider writePlanProvider, ConnectorSession connectorSession, ConnectorTableHandle tableHandle, List connectorColumns, TSortInfo writeSortInfo, WriteOperation writeOperation, boolean requireMergeCardinalityCheck) { + this(targetTable, writePlanProvider, connectorSession, tableHandle, connectorColumns, + writeSortInfo, writeOperation, true, requireMergeCardinalityCheck); + } + + /** + * Plan-provider mode with explicit data-file and merge-cardinality requirements. + */ + public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, + ConnectorWritePlanProvider writePlanProvider, ConnectorSession connectorSession, + ConnectorTableHandle tableHandle, List connectorColumns, + TSortInfo writeSortInfo, WriteOperation writeOperation, boolean writesDataFiles, + boolean requireMergeCardinalityCheck) { super(); this.targetTable = targetTable; this.writePlanProvider = writePlanProvider; @@ -129,6 +142,7 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, this.connectorColumns = connectorColumns; this.writeSortInfo = writeSortInfo; this.writeOperation = writeOperation == null ? WriteOperation.INSERT : writeOperation; + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } @@ -162,7 +176,7 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { // EXPLAIN), so the connector derives the detail from the write handle. ConnectorWriteHandle handle = new PluginDrivenWriteHandle( tableHandle, connectorColumns, false, Collections.emptyMap(), null, Optional.empty(), - writeOperation, requireMergeCardinalityCheck); + writeOperation, writesDataFiles, requireMergeCardinalityCheck); writePlanProvider.appendExplainInfo(sb, prefix, connectorSession, handle); return sb.toString(); } @@ -189,7 +203,7 @@ public void bindDataSink(Optional insertCtx) } ConnectorWriteHandle handle = new PluginDrivenWriteHandle( tableHandle, connectorColumns, overwrite, writeContext, writeSortInfo, branchName, - writeOperation, requireMergeCardinalityCheck); + writeOperation, writesDataFiles, requireMergeCardinalityCheck); ConnectorSinkPlan sinkPlan = writePlanProvider.planWrite(connectorSession, handle); this.tDataSink = sinkPlan.getDataSink(); } @@ -210,12 +224,13 @@ private static final class PluginDrivenWriteHandle implements ConnectorWriteHand private final TSortInfo sortInfo; private final Optional branchName; private final WriteOperation writeOperation; + private final boolean writesDataFiles; private final boolean requireMergeCardinalityCheck; private PluginDrivenWriteHandle(ConnectorTableHandle tableHandle, List columns, boolean overwrite, Map writeContext, TSortInfo sortInfo, Optional branchName, WriteOperation writeOperation, - boolean requireMergeCardinalityCheck) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck) { this.tableHandle = tableHandle; this.columns = columns; this.overwrite = overwrite; @@ -223,9 +238,15 @@ private PluginDrivenWriteHandle(ConnectorTableHandle tableHandle, List PluginDrivenScanNode.checkVariantBackendCompatibility( + true, Collections.singletonList(backend))); + Assert.assertTrue(exception.getMessage().contains("backend 7")); + } + + @Test + public void compatibilityCheckIgnoresScansWithoutComputeVariant() throws UserException { + Backend backend = new Backend(7L, "127.0.0.1", 9050); + backend.setSmoothUpgradeSrc(true); + + PluginDrivenScanNode.checkVariantBackendCompatibility( + false, Collections.singletonList(backend)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java index 3c92606e4e3fcc..ca9c59067c625b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java @@ -45,6 +45,9 @@ import org.apache.doris.nereids.types.DataType; import org.apache.doris.nereids.types.NestedColumnPrunable; import org.apache.doris.nereids.types.NullType; +import org.apache.doris.nereids.types.StructField; +import org.apache.doris.nereids.types.StructType; +import org.apache.doris.nereids.types.VariantType; import org.apache.doris.nereids.util.MemoPatternMatchSupported; import org.apache.doris.nereids.util.PlanChecker; import org.apache.doris.planner.OlapScanNode; @@ -1050,6 +1053,22 @@ public void testDataTypeAccessTree() { ); } + @Test + public void testDataTypeAccessTreeKeepsVariantTerminalPath() { + StructType type = new StructType(ImmutableList.of( + new StructField("payload", VariantType.INSTANCE, true, ""))); + SlotReference slot = new SlotReference("info", type); + DataTypeAccessTree tree = DataTypeAccessTree.ofRoot(slot, ColumnAccessPathType.DATA); + + tree.setAccessByPath(ImmutableList.of("info", "payload", "typed_col"), 0, + ColumnAccessPathType.DATA); + + DataType prunedType = tree.pruneDataType().get(); + Assertions.assertInstanceOf(StructType.class, prunedType); + Assertions.assertEquals(VariantType.INSTANCE, + ((StructType) prunedType).getFields().get(0).getDataType()); + } + @Test public void testWithVariant() throws Exception { connectContext.getSessionVariable().enableDecimal256 = true; diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java index 5268569e0b8861..cba617e96ed631 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java @@ -105,6 +105,25 @@ public void testVariantOrPredicatePaths() throws Exception { ); } + @Test + public void testWholeVariantOutputDominatesPredicateLeafProjection() throws Exception { + String rootOutputSql = "select v from variant_tbl where v['n'] > 1"; + assertAllAccessPathsContain( + rootOutputSql, + ImmutableList.of(path("v")), + ImmutableList.of() + ); + assertPredicateAccessPathsEqual(rootOutputSql, ImmutableList.of(path("v", "n"))); + + String predicateOnlySql = "select count(*) from variant_tbl where v['n'] > 1"; + assertAllAccessPathsContain( + predicateOnlySql, + ImmutableList.of(path("v", "n")), + ImmutableList.of(path("v")) + ); + assertPredicateAccessPathsEqual(predicateOnlySql, ImmutableList.of(path("v", "n"))); + } + @Test public void testVariantIfExpressionPaths() throws Exception { assertVariantSubColumnSlots( diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java index b91a71b62ce111..6a75bdcb8938b9 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java @@ -225,6 +225,20 @@ null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), Assert.assertEquals(WriteOperation.DELETE, provider.seenHandle.getWriteOperation()); } + @Test + public void bindDataSinkThreadsDeleteOnlyMergeToHandle() throws AnalysisException { + RecordingWritePlanProvider provider = new RecordingWritePlanProvider( + new ConnectorSinkPlan(new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK))); + PluginDrivenTableSink sink = new PluginDrivenTableSink( + null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), + null, WriteOperation.MERGE, false, true); + sink.bindDataSink(Optional.empty()); + + // Delete-only MERGE must bypass data-file validation while retaining cardinality enforcement. + Assert.assertFalse(provider.seenHandle.isWritesDataFiles()); + Assert.assertTrue(provider.seenHandle.isRequireMergeCardinalityCheck()); + } + @Test public void getExplainStringThreadsWriteOperationToHandle() { // WHY: EXPLAIN of a post-flip MERGE/DELETE builds a (degraded) handle for appendExplainInfo; the diff --git a/gensrc/thrift/DataSinks.thrift b/gensrc/thrift/DataSinks.thrift index 9efe29a51eb421..7295da1a6874fd 100644 --- a/gensrc/thrift/DataSinks.thrift +++ b/gensrc/thrift/DataSinks.thrift @@ -539,6 +539,8 @@ struct TIcebergMergeSink { 14: optional bool collect_column_stats; // Unset preserves old-FE UPDATE behavior; execution version gates SQL MERGE validation. 15: optional bool require_merge_cardinality_check; + // Unset preserves old-FE UPDATE behavior, which always writes replacement data rows. + 16: optional bool writes_data_files; // delete side (position delete only) 20: optional TFileContent delete_type diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet new file mode 100644 index 0000000000000000000000000000000000000000..f45415374f2f455c0d31dcb1b98c67005ef0d7b5 GIT binary patch literal 34528 zcmeHw2_RM5_y2Lbp$sX`iBgH4>V*uMqBOkMMAP$NB5LYfT z51FST(?ux>iBN{f{qJ*b6HV{=KL6kM_kQ={wC-MKue~bFcOtQ3X1k6Q3riV_I3?TWD?cd zrdLaVixn>smBVh#x_LoKm8(QdE4v9^XhCt&7Kz&rJdN3P7nIf)NW}Gdn&7wti}Hmf zZWXH-&sGgAsfm<`?Nl*gmI^E`BT7U!lo@kc1eP|nOWYYLGht%WD9S|=qV9_r&r;SX zdFY!E^IXJa<|>Wi619Zek79b3)W%09ja4_+Cd9psF`2Qjv8X^M;a0hkF`H&%Ni8KI z_LY$di$Y^@g>^!7bGPvv`^M6i!Gt^SyL%y)wM99cW>ITmOL1L{@BYb<{5u4 ze)-zs;$3F9tAl&k_1Bg*RG7sL1oz_R-z&N=W_GLeV9)IR_evZOzFJUsui~w->2=z@ z;&NlNXlhdrr_H_6<{q;7aSkD~C zccqWh4m^7e;US0)*llJw@cb~9>AQe=^8&!)@1TIiQNXpCl!Ve$nM-MS{A^ZMJ#A;+ z8f|7-m!%wpqj|iwrmnK5=bqQoyUzRQ=oVR5iSq=bXS{W`1+u4R?bJKs<6omEEPH0< zPQtNj|60pPS!cd-Jz}E&BO{{hS^08;@qmAwLwmB*Oi?`@5A~WONV3Z>?+gF%rFarV z_3c)vKiHv0IkzvCXl1Bgtx-$yvWnF|v9P{siwwnmCy8jGS^rRtLOD+&>Dw#RKiFkW z@z8ojw6U+RJ~T+VaN?D|Ba2+sMow$D&5PPix5_=->t*e!yXb}Oa=8cFcMTq<4LIJE zdt|W7##JihxzP)`I=c#+Q*2soNBmN12x2y8R%tyyR+mz16=&nTu(^$xobrfdY;#tj z`ML2>O5KT`At#n~Z8{fhYjhV6xh!Az+@R97)*^K1^!$uA!X4X3hKGh+Wiy_WdTs0M zTZT@}InkyYHe92>e(22V6VDAF57*jc4mmG*-=?26{K(|w&{?JT&rN8<^Bu|k3KQTO ze{}Bp_x+^H#g7Y75b z>4t0NyDi5iz}0vqsbtpFyo-$i*L1`6YVQ7P6X41$FC^oq2G<-0T+9RIt>F$oXN+)KXXyEOFzvhTp*Qq7!mYGm@hm}An> znt6e>WQtXc;kuX9Y*!g_!cL=OQbE*QUkW*iWMrs7rRI29lM}VNkIAM{^Fjy7$tSuE zl|~h^PjQ;ZZ|3h3KdX>?(aZd66ey!CC)s_@Zig}b!j>0jm;A^o4RC@)AB-k%#-b#UalXq&vstiAHP+x zOX7lku1{$HU46xu@|E^E9*6qL`_sFm@7U)Bwe(YL(qFFY9nC(wzCU5NLzh(8Xs%yo zf0BvAONGayITud$CmtT_lFb^;3mflGb{u0{mOZokGhSdOf341aXlio}7j}kwO}Bfs z@HQ--09S)2B^~!&_xjOH-i5L7n)Zd)f`Ot(6X43I$BCJA!Fe+?QoqrCRY3zF0k;*3TUtUzIT0Ccxjau_>$6mu_YH+b%EcW zo(wiOE-tR3Z04_aQD_q37d?%5t}iI>C=icK@Fej+TTnSH{Q59W>YBTX(LA5PvbIR^ z8@E(Q3#tPv-VnvF2b38hiGk(a?cz66%Sej{0xQRm_^UL{AMT|Xa~Nur{sS8+e`6Qr zQJh)q%Y(hlvftf8`A7Fp#`{)pJf(!g@WR7_@D8{$|6L@9B8-EoZ7#hd z+HFOvwyA6{E>mWK;$?aF&jmYSVC#h$7K3ZNOCPZD!$nLme)3M4u3Qc)lzW%pF=I-% z<6#l_S?=2s4u0758^TvWP@ZnnT-=mo|4q#qs-Rn+lksF&g<+My>idko!ssiEzQPy| z|3@bYoUjc1=&64en6F@&JeW=r`RWS;>rM86EYew=v8VPpgbAFUwOKuPZYb+-%jR8l z-}Gnr+z{UUWk`bi=7Ue?hT+FHY&yAYuKV^)e_Xm4%oDK;zO{X`#>0!D?0U<%1By4P zk^8tCMgk5GU*T_SlRde7eroaNL;V-Sa6A!QXI5_7dv0&=yvq?hH&$*wP_j34_MZ{+ zuH4$R@Alr11vL>!(yh%}U3W$r#d-1s1r z(=viPq;u2$dk;buyHxocdJ{S4MCVpVW^wP0>_pbhd$2GUZ$6I`Cs5^7ME0m+u8Ny3+=>AFDf5wpS?e|z{=RvN3*G& z(#NyZ9TNT@9?Y1khIQ06`{`7VZ6T3qD^{+4Pc4%iiJi_efZ|TWGPbCHTRWQRD zIS)%eK4zRWLjbwI{s2Fyigr)AR0BM$iq%L${76*4SBT$!QC&l`Qc%`z|9I&1j8hwq zZu8#&VASJv=J=2U{LXpO(<{d{|3n(_SaH8ve7KS($OP`2AN?-+Q4IY7);Mv|OO48R zGV#~W8yRs2HI}tg;%~$nkrq-LD+aCOuZMIS%};AA?-`80nbA#JGTK-<&M98=gZzQ9 z=3Bcg?xFbuG*gO2@JlRzV7yasep-Ik#1sPB;F7x0I}4jSgk{t2$Aau%V~aF}rcN=N zw8D%Yi!2t!j-_+#^RF)+NLjAfxpKFCL1O4Y`uy~cUsl`Sk3KYTPd2?%)Wp6ptz{r{ zjzfpglF|H|>jzR-J9Mr(JX(;PIgqhrtYf*-=>0n<2NYB-2wOh`Si}s!>vwVgnDH?g zz#`SFX!3Ryl?{ce^pJ4_B{sR65J!oBPC%bN7L3R#E!P9i#|iVW&c<)z`fZzvz#5(5 zGx07$igVqzEfeHE6ZMg(OZuFN(GyaTFWN3uaTlLjr5UWvrr<*L77pxvmM9*m-n^RABXK-@D<`q@&6Tjn${TEW@R= zI(Eq3t9A~36E1tALs{val+!8v0E;tk8JDZ1XRD|7vg^F%yake8N_>@gVM+I56`fzJ z{VUjsk{((9<^1FAOWGwbjEForJg&RcL%nRCFWK|?SmkUr^0`MbI-)PtE16};?&U^0 ztElxAoYrKI=5C$UqxF?coaSzM{H-er*S3;neZPC9L+h%?!xeK*;tp*W&iRZU zaK9T}?PtD+-beS#N-L!#5vT=ZO^;HFmS3}4siw7POk+Qqx--%;l<{PP0M~+fJgl7f zTwcBSrLNM!t>?9W6XI)Fu|+!Y!Fe4OT|TN@fplo!PVG(H0rks-r2~q0>g;fQy;U`! zaZRLjNN2hB7O8*+5u$WZL%Gf_ivX%(yL8xysP;y-1NDMPlK*{Co$bm88pA|%50oBg znXPvB(v{dF`=SptFQ|Q^Z#d$S6sxQC^1x%}@nf48Hq;BtBn6a{bd)t48f7R+A+KI( zZ&he$5VKASYJR1&+rELielRKQ{h}uuHZN+^J}XzF!)fCpwCMTKLb+NquOV4So73Fy z^S1A@KBpSeZmB`6*Sui&?uXYqguxtg-BFn%Q|ntruD2FOJKm zJlJV$?VXSA_9npDD4I#fVl_m8 z-|3nq?Thx^uZcP(Md7`ii7AMn*-qY9H*6E&ItNC;rE7FpmF04|Pr3ajD7n{d$C_|% zXa9Ub88bI!#iu?_F3SWZwr|@ZdCBMWhWCX%Tq6q2%V*(H_s;^*F=S|pgJH&h1jQ%7?8JQtoUHNL|B%3_9moH z*5g3#AHE5}a7^10wS+*&kqgpel^V4Pp&Ukf+(8YwYLtYKMcsPy(;D&)4JL%m;BVtP zE1$d9%Pg2XxQ*wgeBOZyvrx8!ZS(w6bN9uWg)D4pLy}YTw0g|KSQOjlUa-qm4ebk_ zpWX)FvCGqJ=?k6X(8e7$lDj{XX{N#hN5GRCjwH z@Wfa5jFMB*jf|5hMgY^yp%bfDaIv~DpXG4D<3Q@1caAG>#p^BNllO2O#eXykzPqk8 zlKAayoHHUub3HW5p8Cc|+z=u0Iig2mdhdX-@XA;@{Zx!GhhuVNQVi*D4}`93e~?J$ zD0lzEKp0kG{&678I@o~@gg3-2Qdb@9{B?g?K~kJW208?S!4Qmsj?G<;gzC!kECIJXh1nWM{WcCOg6yPcl z_%xH5begG;1>1)c|IQ|y(mNlepucjasc(L%Mn(VlG4xo|;}|12TBE$H_O;EFcH>N< zX7G0iuqfnTYOIkQD<+06tMF zn4gYj^xAJcwLxHwoQl9QuHhVUWV+1+ z^)Fo4x_!4@qttlsfl9j;wnJ|&`DHxacgL=IVapre^dT*1x%U;lxwbu#u+u znQyM#Iq^*M@kq;@lW%;&-ap-+HL{46>5}(n06`F?`Dgde{PtS#lM6aF?@NBJK`AGs zsfmi{@@Z1c8jQ_SeMureK(K|&@y&7}-Ur*Z$ei1Io@l8%ocNaysf{18w(ze$*e-Ly zu3Y~ln|jqIgyQzQDA7z={ox~Gk0{`>`5L=n7Hh@o46J9#;G&=RD?3xKrY;(nZ>Nd@Zb7*5rQz zkGERAK*r1LyuJgsf7NfoGVa@V63tcpAO0CBbKYR5{+I6e1LQs&FC2D;f$urq1t$#K zcdHHQe>h~A1w61e`BYZp-#TZw=bOmU-S4gJP5EZN3QOO__stk`nERbYk)v4#lxnr^ z*q_>%HAa<=vkIZ5eDpvtFIA)2{KN@E-v+iSWZfTROT#| zqk`N)`4PfWsVh|sepL-BxEm>z!B^ICVZ5#KuN>zfjNK>uCZ=a;fa4tfva2-j-&RY! zCq33Yr4t|bU%hD zm8SHgQ;h@>LwRtd@o-GXov%+dIwodv7#$T_NX?IwNlXJ+&}%P^xB})AXt{e>d0JwPmedrV-cqU-1^Mt?1ij8txO^{c#vf z9II}q=v)0_ZRLoV=`{~9;DM2lX7%yQN+zY*(o)M`^_bpFYkIYK=w9Xf#jmfPRqUR7 z;dt4z(AN<+6<_huhNk{aC7h%UQ71nDh^CP+w#6CS;@{jA{*ZU{uiX~^+npZ|yWxSu zv_xO4W&J88T~}LAJBqgzft{Sm%kCWNM>PyU!A+S)6$kATGV#^%%|==Vi|1hT1o%v9N(n)Cug z#pIgbD8iz@6aL3P(DaI52^#DK2C)C+@8322q89m&%mGaKzX7wU%rQ3D{)aDy09#vz z4Hz~+-H6c-7&c(^1Nwl%@CAl1FnodG3v>%GeBoc>3*ed0e;V2C7y@h=Hn3&b0E7fa zKVaAZ9T^z?fMEkhKR_+O=m!iNVBLTb6aOW?Fr_`Xm4n_OC!^@YpXvV&l5dO)VTGSE zBM^jiF~cp)2-rD7ia0Qe6+$9;Fp3>QToz&!7lb@ricvfelDz_>cp)ToHT(qB<%f`Q zX{<~DLYfpYY8iyw+l)~wAjE$MMu|Yk;2w+;hme~67_|mM$cHdW4nn*Mm^?+0@iHck zJTZa~p&DkAz`wx4NU$-ifq2gVQj9?fs&b79%9;sen4&Be2=O#SS)dJFC{Z*EG|L=i zu|Y_<1`@jlOuYljS_C1K6DVs5Xyi$h1uWHr5=BcP zq|6az2?4u1F`+zQJC{=^RrCw6JD>n}IfSI2#<;+qfzB9rC4>wE3UF6J$U_&53v3*J z2IGoCh?gtIT@4`}fC7?%-3!iQTnPw?aKpG#06jnfF2JJ69pg#^^v+>iSqQo8fpGyo z{TNjw4a3R?jm|!B> zS0cSp0__@}7cmO-gnl{&fUm!Vl>wYHeJ~2Z8gUt;0E*|XU=#rDHJt(&)%s#(0FiV* zi~`mV_s1w;X}1830#@yzQ^0~Xfmj)^*1aH%QUq4`YEK^1`f4zG1JK$v1iKSx-A$)} z)(=CmGN5%z7)AlDL$6{K(Awo1Mggt6=oHYpDjX{VT2roL6wo?20;7P|r*B{s(7J<8 z0j(<{u`;0b-J2K%v<{5IsPAj-bc+dgVnNk*if4jPvtZUfoy-J}flvSxAJ3qd0?nQ7 zGr?Zq2B5lE858URNTB?grrKh7Jrw3AB4S1z{A>x1CM_b<2XWGN3Iv z1fzhm{-GEJbRDNtK-IP|tPE&cauuV1qTp+yP{}|~-*AiqYL3wU)M3NBSdi@KMyxjW#<-w$ z1u+;G#KE{&j0*zcFf-Q3qoU5CdLJ!(JKq%0_zWAR1pY`E!h|sgvR_Fj0+Hn z$;Ik|(0DlyJMpl@-!Gp%)utO$@a zj+f2S&Xk~c7<&%{p|y=sfnVGOgipK=y^G~T3hCr~(RyFX`~_0s2TCAL8~FJ79E2f$ zp!hGW5Fe^7j0eR?wkrFm^G#~gNAX{`%zF`%xm*~;Y~jn1IDTFy9DQ;4fA`8Tfh`CCjDEoA z2e#h|^iEDpAfnQP+v#2qeP$sk{W#c_2xbs<=olumvDq&t}kYx zdz3)Z;l0RYFNlt@f@=jWEX}QSEd@mc=`2wLJ;Al&B7#H%f{~tu;95<4L9|3dM9^4= zgx(L{i3pk#%&ZK|34j4=6GT4BGBCCjJRw4_A^E9>_4g}S>zEtppp~>vK)gFlbWM$;_j>t>t5u}Nd mL`i~#oRo}|uJ{^V9fF+1NF&*bj6~45iveB5L|n73oto~N-8MblV|UFi zn`93j6ubzARgw@4f>9z84_-awB3`@*l9M;{2k22xzN+f_=$`4$%iz2V_WOq(c_{NYNf&=Jb-|)LJz}-2jHZ2`S`AD5G7|~M{U`v*|CMw5VKwDyHDV)GY)GKy(GgP-yoSy=fxkspguu8nao+f1m(6B8TO zthgUVuq8_34Qst&1*C)f5w%+EyJ^WbEdeMb_K6qwyKRyx_oG{L4Qrh+B3`y{le31& z08{KSyDGFwkW)tjCOD9T5lv+z4BwST{C(PNDtFZ?H#xq!U4hg@Bm&-e9RrZ_2x;m z9nwv-d6U76q;(gSGkMAk`7KM6i1vcD1+t9 zyj*S`nn?yfiTd5xW6QNwS!fDboTiv{h>(&l@K}8>x0G|XJ@0MV_JYi&K>Blp*{EnG zu+upjs@4rE$#}XF%mN_W>2>a|4*dYGf1|}3x`bjjB@8gh62YnaW-0d5ogSZ?U>}nEP&-Q6wddlR-6-q980RIOPDbeH z?3Fhp41`^nEAy$()tBKZW|rbMj9=Pzbvts+(MHaeFxih0k(g+U0nO=vmgK5!BOq`+gli%LZH zIi^S1Os}$EhAVgnanu3l07Y`he`avXpJt61i{Y5X2BBk$Fx>pPA@8CeK;2S*Gh{PKph4}a` z4*+g*K-mSv zoGL|}uRZ9a=N^>iAI-<|h;IPepFSqV{z_zPj_CmOzm7>WxTO3mK*)8O zcwni23D8b7p#;lIZvdmXs(cgBUKUg>m0kn1{X!`#h?(AQSTO8}dBdLP|6y;N{0)T9 za`{i-LyUGUIjV?XwF_2jyJKp1ox`Haf!oz8Q#%+f2Qb96s5HquYNL_mrV3FXx!btE4!` z%*mOn%*g@lWKQic#{8Y0f0)ru?pZmeK-A&uvM*^tZF!`UajHwkHA-gk@J;Ea!guv0 z1bi@6T1EJl3PAMWTDhj(=)0$^%qsk-?ihs`d&n*-Ah0m(yELW zZsGliH64RF2uN8_&}CJzlt!=rOP}dT^)xXc#j0W)z-t^yrafzxO3&;G-c(;B2QaP* z16>7wN7dlSWz*rc^&@#C535CtJb?d2zk37)%v3dKx KM=u{9@ctj}>K~&3 literal 0 HcmV?d00001 diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro new file mode 100644 index 0000000000000000000000000000000000000000..76465f90e3ae1eb12a2e803ae4e88b04cc078ce2 GIT binary patch literal 4758 zcmbVPU1%It6xNuEf3Y!yTCwVN)apYbGn?IXvl_89sZ>%Gw;^gM%iWo~yQA~7&fLwq z4Y3jmieT|4eNZ0+l|o-cB=t=SQl$9MNAX3Zq9A>dA_%1lo;yGH-r3#RpFZyFo%5ah zopZkP&GY3~Zyeu+*F5)eU#}4xpHS8n+Nu+!pm;=gJwusN9Bi}SHg?Pk@oA}IS_B?f zOkx?nQdnK@I8n#61}CCGKJ`qe3cn3khd4{Fj;ZN7X!*rMsA;0x#XjOvY$28)^37uenGPzqDDmTTqrjRJxQ;z)vV2N@o#-V%=82mQ zY}D+()N5aQdr$}C<)$=SdfDq2>y%vg(NgSzJDI|Hr^eJ?DFd#jnE=DJN1eVDUTfyibcOmGRUio$s z?YYg4ToXKp6tH967|`UV=M3}>${9$FD;Fy!pl{fifYhjRz4+uac?hR;RdE`YrGV^2 zQ;R(hx8QFWb{n*gKW`UDFQ?|?o=#{Y(*bNcJ`5=f9LFuYu0^l|4?>Z{5RH2MI)H<>h6mO*j&4jT zFl{zSL$p-#8~kF$hHYa|40aWQ^_rlw=Y3RJ-CrZEl1(UP`+)aN8k38T?zQT{R)zOa z{2nS>Zn-C^`cE#%exZaz~X4GtcZdRSm<+7@p%V%bG2dPg3 zCL9=1_W!9j%a&KoxswpM!x6WhxllNU1{iS@rRBmsIHo@ zJCQr-bViZy@nBJ!9_-Tsgd%+Ahwwpcph6U_6NNjV6r6;5r#d15k>L6A+?Ev7s0mAvPFY0W>j5VL_M% za(9`+PonucycSPh6UFBI3>}{H6Nq!@X9a>NZ%oh705Lt?;;TXixXv)=C3vY|%ja^* z_1FSe4aeG+Eq@@HGOpaxr)j97bG&nfMGF0xKu>6O?3MairEdOAFq-cI8#pt8nz%E! zX#HTjI5dF@%PRV0Pnxk*P}2dk!fZnDeLjdQ6JTOLB@NinVChaQ;k86uRxdDNVbVPu zm&jFgL7TY}LEFlaWTvOf;xHM(f19C7gGoq0@w$x_@boZD7IUPMUAl$&^&8HgJiL7R zx4X}+TsXD)&(}BqIHTZx$3D2D_Q&DOz2E#hIl6ar_Y==u8hdhc z$NYOMSH@4ilK=aYzrOqM$~)D!cHETvFFP?cw*AP!7XYM)3H$&6 delta 7 OcmeysbBkv~6dM2z8v^qH diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out new file mode 100644 index 00000000000000..a7c41169d8dcb3 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -0,0 +1,123 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !variant_evolution_initial_snapshot -- +1 initial 10 v1 + +-- !variant_evolution_initial_tag -- +1 initial 10 v1 + +-- !variant_evolution_initial_time -- +1 initial 10 v1 + +-- !variant_root_projection -- +1 false {"arr":[1,2],"n":10,"name":"alice","nested":{"city":"hz"},"ok":true,"ratio":1.5} +10 false {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 false {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +2 false {"arr":[3,4],"n":20,"name":"bob","nested":{"city":"sh"},"ok":false,"ratio":2.5} +3 false {"n":30,"name":"same","ok":true} +4 false null +5 true \N +6 false 42 +7 false root-string +8 false {"n":30,"name":"same","ok":true} +9 false {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_path_expressions -- +1 ALICE 11 1.5 true 3 hz +10 DAVE 51 5.5 false 15 sz +11 NULL 61 6.5 true 19 null +2 BOB 21 2.5 false 7 sh +9 CAROL 41 4.5 true 11 bj + +-- !variant_filter -- +11 null 60 +3 same 30 +8 same 30 +9 carol 40 + +-- !variant_cross_file_leaf_projection -- +1 10 +10 50 +11 60 +2 20 +3 30 +4 \N +5 \N +6 \N +7 \N +8 30 +9 40 + +-- !variant_implicit_shredded_filter -- +10 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +9 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_page_pruning_result -- +1095 3001 4095 + +-- !variant_aggregate -- +false 2 70 4 +true 5 170 4.17 + +-- !variant_join -- +10 fifty dave +2 twenty bob + +-- !variant_null_count_distinct -- +11 10 1 9 + +-- !variant_count_pushdown -- +10 11 + +-- !variant_canonical_group -- +{"n":30,"name":"same","ok":true} 2 + +-- !variant_nested_projection -- +1 first {"deep":{"name":"inside"},"x":11} {"kind":"open","score":101} {"enabled":true,"score":1001} +2 second \N null {"enabled":false,"score":2002} + +-- !variant_nested_filter -- +1 + +-- !variant_nested_expressions -- +1 11 inside open \N true 1002 +2 \N \N \N 202 false 2003 + +-- !variant_evolution_renamed_snapshot -- +1 initial 10 v1 +2 renamed 20 v2 + +-- !variant_evolution_renamed_tag -- +1 initial 10 v1 +2 renamed 20 v2 + +-- !variant_evolution_added_reordered -- +1 initial \N v1 +2 renamed \N v2 +3 with-aux 300 v3 + +-- !variant_evolution_dropped -- +1 \N v1 +2 \N v2 +3 300 v3 +4 400 v4 + +-- !variant_evolution_drop_readd -- +1 \N \N \N v1 +2 \N \N \N v2 +3 300 \N \N v3 +4 400 \N \N v4 +5 500 readded 50 v5 + +-- !variant_delete_only_merge -- +0 + +-- !variant_orc_missing_column -- +1 \N + +-- !variant_orc_count_star -- +1 + +-- !variant_mixed_format -- +1 {"format":"parquet"} +2 \N diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy index a5f091298a7a2d..c6d87e32693c45 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy @@ -22,6 +22,7 @@ suite("test_iceberg_varbinary", "p0,external") { logger.info("disable iceberg test.") return } + sql "SET ENABLE_VARIANT_V2=true" String catalog_name_no_mapping = "test_iceberg_no_mapping" String catalog_name_with_mapping = "test_iceberg_with_mapping" @@ -29,13 +30,18 @@ suite("test_iceberg_varbinary", "p0,external") { String rest_port = context.config.otherConfigs.get("iceberg_rest_uri_port") String minio_port = context.config.otherConfigs.get("iceberg_minio_port") String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + // A container-advertised REST URI may differ from the host port-forward address. + String restUri = context.config.otherConfigs.get("iceberg_rest_uri") + if (restUri == null) { + restUri = "http://${externalEnvIp}:${rest_port}" + } sql """drop catalog if exists ${catalog_name_no_mapping}""" sql """ CREATE CATALOG ${catalog_name_no_mapping} PROPERTIES ( 'type'='iceberg', 'iceberg.catalog.type'='rest', - 'uri' = 'http://${externalEnvIp}:${rest_port}', + 'uri' = '${restUri}', "s3.access_key" = "admin", "s3.secret_key" = "password", "s3.endpoint" = "http://${externalEnvIp}:${minio_port}", @@ -50,7 +56,7 @@ suite("test_iceberg_varbinary", "p0,external") { CREATE CATALOG ${catalog_name_with_mapping} PROPERTIES ( 'type'='iceberg', 'iceberg.catalog.type'='rest', - 'uri' = 'http://${externalEnvIp}:${rest_port}', + 'uri' = '${restUri}', "s3.access_key" = "admin", "s3.secret_key" = "password", "s3.endpoint" = "http://${externalEnvIp}:${minio_port}", @@ -166,10 +172,8 @@ suite("test_iceberg_varbinary", "p0,external") { qt_select23 """ select id from test_variant_repro; """ - test { - sql """ - select * from test_variant_repro; - """ - exception "UNSUPPORTED" - } + sql """set enable_file_scanner_v2=true""" + qt_select_variant """ + select id, cast(v as string) from test_variant_repro order by id; + """ } diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy new file mode 100644 index 00000000000000..7b3bc3e833d165 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -0,0 +1,551 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import java.util.regex.Matcher +import java.util.regex.Pattern +import com.amazonaws.auth.AWSStaticCredentialsProvider +import com.amazonaws.auth.BasicAWSCredentials +import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration +import com.amazonaws.services.s3.AmazonS3ClientBuilder +import org.apache.doris.regression.action.ProfileAction + +suite("test_iceberg_variant_read", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + sql "SET ENABLE_VARIANT_V2=true" + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String restUri = context.config.otherConfigs.get("iceberg_rest_uri") + if (restUri == null) { + restUri = "http://${externalEnvIp}:${restPort}" + } + String catalogName = "test_iceberg_variant_read" + String dbName = "iceberg_variant_read_db" + String fixtureKey = "doris-regression/iceberg-variant/iceberg_variant_shredded.parquet" + File shreddedFixture = new File(context.dataPath, "iceberg_variant_shredded.parquet") + File shreddedTableFixture = new File(context.dataPath, "iceberg_variant_shredded_table") + String shreddedMetadataName = + "00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json" + assertTrue(shreddedFixture.isFile(), "Missing shredded Variant Parquet fixture") + assertTrue(shreddedTableFixture.isDirectory(), "Missing shredded Variant table fixture") + def credentials = new BasicAWSCredentials("admin", "password") + def endpoint = new EndpointConfiguration( + "http://${externalEnvIp}:${minioPort}", "us-east-1") + def minioClient = AmazonS3ClientBuilder.standard() + .withEndpointConfiguration(endpoint) + .withPathStyleAccessEnabled(true) + .withCredentials(new AWSStaticCredentialsProvider(credentials)) + .build() + + def latestSnapshotId = { String tableName -> + List> rows = spark_iceberg """ + SELECT snapshot_id + FROM demo.${dbName}.${tableName}.snapshots + ORDER BY committed_at DESC + LIMIT 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + spark_iceberg_multi """ + CREATE NAMESPACE IF NOT EXISTS demo.${dbName}; + DROP TABLE IF EXISTS demo.${dbName}.variant_values; + CREATE TABLE demo.${dbName}.variant_values ( + id INT, + v VARIANT + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.merge.mode'='merge-on-read' + ); + INSERT INTO demo.${dbName}.variant_values VALUES + (1, parse_json('{"name":"alice","n":10,"ratio":1.5,"ok":true,"arr":[1,2],"nested":{"city":"hz"}}')), + (2, parse_json('{"name":"bob","n":20,"ratio":2.5,"ok":false,"arr":[3,4],"nested":{"city":"sh"}}')), + (3, parse_json('{"name":"same","n":30,"ok":true}')), + (4, parse_json('null')), + (5, NULL), + (6, parse_json('42')), + (7, parse_json('"root-string"')); + ALTER TABLE demo.${dbName}.variant_values SET TBLPROPERTIES ( + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_values + WITH (`shred-variants`=true, `variant-inference-buffer-size`=100) VALUES + (8, parse_json('{"ok":true,"n":30,"name":"same"}')), + (9, parse_json('{"name":"carol","n":40,"ratio":4.5,"ok":true,"arr":[5,6],"nested":{"city":"bj"},"new_key":"new"}')), + (10, parse_json('{"name":"dave","n":50,"ratio":5.5,"ok":false,"arr":[7,8],"nested":{"city":"sz"}}')), + (11, parse_json('{"name":null,"n":60,"ratio":6.5,"ok":true,"arr":[9,10],"nested":{"city":null}}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_page_pruning; + + DROP TABLE IF EXISTS demo.${dbName}.variant_nested; + CREATE TABLE demo.${dbName}.variant_nested ( + id INT, + info STRUCT, + events ARRAY, + attrs MAP + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_nested SELECT + 1, + named_struct('label', 'first', 'payload', parse_json('{"x":11,"deep":{"name":"inside"}}')), + array(parse_json('{"kind":"open","score":101}'), parse_json('2')), + map('primary', parse_json('{"enabled":true,"score":1001}')); + INSERT INTO demo.${dbName}.variant_nested SELECT + 2, + named_struct('label', 'second', 'payload', CAST(NULL AS VARIANT)), + array(parse_json('null'), parse_json('{"kind":"close","score":202}')), + map('primary', parse_json('{"enabled":false,"score":2002}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_signed_selector; + CREATE TABLE demo.${dbName}.variant_signed_selector ( + id INT, + v VARIANT + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_signed_selector + VALUES (1, parse_json('{"-1":41}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_evolution; + CREATE TABLE demo.${dbName}.variant_evolution ( + id INT, + payload VARIANT, + note STRING + ) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_evolution + VALUES (1, parse_json('{"stage":"initial","metric":10}'), 'v1'); + + DROP TABLE IF EXISTS demo.${dbName}.variant_write_guard; + CREATE TABLE demo.${dbName}.variant_write_guard (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_write_guard VALUES (1); + """ + + String writeGuardSourceSnapshot = latestSnapshotId("variant_write_guard") + spark_iceberg """ + ALTER TABLE demo.${dbName}.variant_write_guard ADD COLUMN payload VARIANT + """ + + // Register a stable Iceberg metadata fixture so the page-pruning case always uses a + // standards-compliant shredded Variant file, independent of the Spark writer version. + minioClient.putObject("warehouse", fixtureKey, shreddedFixture) + shreddedTableFixture.eachFile { File fixtureFile -> + minioClient.putObject("warehouse", + "wh/${dbName}/variant_page_pruning/metadata/${fixtureFile.name}", fixtureFile) + } + minioClient.shutdown() + spark_iceberg """ + CALL demo.system.register_table( + table => '${dbName}.variant_page_pruning', + metadata_file => + 's3a://warehouse/wh/${dbName}/variant_page_pruning/metadata/${shreddedMetadataName}') + """ + + sql """drop catalog if exists ${catalogName}""" + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='${restUri}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0', + 'meta.cache.iceberg.schema.ttl-second'='0' + ) + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """set enable_file_scanner_v2=true""" + sql """set enable_profile=true""" + sql """set profile_level=2""" + + def profileAction = new ProfileAction(context) + def getProfileByToken = { String token -> + for (int retry = 0; retry < 20; ++retry) { + List profileData = profileAction.getProfileList() + for (final def profileItem in profileData) { + if (profileItem["Sql Statement"].toString().contains(token)) { + return profileAction.getProfile(profileItem["Profile ID"].toString()) + } + } + Thread.sleep(500) + } + throw new IllegalStateException("Missing profile for token: " + token) + } + def counterSum = { String profile, String counterName -> + Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)") + Matcher matcher = pattern.matcher(profile) + long sum = 0 + while (matcher.find()) { + sum += Long.parseLong(matcher.group(1).replace(",", "")) + } + return sum + } + + String evolutionInitial = latestSnapshotId("variant_evolution") + sql """ALTER TABLE variant_evolution CREATE TAG variant_initial""" + + order_qt_variant_evolution_initial_snapshot """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionInitial} + ORDER BY id + """ + + order_qt_variant_evolution_initial_tag """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF 'variant_initial' + ORDER BY id + """ + + List> initialTime = sql """ + SELECT date_format(date_add(committed_at, interval 1 second), '%Y-%m-%d %H:%i:%s') + FROM variant_evolution\$snapshots + WHERE snapshot_id = ${evolutionInitial} + """ + assertEquals(1, initialTime.size()) + order_qt_variant_evolution_initial_time """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR TIME AS OF "${initialTime[0][0]}" + ORDER BY id + """ + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution RENAME COLUMN payload TO event; + INSERT INTO demo.${dbName}.variant_evolution + VALUES (2, parse_json('{"stage":"renamed","metric":20}'), 'v2'); + """ + String evolutionRenamed = latestSnapshotId("variant_evolution") + sql """ALTER TABLE variant_evolution CREATE TAG variant_renamed""" + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution ADD COLUMN aux VARIANT; + ALTER TABLE demo.${dbName}.variant_evolution ALTER COLUMN aux FIRST; + INSERT INTO demo.${dbName}.variant_evolution (id, event, note, aux) + VALUES (3, parse_json('{"stage":"with-aux","metric":30}'), 'v3', + parse_json('{"side":300}')); + """ + String evolutionWithAux = latestSnapshotId("variant_evolution") + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution RENAME COLUMN aux TO sidecar; + ALTER TABLE demo.${dbName}.variant_evolution DROP COLUMN event; + INSERT INTO demo.${dbName}.variant_evolution (id, note, sidecar) + VALUES (4, 'v4', parse_json('{"side":400}')); + """ + String evolutionDropped = latestSnapshotId("variant_evolution") + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution ADD COLUMN event VARIANT; + INSERT INTO demo.${dbName}.variant_evolution (id, note, sidecar, event) + VALUES (5, 'v5', parse_json('{"side":500}'), + parse_json('{"stage":"readded","metric":50}')); + + -- Write ORC before evolving the logical schema to Variant. This retains valid ORC files + -- in the snapshots while using Iceberg FileIO instead of Spark native ORC, whose optional + -- S3A implementation may not be installed. + DROP TABLE IF EXISTS demo.${dbName}.variant_orc; + CREATE TABLE demo.${dbName}.variant_orc (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='orc'); + INSERT INTO demo.${dbName}.variant_orc VALUES (1); + ALTER TABLE demo.${dbName}.variant_orc ADD COLUMN v VARIANT; + + DROP TABLE IF EXISTS demo.${dbName}.variant_mixed_format; + CREATE TABLE demo.${dbName}.variant_mixed_format (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='orc'); + INSERT INTO demo.${dbName}.variant_mixed_format VALUES (2); + ALTER TABLE demo.${dbName}.variant_mixed_format ADD COLUMN v VARIANT; + ALTER TABLE demo.${dbName}.variant_mixed_format SET TBLPROPERTIES + ('write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_mixed_format + VALUES (1, parse_json('{"format":"parquet"}')); + """ + String evolutionReadded = latestSnapshotId("variant_evolution") + + // Root projection covers objects, arrays, scalars, Variant null and SQL NULL. + order_qt_variant_root_projection """ + SELECT id, v IS NULL, CAST(v AS STRING) + FROM variant_values + ORDER BY id + """ + + order_qt_variant_path_expressions """ + SELECT id, + UPPER(CAST(v['name'] AS STRING)), + CAST(v['n'] AS INT) + 1, + ROUND(CAST(v['ratio'] AS DOUBLE), 1), + CAST(v['ok'] AS BOOLEAN), + ARRAY_SUM(CAST(v['arr'] AS ARRAY)), + CAST(v['nested']['city'] AS STRING) + FROM variant_values + WHERE id IN (1, 2, 9, 10, 11) + ORDER BY id + """ + + order_qt_variant_filter """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT) + FROM variant_values + WHERE CAST(v['n'] AS INT) >= 20 + AND CAST(v['ok'] AS BOOLEAN) = true + ORDER BY id + """ + + // The first INSERT is unshredded while the second is shredded. Keep both small files on one + // scanner so their complete and leaf-only physical states must be projected before batching. + sql "set parallel_pipeline_task_num=1" + sql "set max_file_scanners_concurrency=1" + order_qt_variant_cross_file_leaf_projection """ + SELECT id, CAST(v['n'] AS INT) + FROM variant_values + ORDER BY id + """ + + // Keep the root Variant as output while the implicit scalar comparison drives the shredded + // typed_value statistics/page-index path. + order_qt_variant_implicit_shredded_filter """ + SELECT id, CAST(v AS STRING) + FROM variant_values + WHERE v['n'] > 35 + ORDER BY id + """ + + // The query projects the root Variant, while the predicate uses typed_value page metadata. + String pagePruningToken = "iceberg_variant_page_pruning_" + UUID.randomUUID().toString() + sql """ + SELECT '${pagePruningToken}', id, CAST(v AS STRING) + FROM variant_page_pruning + WHERE v['n'] > 3000 + ORDER BY id + """ + String pagePruningProfile = getProfileByToken(pagePruningToken).toString() + assertTrue(counterSum(pagePruningProfile, "FilteredRowsByPage") > 0, + "Shredded Variant typed_value did not filter any Parquet page") + // The predicate_access_paths contract keeps the typed leaf eager while the complete Variant + // root is read through the independent deferred-output projection. + assertTrue(counterSum(pagePruningProfile, "VariantLeafProjections") > 0, + "A root Variant output query did not retain its typed predicate leaf projection") + String leafProjectionToken = + "iceberg_variant_leaf_projection_" + UUID.randomUUID().toString() + sql """ + SELECT '${leafProjectionToken}', COUNT(*) + FROM variant_page_pruning + WHERE v['n'] > 3000 + """ + String leafProjectionProfile = getProfileByToken(leafProjectionToken).toString() + assertTrue(counterSum(leafProjectionProfile, "VariantLeafProjections") > 0, + "Variant typed predicate did not retain a physical leaf projection") + qt_variant_page_pruning_result """ + SELECT COUNT(*), MIN(id), MAX(id) + FROM variant_page_pruning + WHERE v['n'] > 3000 + """ + + order_qt_variant_aggregate """ + SELECT CAST(v['ok'] AS BOOLEAN), + COUNT(*), + SUM(CAST(v['n'] AS INT)), + ROUND(AVG(CAST(v['ratio'] AS DOUBLE)), 2) + FROM variant_values + WHERE v['name'] IS NOT NULL + GROUP BY CAST(v['ok'] AS BOOLEAN) + ORDER BY 1 + """ + + order_qt_variant_join """ + WITH thresholds AS ( + SELECT 20 AS n, 'twenty' AS label + UNION ALL + SELECT 50 AS n, 'fifty' AS label + ) + SELECT t.id, d.label, CAST(t.v['name'] AS STRING) + FROM variant_values t + JOIN thresholds d ON CAST(t.v['n'] AS INT) = d.n + ORDER BY t.id + """ + + qt_variant_null_count_distinct """ + SELECT COUNT(*), COUNT(v), SUM(v IS NULL), COUNT(DISTINCT v) + FROM variant_values + """ + + qt_variant_count_pushdown """ + SELECT COUNT(v), COUNT(*) + FROM variant_values + """ + + order_qt_variant_canonical_group """ + SELECT CAST(v AS STRING), COUNT(*) + FROM variant_values + GROUP BY v + HAVING COUNT(*) > 1 + ORDER BY 1 + """ + + order_qt_variant_nested_projection """ + SELECT id, + info.label, + CAST(info.payload AS STRING), + CAST(events[1] AS STRING), + CAST(element_at(attrs, 'primary') AS STRING) + FROM variant_nested + ORDER BY id + """ + + // Spark may leave nested Variant values unshredded even when top-level shredding is enabled. + // Keep the external-table regression focused on correctness; mapper/reader unit tests use a + // physical typed_value fixture to verify nested leaf projection. + order_qt_variant_nested_filter """ + SELECT id + FROM variant_nested + WHERE CAST(info.payload['x'] AS INT) > 0 + ORDER BY id + """ + + // Signed integer selectors are array indexes, even when a shredded object has a key with the + // same serialized token. The ambiguous scanner path must retain enough state for both results. + List> signedSelectorRows = sql """ + SELECT CAST(v[-1] AS INT), CAST(v['-1'] AS INT) + FROM variant_signed_selector + """ + assertEquals(1, signedSelectorRows.size()) + assertEquals(null, signedSelectorRows[0][0]) + assertEquals("41", signedSelectorRows[0][1].toString()) + + order_qt_variant_nested_expressions """ + SELECT id, + CAST(info.payload['x'] AS INT), + CAST(info.payload['deep']['name'] AS STRING), + CAST(events[1]['kind'] AS STRING), + CAST(events[2]['score'] AS INT), + CAST(element_at(attrs, 'primary')['enabled'] AS BOOLEAN), + CAST(element_at(attrs, 'primary')['score'] AS INT) + 1 + FROM variant_nested + ORDER BY id + """ + + order_qt_variant_evolution_renamed_snapshot """ + SELECT id, CAST(event['stage'] AS STRING), CAST(event['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionRenamed} + ORDER BY id + """ + + order_qt_variant_evolution_renamed_tag """ + SELECT id, CAST(event['stage'] AS STRING), CAST(event['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF 'variant_renamed' + ORDER BY id + """ + + order_qt_variant_evolution_added_reordered """ + SELECT id, + CAST(event['stage'] AS STRING), + CAST(aux['side'] AS INT), + note + FROM variant_evolution FOR VERSION AS OF ${evolutionWithAux} + ORDER BY id + """ + + order_qt_variant_evolution_dropped """ + SELECT id, CAST(sidecar['side'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionDropped} + ORDER BY id + """ + + test { + sql """ + SELECT event + FROM variant_evolution FOR VERSION AS OF ${evolutionDropped} + """ + exception "event" + } + + order_qt_variant_evolution_drop_readd """ + SELECT id, + CAST(sidecar['side'] AS INT), + CAST(event['stage'] AS STRING), + CAST(event['metric'] AS INT), + note + FROM variant_evolution FOR VERSION AS OF ${evolutionReadded} + ORDER BY id + """ + + test { + sql """SELECT payload FROM variant_evolution""" + exception "payload" + } + + test { + sql """ + INSERT INTO variant_write_guard (id) + SELECT id + FROM variant_write_guard FOR VERSION AS OF ${writeGuardSourceSnapshot} + """ + exception "Iceberg VARIANT columns are read-only and cannot be written" + } + + // A delete-only MERGE emits only position deletes. It must remain available even though + // update/insert actions would route the unchanged Variant through the unsupported data writer. + sql """ + MERGE INTO variant_values t + USING (SELECT 11 AS id) s + ON t.id = s.id + WHEN MATCHED THEN DELETE + """ + qt_variant_delete_only_merge "SELECT COUNT(*) FROM variant_values WHERE id = 11" + + // Files written before the Variant field existed have no physical Variant payload. Schema + // evolution must synthesize NULL instead of rejecting their non-Parquet file format. + order_qt_variant_orc_missing_column """ + SELECT id, CAST(v AS STRING) FROM variant_orc ORDER BY id + """ + qt_variant_orc_count_star "SELECT COUNT(*) FROM variant_orc" + order_qt_variant_mixed_format """ + SELECT id, CAST(v AS STRING) FROM variant_mixed_format ORDER BY id + """ + + sql """set enable_file_scanner_v2=false""" + try { + test { + sql """SELECT CAST(v AS STRING) FROM variant_values ORDER BY id""" + exception "legacy file scanner does not support VARIANT" + } + } finally { + sql """set enable_file_scanner_v2=true""" + } +} diff --git a/regression-test/suites/variant_p0/variant_with_rowstore.groovy b/regression-test/suites/variant_p0/variant_with_rowstore.groovy index db83c8ae1158e4..f7bd05f4922f36 100644 --- a/regression-test/suites/variant_p0/variant_with_rowstore.groovy +++ b/regression-test/suites/variant_p0/variant_with_rowstore.groovy @@ -44,7 +44,9 @@ suite("regression_test_variant_rowstore", "variant_type"){ sql """insert into ${table_name} select * from (select -2, '{"a": 11245, "b" : [123, {"xx" : 1}], "c" : {"c" : 456, "d" : "null", "e" : 7.111}}' as json_str union all select -1, '{"a": 1123}' as json_str union all select *, '{"a" : 1234, "xxxx" : "kaana"}' as json_str from numbers("number" = "4096"))t order by 1 limit 4096 ;""" sql "sync" - qt_sql "select * from ${table_name} order by k limit 10" + // Row-store and column-store Variant readers may emit different insignificant JSON spacing. + // Normalize it so this suite continues to verify that both paths preserve the same values. + qt_sql "select k, replace(cast(v as string), ', ', ',') from ${table_name} order by k limit 10" table_name = "multi_var_rs" @@ -60,7 +62,8 @@ suite("regression_test_variant_rowstore", "variant_type"){ properties("replication_num" = "1", "disable_auto_compaction" = "false", "store_row_column" = "true"); """ sql """insert into ${table_name} select k, cast(v as string), cast(v as string) from var_rowstore""" - qt_sql "select * from ${table_name} order by k limit 10" + qt_sql """select k, replace(cast(v as string), ', ', ','), + replace(cast(v1 as string), ', ', ',') from ${table_name} order by k limit 10""" // Parse url def user = context.config.jdbcUser From 4f0f065bf78a7d3b3a2c43709e93371f65fca868 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 12:04:16 +0800 Subject: [PATCH 03/11] [fix](catalog) Safely publish storage adapter snapshots (#66392) ### What problem does this PR solve? Issue Number: None Related PR: #66392 Problem Summary: Master replaced the legacy Hadoop property cache with a shared storage adapter snapshot. Preserve the original fix invariant by publishing an immutable type-keyed map so connector consumers cannot mutate catalog-wide state after publication. Add deterministic coverage for atomic publication and snapshot immutability. ### Release note Prevent connector consumers from modifying shared catalog storage adapter snapshots. ### Check List (For Author) - Test: Unit Test (`CatalogPropertyTest`) - Behavior changed: No. This hardens the existing snapshot contract. - Does this need documentation: No --- .../doris/datasource/CatalogProperty.java | 4 +- .../doris/datasource/CatalogPropertyTest.java | 104 ++++++++++++++++++ 2 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java index 2fb365e9e4db41..b378ee5f426fa2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java @@ -199,7 +199,9 @@ private StorageBindings initStorageAdapters() { throw new IllegalStateException( "Duplicate storage type: " + a.getType()); }, LinkedHashMap::new)); - local = new StorageBindings(ordered, byType); + // Consumers share the published map without locking, so prevent caller-specific + // mutations from changing the catalog-wide snapshot after publication. + local = new StorageBindings(ordered, Collections.unmodifiableMap(byType)); this.storageBindings = local; } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java new file mode 100644 index 00000000000000..6a47944748e4ed --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java @@ -0,0 +1,104 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.datasource.storage.StorageTypeId; + +import org.junit.Assert; +import org.junit.Test; + +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + +public class CatalogPropertyTest { + + @Test + public void testStorageAdaptersArePublishedAfterInitialization() throws Exception { + CountDownLatch initializationStarted = new CountDownLatch(1); + CountDownLatch allowInitialization = new CountDownLatch(1); + CatalogProperty catalogProperty = new CatalogProperty( + null, Collections.singletonMap("fs.defaultFS", "hdfs://test-ns")); + catalogProperty.setPluginDerivedStorageDefaultsSupplier(() -> { + initializationStarted.countDown(); + awaitInitialization(allowInitialization); + return Collections.emptyMap(); + }); + + ExecutorService executor = Executors.newSingleThreadExecutor(); + AtomicReference> readerResult = new AtomicReference<>(); + Thread concurrentReader = new Thread( + () -> readerResult.set(catalogProperty.getStorageAdaptersMap())); + try { + Future> initializer = + executor.submit(catalogProperty::getStorageAdaptersMap); + Assert.assertTrue(initializationStarted.await(5, TimeUnit.SECONDS)); + + concurrentReader.start(); + Assert.assertTrue(waitUntilBlockedOrTerminated(concurrentReader, 5, TimeUnit.SECONDS)); + Assert.assertEquals("The reader must block until initialization publishes the completed map", + Thread.State.BLOCKED, concurrentReader.getState()); + + allowInitialization.countDown(); + Map initialized = initializer.get(5, TimeUnit.SECONDS); + concurrentReader.join(TimeUnit.SECONDS.toMillis(5)); + Assert.assertFalse(concurrentReader.isAlive()); + Assert.assertSame(initialized, readerResult.get()); + } finally { + allowInitialization.countDown(); + concurrentReader.interrupt(); + executor.shutdownNow(); + } + } + + @Test + public void testStorageAdaptersCacheIsImmutable() { + CatalogProperty catalogProperty = new CatalogProperty( + null, Collections.singletonMap("fs.defaultFS", "hdfs://test-ns")); + catalogProperty.setPluginDerivedStorageDefaultsSupplier(Collections::emptyMap); + + Map storageAdapters = catalogProperty.getStorageAdaptersMap(); + Assert.assertThrows(UnsupportedOperationException.class, storageAdapters::clear); + } + + private static boolean waitUntilBlockedOrTerminated(Thread thread, long timeout, TimeUnit timeUnit) { + long deadline = System.nanoTime() + timeUnit.toNanos(timeout); + while (thread.isAlive() && thread.getState() != Thread.State.BLOCKED + && System.nanoTime() < deadline) { + Thread.yield(); + } + return !thread.isAlive() || thread.getState() == Thread.State.BLOCKED; + } + + private static void awaitInitialization(CountDownLatch allowInitialization) { + try { + if (!allowInitialization.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("Timed out waiting to continue storage adapter initialization"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("Interrupted while initializing storage adapters", e); + } + } +} From bbc85c4c479ea57c2a51537f4ad0808de1914c81 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 18:44:15 +0800 Subject: [PATCH 04/11] [fix](iceberg) Harden Variant compatibility and validation Issue Number: None Related PR: #66413 Problem Summary: Review follow-ups found that Variant metadata pruning could bypass an earlier error-producing predicate, late runtime-filter refresh could shift deferred Variant output slots, and mixed-version clusters could schedule unsupported Variant scans or delete-only MERGE plans. The connector SPI version also did not reflect its expanded public surface. In addition, debug Boolean validation filtered and copied large nullable complex columns even when they contained no Boolean values, which could exhaust query memory. This change preserves the safe pruning prefix and scan layout, introduces execution-version compatibility gates, bumps and freezes the connector SPI surface, pins metadata-count checks to the selected snapshot, and skips allocation-heavy Boolean filtering when no Boolean subcolumn exists. Iceberg Variant scans and delete-only MERGE now enforce rolling-upgrade compatibility, and debug column validation avoids copying non-Boolean complex payloads. - Test: Unit Test and Regression Test - Focused ASAN BE unit tests for Boolean validation, Variant scan refresh, metadata pruning, page filtering, and MERGE compatibility - FE compatibility and connector SPI surface unit tests - Generated Iceberg Variant regression golden output - FE Checkstyle and BE clang-format - Behavior changed: Yes. Unsafe metadata pruning and mixed-version Variant execution are rejected or conservatively evaluated, and non-Boolean nullable complex columns are validated without copying their payload. - Does this need documentation: No --- be/src/agent/be_exec_version_manager.cpp | 4 +- be/src/agent/be_exec_version_manager.h | 1 + be/src/core/column/column.cpp | 27 +++++++ be/src/exec/sink/viceberg_merge_sink.cpp | 8 +++ be/src/format_v2/column_mapper.cpp | 12 +++- be/src/format_v2/column_mapper.h | 3 +- be/src/format_v2/file_reader.h | 4 ++ .../format_v2/parquet/parquet_statistics.cpp | 28 +++++--- be/src/format_v2/table_reader.cpp | 57 +++++++-------- be/src/format_v2/table_reader.h | 1 + be/test/core/column/column_self_check.cpp | 70 ++++++++++++++++++- .../exec/sink/viceberg_merge_sink_test.cpp | 22 ++++++ be/test/format_v2/column_mapper_test.cpp | 61 ++++++++++++++++ .../parquet/parquet_statistics_test.cpp | 21 ++++++ be/test/format_v2/table_reader_test.cpp | 42 +++++++++++ .../java/org/apache/doris/common/Config.java | 2 +- .../spi/ConnectorPluginSurfaceTest.java | 9 ++- .../resources/connector-plugin-surface.txt | 32 +++++++++ .../datasource/scan/PluginDrivenScanNode.java | 19 ++++- .../doris/planner/PluginDrivenTableSink.java | 9 +++ ...PluginDrivenScanNodeCompatibilityTest.java | 41 +++++++++++ .../planner/PluginDrivenTableSinkTest.java | 21 ++++++ .../iceberg/test_iceberg_variant_read.out | 3 + .../iceberg/test_iceberg_variant_read.groovy | 16 +++-- 24 files changed, 455 insertions(+), 58 deletions(-) diff --git a/be/src/agent/be_exec_version_manager.cpp b/be/src/agent/be_exec_version_manager.cpp index 4c3bdbd9f31445..bbaa0565a9fa3c 100644 --- a/be/src/agent/be_exec_version_manager.cpp +++ b/be/src/agent/be_exec_version_manager.cpp @@ -126,8 +126,10 @@ void BeExecVersionManager::check_function_compatibility(int current_be_exec_vers // a. use new fixed object serialization way. // 11: start from master // a. enforce Iceberg SQL MERGE cardinality only when every executing BE supports it. +// 12: start from master +// a. support Variant columns and delete-only writer omission in Iceberg SQL MERGE. -const int BeExecVersionManager::max_be_exec_version = 11; +const int BeExecVersionManager::max_be_exec_version = 12; const int BeExecVersionManager::min_be_exec_version = 0; std::map> BeExecVersionManager::_function_change_map {}; std::set BeExecVersionManager::_function_restrict_map; diff --git a/be/src/agent/be_exec_version_manager.h b/be/src/agent/be_exec_version_manager.h index c1a40e35a075e1..4582b84c61d744 100644 --- a/be/src/agent/be_exec_version_manager.h +++ b/be/src/agent/be_exec_version_manager.h @@ -27,6 +27,7 @@ namespace doris { constexpr inline int USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10; constexpr inline int SUPPORT_ICEBERG_MERGE_CARDINALITY_VERSION = 11; +constexpr inline int SUPPORT_ICEBERG_VARIANT_VERSION = 12; class BeExecVersionManager { public: diff --git a/be/src/core/column/column.cpp b/be/src/core/column/column.cpp index 68048b10053b2b..37a5f37c5b19b7 100644 --- a/be/src/core/column/column.cpp +++ b/be/src/core/column/column.cpp @@ -28,6 +28,28 @@ namespace doris { +namespace { + +bool contains_boolean_value_column(const IColumn& column) { + if (const auto* nullable = check_and_get_column(column)) { + return contains_boolean_value_column(nullable->get_nested_column()); + } + if (check_and_get_column(column)) { + return true; + } + + bool contains_boolean = false; + IColumn::ColumnCallback callback = [&](const IColumn& subcolumn) { + if (!contains_boolean && contains_boolean_value_column(subcolumn)) { + contains_boolean = true; + } + }; + column.for_each_subcolumn(callback); + return contains_boolean; +} + +} // namespace + std::string IColumn::dump_structure() const { std::stringstream res; res << get_name() << "(size = " << size(); @@ -64,6 +86,11 @@ bool IColumn::column_boolean_check() const { if (const auto* col_nullable = check_and_get_column(*this)) { // for column nullable, we need to skip null values check const auto& nested_col = col_nullable->get_nested_column(); + // Do not materialize complex payloads that cannot contain Boolean values; filtering a + // multi-GB nested column solely for debug validation can exhaust query memory. + if (!contains_boolean_value_column(nested_col)) { + return true; + } const auto& null_map = col_nullable->get_null_map_data(); Filter not_null_filter; not_null_filter.reserve(nested_col.size()); diff --git a/be/src/exec/sink/viceberg_merge_sink.cpp b/be/src/exec/sink/viceberg_merge_sink.cpp index 5008b217228cd2..0b1e77ee1a47be 100644 --- a/be/src/exec/sink/viceberg_merge_sink.cpp +++ b/be/src/exec/sink/viceberg_merge_sink.cpp @@ -71,6 +71,14 @@ Status VIcebergMergeSink::init_properties(ObjectPool* pool, const RowDescriptor& Status VIcebergMergeSink::open(RuntimeState* state, RuntimeProfile* profile) { _state = state; + if (!_writes_data_files && state->be_exec_version() < SUPPORT_ICEBERG_VARIANT_VERSION) { + // The query-wide version keeps delete-only writer omission all-or-nothing; an older BE + // would ignore writes_data_files and parse the unsupported Variant data-writer schema. + return Status::NotSupported( + "Delete-only Iceberg MERGE requires backend execution version {}", + SUPPORT_ICEBERG_VARIANT_VERSION); + } + _written_rows_counter = ADD_COUNTER(profile, "RowsWritten", TUnit::UNIT); _insert_rows_counter = ADD_COUNTER(profile, "InsertRows", TUnit::UNIT); _delete_rows_counter = ADD_COUNTER(profile, "DeleteRows", TUnit::UNIT); diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index 7e48255e0fcdb1..5c37bc873d1bb0 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -2221,7 +2221,8 @@ Status TableColumnMapper::create_scan_request( const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, RuntimeState* runtime_state, - const std::map* fixed_local_positions) { + const std::map* fixed_local_positions, + const std::map* fixed_non_predicate_positions) { // FileReader evaluates expressions against a file-local block. This mapper owns the // table-column to file-column conversion, so it also owns the file-local block positions. file_request->predicate_columns.clear(); @@ -2235,7 +2236,13 @@ Status TableColumnMapper::create_scan_request( file_request->local_positions = *fixed_local_positions; } file_request->non_predicate_positions.clear(); + if (fixed_non_predicate_positions != nullptr) { + // Deferred output slots are part of the active reader's immutable block layout, just like + // eager slots; retaining only local_positions can shift a later complex root out of bounds. + file_request->non_predicate_positions = *fixed_non_predicate_positions; + } file_request->conjuncts.clear(); + file_request->metadata_pruning_safe_conjunct_count = 0; file_request->delete_conjuncts.clear(); _filter_entries.clear(); // 1. Build referenced non-predicate columns @@ -2462,6 +2469,9 @@ Status TableColumnMapper::localize_filters(const std::vector& table auto localized_conjunct = VExprContext::create_shared(std::move(localized_root)); RETURN_IF_ERROR(rewrite_context.prepare_created_exprs(localized_conjunct.get())); file_request->conjuncts.push_back(std::move(localized_conjunct)); + if (table_filter.metadata_pruning_safe) { + ++file_request->metadata_pruning_safe_conjunct_count; + } for (const auto global_index : table_filter.global_indices) { const auto* mapping = _find_filter_mapping(global_index); if (mapping != nullptr && mapping->file_local_id.has_value() && diff --git a/be/src/format_v2/column_mapper.h b/be/src/format_v2/column_mapper.h index ccfbd090407a52..daa628e1b63a3d 100644 --- a/be/src/format_v2/column_mapper.h +++ b/be/src/format_v2/column_mapper.h @@ -204,7 +204,8 @@ class TableColumnMapper { const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, RuntimeState* runtime_state = nullptr, - const std::map* fixed_local_positions = nullptr); + const std::map* fixed_local_positions = nullptr, + const std::map* fixed_non_predicate_positions = nullptr); // Localize table-level filters to the file schema. // Trivial mappings can copy structured predicates directly. Type changes may be localized with diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 0256b8c1eebcb2..ddbabf6329864d 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -83,6 +84,9 @@ struct FileScanRequest { std::map non_predicate_positions; // Row-level filters converted to file-local expressions from table-level predicates. VExprContextSPtrs conjuncts; + // Only this leading subset may participate in footer/page metadata pruning. The boundary is + // inherited from table-conjunct order so an omitted slotless unsafe expression remains a fence. + size_t metadata_pruning_safe_conjunct_count = std::numeric_limits::max(); // Delete predicates converted to file-local expressions. A TRUE result means that the row is // deleted, so readers must invert each result when building their keep filter. VExprContextSPtrs delete_conjuncts; diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index b4f036773573bb..20c72e8c7360ae 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -585,8 +585,15 @@ std::optional extract_variant_shredded_predicate( .op = *op}; } +VExprContextSPtrs metadata_pruning_conjuncts(const format::FileScanRequest& request) { + const size_t safe_count = + std::min(request.metadata_pruning_safe_conjunct_count, request.conjuncts.size()); + return VExprContextSPtrs(request.conjuncts.begin(), request.conjuncts.begin() + safe_count); +} + bool has_variant_shredded_filter(const format::FileScanRequest& request) { - return std::ranges::any_of(request.conjuncts, [](const auto& conjunct) { + const auto conjuncts = metadata_pruning_conjuncts(request); + return std::ranges::any_of(conjuncts, [](const auto& conjunct) { return extract_variant_shredded_predicate(conjunct).has_value(); }); } @@ -774,9 +781,7 @@ bool variant_statistics_exclude(const VariantShreddedPredicate& predicate, bool has_expr_zonemap_filter(const format::FileScanRequest& request, const RuntimeState*) { // FileScannerV2 metadata pruning is a fixed part of its scan pipeline and must not inherit // the legacy scanner's expression ZoneMap session gate. - // TODO: Fence metadata pruning at the first unsafe/error-preserving conjunct so a later - // ZoneMap predicate cannot bypass its row-level evaluation. - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { if (conjunct != nullptr && conjunct->root() != nullptr && conjunct->root()->can_evaluate_zonemap_filter()) { return true; @@ -970,7 +975,8 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, const std::vector>& file_schema, const format::FileScanRequest& request, ParquetPruningStats* pruning_stats, const cctz::time_zone* timezone) { - const auto slot_indexes = collect_expr_zonemap_slot_indexes(request.conjuncts); + const auto conjuncts = metadata_pruning_conjuncts(request); + const auto slot_indexes = collect_expr_zonemap_slot_indexes(conjuncts); if (slot_indexes.empty()) { return false; } @@ -1005,7 +1011,7 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, } add_slot_zonemap(&ctx, slot_index, column_schema->type, std::move(zone_map)); } - const auto result = VExprContext::evaluate_zonemap_filter(request.conjuncts, ctx); + const auto result = VExprContext::evaluate_zonemap_filter(conjuncts, ctx); accumulate_zonemap_stats(ctx, pruning_stats); return result == ZoneMapFilterResult::kNoMatch; } @@ -1014,7 +1020,7 @@ bool check_shredded_variant_statistics( const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::vector>& file_schema, const format::FileScanRequest& request, const cctz::time_zone* timezone) { - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto predicate = extract_variant_shredded_predicate(conjunct); if (!predicate.has_value()) { continue; @@ -1116,7 +1122,7 @@ ParquetRowGroupPruneReason native_dictionary_prune_reason( return ParquetRowGroupPruneReason::NONE; } const auto conjuncts_by_slot = collect_conjuncts_by_single_slot( - request.conjuncts, expr_zonemap::single_slot_dictionary_index); + metadata_pruning_conjuncts(request), expr_zonemap::single_slot_dictionary_index); for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { const auto file_column_id = file_column_id_by_block_position(request, slot_index); if (!file_column_id.has_value()) { @@ -1185,7 +1191,7 @@ ParquetRowGroupPruneReason native_bloom_filter_prune_reason( return ParquetRowGroupPruneReason::NONE; } const auto conjuncts_by_slot = collect_conjuncts_by_single_slot( - request.conjuncts, expr_zonemap::single_slot_bloom_filter_index); + metadata_pruning_conjuncts(request), expr_zonemap::single_slot_bloom_filter_index); for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { const auto file_column_id = file_column_id_by_block_position(request, slot_index); if (!file_column_id.has_value()) { @@ -1648,7 +1654,7 @@ Status select_row_group_ranges_by_native_page_index( } std::map conjuncts_by_slot; - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto slot_index = expr_zonemap::single_slot_zonemap_index(conjunct); if (slot_index >= 0) { conjuncts_by_slot[slot_index].push_back(conjunct); @@ -1709,7 +1715,7 @@ Status select_row_group_ranges_by_native_page_index( } } - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto predicate = extract_variant_shredded_predicate(conjunct); if (!predicate.has_value()) { continue; diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index 2691e32f8bdbf8..c82d27999a913c 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -1057,8 +1057,15 @@ Status TableReader::_build_table_filters_from_conjuncts() { if (in_safe_prefix && !_is_safe_to_pre_execute(conjunct)) { in_safe_prefix = false; } + const size_t first_new_filter = _table_filters.size(); RETURN_IF_ERROR( build_table_filters_from_conjunct(conjunct, _runtime_state, &_table_filters)); + for (size_t filter_idx = first_new_filter; filter_idx < _table_filters.size(); + ++filter_idx) { + // Preserve the original conjunct-order fence even when the unsafe expression itself + // had no slot and therefore produced no TableFilter entry. + _table_filters[filter_idx].metadata_pruning_safe = in_safe_prefix; + } if (in_safe_prefix) { _constant_pruning_safe_filter_count = _table_filters.size(); } @@ -1068,47 +1075,29 @@ Status TableReader::_build_table_filters_from_conjuncts() { namespace { -bool same_scan_projection(const LocalColumnIndex& lhs, const LocalColumnIndex& rhs) { - if (lhs.index != rhs.index || lhs.project_all_children != rhs.project_all_children || - lhs.children.size() != rhs.children.size()) { +bool same_scan_projections(const std::vector& lhs, + const std::vector& rhs) { + if (lhs.size() != rhs.size()) { return false; } - for (size_t index = 0; index < lhs.children.size(); ++index) { - if (!same_scan_projection(lhs.children[index], rhs.children[index])) { + for (const auto& lhs_projection : lhs) { + const auto rhs_it = std::ranges::find_if(rhs, [&](const LocalColumnIndex& rhs_projection) { + return rhs_projection.column_id() == lhs_projection.column_id(); + }); + if (rhs_it == rhs.end() || !same_local_column_index(lhs_projection, *rhs_it)) { return false; } } return true; } -const LocalColumnIndex* find_scan_projection(const FileScanRequest& request, - LocalColumnId column_id) { - const auto find_by_id = [column_id](const std::vector& projections) { - return std::ranges::find_if(projections, [column_id](const LocalColumnIndex& projection) { - return projection.column_id() == column_id; - }); - }; - auto it = find_by_id(request.predicate_columns); - if (it != request.predicate_columns.end()) { - return &*it; - } - it = find_by_id(request.non_predicate_columns); - return it == request.non_predicate_columns.end() ? nullptr : &*it; -} - bool same_physical_scan_layout(const FileScanRequest& lhs, const FileScanRequest& rhs) { - if (lhs.local_positions != rhs.local_positions) { - return false; - } - for (const auto& [column_id, _] : lhs.local_positions) { - const auto* lhs_projection = find_scan_projection(lhs, column_id); - const auto* rhs_projection = find_scan_projection(rhs, column_id); - if (lhs_projection == nullptr || rhs_projection == nullptr || - !same_scan_projection(*lhs_projection, *rhs_projection)) { - return false; - } - } - return true; + // Deferred complex roots occupy independent output slots. Comparing only eager positions can + // accept a refresh whose second Variant root now aliases or overruns the active block layout. + return lhs.local_positions == rhs.local_positions && + lhs.non_predicate_positions == rhs.non_predicate_positions && + same_scan_projections(lhs.predicate_columns, rhs.predicate_columns) && + same_scan_projections(lhs.non_predicate_columns, rhs.non_predicate_columns); } } // namespace @@ -1137,7 +1126,9 @@ Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { auto refreshed_request = std::make_shared(); RETURN_IF_ERROR(refreshed_mapper->create_scan_request( _table_filters, _projected_columns, refreshed_request.get(), _runtime_state, - _file_scan_request == nullptr ? nullptr : &_file_scan_request->local_positions)); + _file_scan_request == nullptr ? nullptr : &_file_scan_request->local_positions, + _file_scan_request == nullptr ? nullptr + : &_file_scan_request->non_predicate_positions)); // A refresh does not prove that every future runtime filter has arrived. Keep carrier values // available whenever the split started with pending filters. if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index d09bf9e469c190..4d0bd7474f3808 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -82,6 +82,7 @@ using DeleteRows = std::vector; struct TableFilter { VExprContextSPtr conjunct; std::vector global_indices; + bool metadata_pruning_safe = true; }; struct ScanTask { diff --git a/be/test/core/column/column_self_check.cpp b/be/test/core/column/column_self_check.cpp index 4a906cc62a7417..4f9a6f9b47b29a 100644 --- a/be/test/core/column/column_self_check.cpp +++ b/be/test/core/column/column_self_check.cpp @@ -17,9 +17,14 @@ #include +#include "common/config.h" +#include "common/exception.h" #include "core/column/column.h" #include "core/column/column_array.h" #include "core/column/column_const.h" +#include "core/column/column_map.h" +#include "core/column/column_nullable.h" +#include "core/column/column_string.h" #include "core/data_type/data_type_number.h" #include "core/data_type/primitive_type.h" #include "exprs/function/function.h" @@ -28,6 +33,29 @@ namespace doris { +namespace { + +class ScopedMemAllocFaultInjection { +public: + ScopedMemAllocFaultInjection() : _old_probability(config::mem_alloc_fault_probability) { + config::mem_alloc_fault_probability = 1.0; + ++enable_thread_catch_bad_alloc; + } + + ~ScopedMemAllocFaultInjection() { + --enable_thread_catch_bad_alloc; + config::mem_alloc_fault_probability = _old_probability; + } + + ScopedMemAllocFaultInjection(const ScopedMemAllocFaultInjection&) = delete; + ScopedMemAllocFaultInjection& operator=(const ScopedMemAllocFaultInjection&) = delete; + +private: + double _old_probability; +}; + +} // namespace + TEST(ColumnSelfCheckTest, const_check_test) { { ColumnPtr col = ColumnHelper::create_column({1, 2, 3}); @@ -147,4 +175,44 @@ TEST(ColumnSelfCheckTest, boolean_check) { EXPECT_EQ(column_nullable_bool->column_boolean_check(), true); } } -} // namespace doris \ No newline at end of file + +TEST(ColumnSelfCheckTest, non_boolean_complex_payload_does_not_allocate_during_boolean_check) { + auto keys = ColumnString::create(); + keys->insert_data("key", 3); + auto values = ColumnString::create(); + values->insert_data("value", 5); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + auto map = ColumnMap::create(std::move(keys), std::move(values), std::move(offsets)); + auto nullable_map = ColumnNullable::create(std::move(map), ColumnUInt8::create(1, 0)); + + bool is_valid = false; + { + ScopedMemAllocFaultInjection inject_allocation_failure; + EXPECT_NO_THROW(is_valid = nullable_map->column_boolean_check()); + } + EXPECT_TRUE(is_valid); +} + +TEST(ColumnSelfCheckTest, nested_boolean_check_respects_parent_null_map) { + auto create_nullable_map = [](UInt8 null_row_value, UInt8 non_null_row_value) { + auto keys = ColumnString::create(); + keys->insert_data("first", 5); + keys->insert_data("second", 6); + auto values = ColumnUInt8::create(); + values->insert_value(null_row_value); + values->insert_value(non_null_row_value); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + offsets->insert_value(2); + auto map = ColumnMap::create(std::move(keys), std::move(values), std::move(offsets)); + auto null_map = ColumnUInt8::create(); + null_map->insert_value(1); + null_map->insert_value(0); + return ColumnNullable::create(std::move(map), std::move(null_map)); + }; + + EXPECT_TRUE(create_nullable_map(2, 1)->column_boolean_check()); + EXPECT_FALSE(create_nullable_map(1, 2)->column_boolean_check()); +} +} // namespace doris diff --git a/be/test/exec/sink/viceberg_merge_sink_test.cpp b/be/test/exec/sink/viceberg_merge_sink_test.cpp index d1da8fb7bfe1d5..e965fab05e1305 100644 --- a/be/test/exec/sink/viceberg_merge_sink_test.cpp +++ b/be/test/exec/sink/viceberg_merge_sink_test.cpp @@ -473,6 +473,28 @@ TEST_F(VIcebergMergeSinkTest, TestRollingUpgradeSkipsCardinalityState) { EXPECT_TRUE(sink->_matched_row_positions.empty()); } +TEST_F(VIcebergMergeSinkTest, TestRollingUpgradeRejectsDeleteOnlyWriterOmission) { + ObjectPool pool; + MockRuntimeState state; + state.set_be_exec_version(SUPPORT_ICEBERG_VARIANT_VERSION - 1); + + DataTypes types {std::make_shared(), + std::make_shared(DataTypes {std::make_shared(), + std::make_shared()}, + Strings {"file_path", "row_position"}), + std::make_shared(), std::make_shared()}; + MockRowDescriptor row_desc(types, &pool); + auto output_exprs = build_output_exprs(&pool, &state, row_desc); + auto t_sink = build_sink(); + t_sink.iceberg_merge_sink.__set_writes_data_files(false); + auto sink = std::make_shared(t_sink, output_exprs, nullptr, nullptr); + + ASSERT_TRUE(sink->init_properties(&pool, row_desc).ok()); + RuntimeProfile profile("rolling_upgrade_delete_only_iceberg_merge_sink"); + const Status status = sink->open(&state, &profile); + EXPECT_TRUE(status.is()) << status; +} + TEST_F(VIcebergMergeSinkTest, TestErrorCloseRemovesRolledDataFiles) { ObjectPool pool; MockRuntimeState state; diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index ab644372333081..b53164bac4ae28 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -4118,6 +4118,67 @@ TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredVariantRootProjection) EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); } +TEST(ColumnMapperTest, RowGroupRefreshPreservesTwoDeferredVariantRootLayouts) { + auto make_file_variant = [](const std::string& name, int32_t field_id, int32_t local_id) { + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, + 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto variant = field_id_col(name, field_id, variant_v2(), local_id); + variant.children = {name_col("metadata", varbinary(), 0), name_col("value", varbinary(), 1), + std::move(typed_value)}; + return variant; + }; + auto first = field_id_col("v1", 10, variant_v2()); + first.has_predicate_access_paths = true; + first.predicate_variant_access_paths = {{"typed_col"}}; + auto second = field_id_col("v2", 11, variant_v2()); + second.has_predicate_access_paths = true; + second.predicate_variant_access_paths = {{"typed_col"}}; + const std::vector table_columns {first, second}; + const std::vector file_columns {make_file_variant("v1", 10, 0), + make_file_variant("v2", 11, 1)}; + + std::vector filters; + for (int32_t index = 0; index < 2; ++index) { + auto typed_col = element_at( + table_slot(index, index, table_columns[index].type, table_columns[index].name), + variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i64()), + literal(i64(), Field::create_field(0))); + filters.push_back({.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(index)}}); + } + + ParquetColumnMapper initial_mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(initial_mapper.create_mapping(table_columns, {}, file_columns).ok()); + FileScanRequest initial_request; + ASSERT_TRUE(initial_mapper.create_scan_request(filters, table_columns, &initial_request).ok()); + EXPECT_EQ(initial_request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(initial_request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_EQ(initial_request.local_positions.at(LocalColumnId(1)), LocalIndex(2)); + EXPECT_EQ(initial_request.non_predicate_position(LocalColumnId(1)), LocalIndex(3)); + + ParquetColumnMapper refreshed_mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(refreshed_mapper.create_mapping(table_columns, {}, file_columns).ok()); + FileScanRequest refreshed_request; + ASSERT_TRUE(refreshed_mapper + .create_scan_request(filters, table_columns, &refreshed_request, nullptr, + &initial_request.local_positions, + &initial_request.non_predicate_positions) + .ok()); + EXPECT_EQ(refreshed_request.local_positions, initial_request.local_positions); + EXPECT_EQ(refreshed_request.non_predicate_positions, initial_request.non_predicate_positions); + ASSERT_EQ(refreshed_request.predicate_columns.size(), 2); + ASSERT_EQ(refreshed_request.non_predicate_columns.size(), 2); + for (size_t index = 0; index < 2; ++index) { + EXPECT_TRUE(same_local_column_index(refreshed_request.predicate_columns[index], + initial_request.predicate_columns[index])); + EXPECT_TRUE(same_local_column_index(refreshed_request.non_predicate_columns[index], + initial_request.non_predicate_columns[index])); + } +} + TEST(ColumnMapperTest, NestedVariantAccessPathProjectsPhysicalTypedLeaf) { auto table_variant = field_id_col("payload", 2, variant_v2()); table_variant.variant_access_paths = {{"typed_col"}}; diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 1a13e8ecea362e..4c3fdcafa06ffe 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -987,6 +987,17 @@ TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) .ok()); EXPECT_TRUE(selected_row_groups.empty()); + // The same predicate can be localized after an earlier unsafe conjunct. Metadata pruning must + // preserve that earlier expression's row-level error instead of skipping the whole row group. + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + auto leaf_projection = format::LocalColumnIndex::partial_local(0); auto typed_object_projection = format::LocalColumnIndex::partial_local(2); auto field_projection = format::LocalColumnIndex::partial_local(0); @@ -1083,6 +1094,16 @@ TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) EXPECT_EQ(pruning_stats.page_index_read_calls, 1); EXPECT_EQ(pruning_stats.filtered_page_rows, 50); + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + // Direct Variant numeric comparisons coerce integral literals to a wide DECIMAL domain. request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( diff --git a/be/test/format_v2/table_reader_test.cpp b/be/test/format_v2/table_reader_test.cpp index 219f2bff269e38..19d2f977cb2255 100644 --- a/be/test/format_v2/table_reader_test.cpp +++ b/be/test/format_v2/table_reader_test.cpp @@ -1569,6 +1569,48 @@ TEST(TableReaderTest, ConstantPruningStopsAtUnsafeSlotlessPredicate) { ASSERT_TRUE(reader.close().ok()); } +TEST(TableReaderTest, MetadataPruningBoundaryKeepsUnsafeSlotlessBarrier) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + bool predicate_executed = false; + auto fake_state = std::make_shared(); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE( + reader + .init({ + .projected_columns = projected_columns, + .conjuncts = + {prepared_conjunct( + &state, + std::make_shared( + &predicate_executed)), + prepared_conjunct(&state, + table_int32_greater_than_expr(0, 0, 10))}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + + SplitReadOptions split; + split.current_range.__set_path("fake-table-reader-input"); + ASSERT_TRUE(reader.prepare_split(split).ok()); + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + ASSERT_NE(fake_state->last_request, nullptr); + ASSERT_EQ(fake_state->last_request->conjuncts.size(), 1); + EXPECT_EQ(fake_state->last_request->metadata_pruning_safe_conjunct_count, 0); + ASSERT_TRUE(reader.close().ok()); +} + TEST(TableReaderTest, CanUseInjectedFileReaderForStandaloneUnitTest) { std::vector file_schema; file_schema.push_back(make_file_column(0, "id", std::make_shared())); diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java index 765baf5393d1b2..5765f68fe7f6f5 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java +++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java @@ -2036,7 +2036,7 @@ public class Config extends ConfigBase { * Max data version of backends serialize block. */ @ConfField(mutable = false) - public static int max_be_exec_version = 11; + public static int max_be_exec_version = 12; /** * Min data version of backends serialize block. diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index cfe2fc607999af..0a6afdaa5bc5b4 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -18,6 +18,8 @@ package org.apache.doris.connector.spi; import org.apache.doris.connector.spi.handle.ConnectorColumnHandle; +import org.apache.doris.connector.spi.handle.ConnectorWriteHandle; +import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.spi.write.ConnectorWritePlanProvider; import org.junit.jupiter.api.Assertions; @@ -71,9 +73,8 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException Assertions.assertNotNull(in, "missing connector plugin API version resource"); version.load(in); } - // ConnectorWritePlanProvider and ConnectorColumnHandle gained public default methods in this - // surface revision. A plugin built against major 3 must be refused rather than silently run - // against an expanded contract it did not compile against. + // API 4 covers both the SPI namespace migration and the expanded column/write contracts. + // Older plugins must be rejected instead of failing later on missing types or methods. Assertions.assertEquals("4.0", version.getProperty("api.version")); } @@ -83,6 +84,8 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException ConnectorContext.class, Connector.class, ConnectorColumnHandle.class, + ConnectorScanPlanProvider.class, + ConnectorWriteHandle.class, ConnectorWritePlanProvider.class, org.apache.doris.extension.spi.Plugin.class, org.apache.doris.extension.spi.PluginFactory.class, diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt index 7e2854758a24e1..14174dbe23aa78 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt @@ -45,6 +45,38 @@ org.apache.doris.connector.spi.ConnectorProvider#validatePropertiesForUpdate(jav org.apache.doris.connector.spi.handle.ConnectorColumnHandle#equals(java.lang.Object):boolean org.apache.doris.connector.spi.handle.ConnectorColumnHandle#hashCode():int org.apache.doris.connector.spi.handle.ConnectorColumnHandle#withProjectedFieldIds(java.util.Set):org.apache.doris.connector.spi.handle.ConnectorColumnHandle +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getBranchName():java.util.Optional +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getColumns():java.util.List +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getSortInfo():org.apache.doris.thrift.TSortInfo +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getStaticPartitionSpec():java.util.Map +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getTableHandle():org.apache.doris.connector.spi.handle.ConnectorTableHandle +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getWriteOperation():org.apache.doris.connector.spi.handle.WriteOperation +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isOverwrite():boolean +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isRequireMergeCardinalityCheck():boolean +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isWritesDataFiles():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#adjustFileCompressType(org.apache.doris.thrift.TFileCompressType):org.apache.doris.thrift.TFileCompressType +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,java.util.Map):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#canServeMetadataOnlyCount(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#classifyColumn(java.lang.String):org.apache.doris.connector.spi.scan.ConnectorColumnCategory +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#collectScanProfiles(org.apache.doris.connector.spi.ConnectorSession):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getDeleteFiles(org.apache.doris.thrift.TTableFormatFileDesc):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getMustReadColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):java.util.Set +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getScanNodeProperties(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional):java.util.Map +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getScanNodePropertiesResult(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional):org.apache.doris.connector.spi.scan.ScanNodePropertiesResult +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#ignorePartitionPruneShortCircuit():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScan(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScanForPartitionBatch(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest,java.util.List):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#populateScanLevelParams(org.apache.doris.thrift.TFileScanRangeParams,java.util.Map):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#releaseReadTransaction(java.lang.String):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#scannedPartitionCount(java.util.List):java.util.OptionalLong +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#streamingSplitEstimate(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional,boolean):long +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#streamSplits(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional,long):org.apache.doris.connector.spi.scan.ConnectorSplitSource +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsBatchScan(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsFileCache():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableIncrementalRead(java.lang.String):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableOptions(java.lang.String):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableTimeTravel():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsTableSample():boolean org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorWriteHandle):void org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getSyntheticWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):java.util.List org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional):java.util.Optional diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index 6f51288d5ceed9..3437b453bf7573 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -34,6 +34,7 @@ import org.apache.doris.catalog.TableIf; import org.apache.doris.catalog.Type; import org.apache.doris.catalog.VariantType; +import org.apache.doris.common.Config; import org.apache.doris.common.UserException; import org.apache.doris.common.profile.RuntimeProfile; import org.apache.doris.common.profile.SummaryProfile; @@ -129,6 +130,7 @@ * */ public class PluginDrivenScanNode extends FileQueryScanNode { + private static final int SUPPORT_ICEBERG_VARIANT_EXEC_VERSION = 12; private static final Logger LOG = LogManager.getLogger(PluginDrivenScanNode.class); @@ -200,6 +202,9 @@ public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, @Override protected void doInitialize() throws UserException { super.doInitialize(); + // Compatibility must inspect the snapshot-specific handle: latest metadata may answer + // COUNT(*) while an older time-travel snapshot still requires a Variant data scan. + pinMvccSnapshot(); checkVariantBackendCompatibilityForCurrentScan(backendPolicy.getBackends()); } @@ -209,13 +214,17 @@ void checkVariantBackendCompatibilityForCurrentScan(Iterable backends) ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (isTableLevelCountStarPushdown() && conjuncts.isEmpty() && scanProvider != null) { metadataCountProven = onPluginClassLoader(scanProvider, - () -> scanProvider.canServeMetadataOnlyCount( - connectorSession, currentHandle, Optional.empty())); + () -> canServeMetadataOnlyCount(scanProvider, connectorSession, currentHandle)); } checkVariantBackendCompatibility( !metadataCountProven && projectsComputeVariant(desc), backends); } + static boolean canServeMetadataOnlyCount(ConnectorScanPlanProvider scanProvider, + ConnectorSession session, ConnectorTableHandle handle) { + return scanProvider.canServeMetadataOnlyCount(session, handle, Optional.empty()); + } + static boolean projectsComputeVariant(TupleDescriptor tuple) { // Nested-column pruning updates the effective slot type but deliberately keeps the original // Column metadata; compatibility must follow the payload this scan actually projects. @@ -245,6 +254,12 @@ static void checkVariantBackendCompatibility(boolean projectsVariant, Iterable */ public class PluginDrivenTableSink extends BaseExternalTableDataSink { + private static final int SUPPORT_ICEBERG_VARIANT_EXEC_VERSION = 12; private final PluginDrivenExternalTable targetTable; // Plan-provider mode (W5): the connector builds its own opaque TDataSink via planWrite(). @@ -192,6 +194,13 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { @Override public void bindDataSink(Optional insertCtx) throws AnalysisException { + if (writeOperation == WriteOperation.MERGE && !writesDataFiles + && Config.be_exec_version < SUPPORT_ICEBERG_VARIANT_EXEC_VERSION) { + // Older BEs ignore writes_data_files and instantiate the omitted data writer, so reject + // the all-or-nothing query before a Variant schema reaches any rolling-upgrade backend. + throw new AnalysisException("Delete-only Iceberg MERGE with Variant is unavailable " + + "during rolling upgrade"); + } boolean overwrite = false; Map writeContext = Collections.emptyMap(); Optional branchName = Optional.empty(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java index 0c8cbd43cc1b10..e25cd39b7ef699 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java @@ -17,17 +17,24 @@ package org.apache.doris.datasource.scan; +import org.apache.doris.common.Config; import org.apache.doris.common.UserException; +import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; import org.apache.doris.system.Backend; import org.junit.Assert; import org.junit.Test; import java.util.Collections; +import java.util.Optional; /** Tests the mixed-version safety gate for plugin-driven Variant scans. */ public class PluginDrivenScanNodeCompatibilityTest { + private static final int VARIANT_EXEC_VERSION = 12; + @Test public void computeVariantRejectsSmoothUpgradeSourceBackend() { Backend backend = new Backend(7L, "127.0.0.1", 9050); @@ -47,4 +54,38 @@ public void compatibilityCheckIgnoresScansWithoutComputeVariant() throws UserExc PluginDrivenScanNode.checkVariantBackendCompatibility( false, Collections.singletonList(backend)); } + + @Test + public void computeVariantRejectsOldQueryWideExecutionVersion() { + int original = Config.be_exec_version; + try { + Config.be_exec_version = VARIANT_EXEC_VERSION - 1; + Backend backend = new Backend(8L, "127.0.0.1", 9050); + + UserException exception = Assert.assertThrows(UserException.class, + () -> PluginDrivenScanNode.checkVariantBackendCompatibility( + true, Collections.singletonList(backend))); + Assert.assertTrue(exception.getMessage().contains("execution version")); + } finally { + Config.be_exec_version = original; + } + } + + @Test + public void metadataCountCapabilityUsesPinnedHandle() { + ConnectorSession session = org.mockito.Mockito.mock(ConnectorSession.class); + ConnectorTableHandle latest = new ConnectorTableHandle() { }; + ConnectorTableHandle pinned = new ConnectorTableHandle() { }; + ConnectorScanPlanProvider provider = + org.mockito.Mockito.mock(ConnectorScanPlanProvider.class); + org.mockito.Mockito.doAnswer(invocation -> invocation.getArgument(1) == latest) + .when(provider).canServeMetadataOnlyCount(org.mockito.Mockito.same(session), + org.mockito.Mockito.any(ConnectorTableHandle.class), + org.mockito.Mockito.eq(Optional.empty())); + + Assert.assertTrue(PluginDrivenScanNode.canServeMetadataOnlyCount( + provider, session, latest)); + Assert.assertFalse(PluginDrivenScanNode.canServeMetadataOnlyCount( + provider, session, pinned)); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java index 6a75bdcb8938b9..527f5d4582f966 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java @@ -18,6 +18,7 @@ package org.apache.doris.planner; import org.apache.doris.common.AnalysisException; +import org.apache.doris.common.Config; import org.apache.doris.connector.spi.ConnectorColumn; import org.apache.doris.connector.spi.ConnectorSession; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; @@ -239,6 +240,26 @@ null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), Assert.assertTrue(provider.seenHandle.isRequireMergeCardinalityCheck()); } + @Test + public void deleteOnlyMergeRejectsOldQueryWideExecutionVersion() { + int original = Config.be_exec_version; + try { + Config.be_exec_version = 11; + RecordingWritePlanProvider provider = new RecordingWritePlanProvider( + new ConnectorSinkPlan(new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK))); + PluginDrivenTableSink sink = new PluginDrivenTableSink( + null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), + null, WriteOperation.MERGE, false, true); + + AnalysisException exception = Assert.assertThrows(AnalysisException.class, + () -> sink.bindDataSink(Optional.empty())); + Assert.assertTrue(exception.getMessage().contains("rolling upgrade")); + Assert.assertNull(provider.seenHandle); + } finally { + Config.be_exec_version = original; + } + } + @Test public void getExplainStringThreadsWriteOperationToHandle() { // WHY: EXPLAIN of a post-flip MERGE/DELETE builds a (degraded) handle for appendExplainInfo; the diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out index a7c41169d8dcb3..0157263d8a83fc 100644 --- a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -79,6 +79,9 @@ true 5 170 4.17 -- !variant_nested_filter -- 1 +-- !variant_signed_selector -- +\N 41 + -- !variant_nested_expressions -- 1 11 inside open \N true 1002 2 \N \N \N 202 false 2003 diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index 7b3bc3e833d165..43c6735a6dad74 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -377,6 +377,17 @@ suite("test_iceberg_variant_read", WHERE v['n'] > 3000 """ + // A later Variant metadata predicate must not prune away an earlier error-producing conjunct. + test { + sql """ + SELECT COUNT(*) + FROM variant_page_pruning + WHERE assert_true(id != 1, 'variant_metadata_error_barrier') + AND v['n'] > 5000 + """ + exception "variant_metadata_error_barrier" + } + order_qt_variant_aggregate """ SELECT CAST(v['ok'] AS BOOLEAN), COUNT(*), @@ -440,13 +451,10 @@ suite("test_iceberg_variant_read", // Signed integer selectors are array indexes, even when a shredded object has a key with the // same serialized token. The ambiguous scanner path must retain enough state for both results. - List> signedSelectorRows = sql """ + order_qt_variant_signed_selector """ SELECT CAST(v[-1] AS INT), CAST(v['-1'] AS INT) FROM variant_signed_selector """ - assertEquals(1, signedSelectorRows.size()) - assertEquals(null, signedSelectorRows[0][0]) - assertEquals("41", signedSelectorRows[0][1].toString()) order_qt_variant_nested_expressions """ SELECT id, From 6851120dbcb5ba060da4e608ddeb90cc794b2c66 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 14:33:36 +0800 Subject: [PATCH 05/11] test: expand Iceberg Variant read coverage --- .../parquet/variant_column_reader_test.cpp | 718 ++++++++++++++++++ .../iceberg/test_iceberg_variant_read.out | 59 +- .../iceberg/test_iceberg_variant_read.groovy | 444 ++++++++++- 3 files changed, 1196 insertions(+), 25 deletions(-) diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 40c18635aad339..5bf2d33c10f1af 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -20,20 +20,33 @@ #include #include +#include +#include +#include +#include #include #include +#include "common/exception.h" #include "core/assert_cast.h" #include "core/column/column_array.h" +#include "core/column/column_decimal.h" +#include "core/column/column_map.h" #include "core/column/column_nullable.h" #include "core/column/column_string.h" #include "core/column/column_struct.h" #include "core/column/variant_v2/column_variant_v2.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_date_or_datetime_v2.h" +#include "core/data_type/data_type_decimal.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_variant_v2.h" +#include "core/value/timestamptz_value.h" #include "core/value/variant/variant_batch_builder.h" #include "core/value/variant/variant_parquet_encoding.h" #include "exprs/function/function_variant_element_v2.h" @@ -90,6 +103,16 @@ ParquetColumnSchema shredded_int64_schema() { return schema; } +ParquetColumnSchema shredded_primitive_schema(DataTypePtr type) { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + typed->type = make_nullable(std::move(type)); + schema.children.push_back(std::move(typed)); + return schema; +} + ParquetColumnSchema shredded_object_schema() { auto schema = unshredded_schema(); auto typed = std::make_unique(); @@ -110,6 +133,12 @@ ParquetColumnSchema shredded_object_schema() { return schema; } +ParquetColumnSchema shredded_named_object_schema(std::string field_name) { + auto schema = shredded_object_schema(); + schema.children.back()->children[0]->name = std::move(field_name); + return schema; +} + ParquetColumnSchema shredded_binary_object_schema() { auto schema = shredded_object_schema(); auto* leaf = schema.children.back()->children[0]->children[0].get(); @@ -136,6 +165,17 @@ ParquetColumnSchema shredded_array_schema() { return schema; } +ParquetColumnSchema shredded_mixed_array_schema() { + auto schema = shredded_array_schema(); + auto* element = schema.children.back()->children[0].get(); + auto value = std::make_unique(); + value->name = "value"; + value->kind = ParquetColumnSchemaKind::PRIMITIVE; + value->type = make_nullable(std::make_shared()); + element->children.insert(element->children.begin(), std::move(value)); + return schema; +} + MutableColumnPtr shredded_int64_physical(const std::vector& values) { const std::array ignored {0}; const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); @@ -157,6 +197,20 @@ MutableColumnPtr shredded_int64_physical(const std::vector& values) { return ColumnNullable::create(std::move(structure), std::move(root_nulls)); } +MutableColumnPtr shredded_primitive_physical(MutableColumnPtr typed) { + const size_t rows = typed->size(); + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings(std::vector(rows, metadata), + std::vector(rows, 0))); + fields.push_back(nullable_strings(std::vector(rows, {ignored.data(), 0}), + std::vector(rows, 1))); + fields.push_back(std::move(typed)); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), + ColumnUInt8::create(rows, 0)); +} + MutableColumnPtr projected_shredded_object_physical(const std::vector& values, const IColumn** decoded_leaf = nullptr) { auto integers = ColumnInt64::create(); @@ -182,6 +236,37 @@ MutableColumnPtr projected_shredded_object_physical(const std::vector& return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); } +MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); +MutableColumnPtr nullable_int64(const std::vector& values, + const std::vector& nulls); + +MutableColumnPtr complete_shredded_object_physical(std::string_view residual_key, + int64_t residual_value, int64_t typed_value) { + VariantBatchBuilder builder; + auto row = builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef(residual_key.data(), residual_key.size())); + row.add_int(residual_value); + object.finish(); + row.finish(); + VariantBatchBuilder batch = builder.finish_batch(); + const VariantRef residual = batch.value_at(0); + + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({typed_value}, {0})); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(wrapper_fields)), + ColumnUInt8::create(1, 0))); + MutableColumns root_fields; + root_fields.push_back( + nullable_strings({StringRef(residual.metadata.data, residual.metadata.size)}, {0})); + root_fields.push_back( + nullable_strings({StringRef(residual.value.data, residual.value.size)}, {0})); + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return root_wrapper(std::move(root_fields)); +} + MutableColumnPtr projected_two_field_object_physical(const std::vector& first, const std::vector& second) { DORIS_CHECK(first.size() == second.size()); @@ -226,6 +311,58 @@ MutableColumnPtr projected_wide_object_physical(size_t field_count, int64_t valu ColumnUInt8::create(1, 0)); } +std::string materialization_error(const ParquetColumnSchema& schema, ColumnPtr physical) { + auto output = make_nullable(std::make_shared())->create_column(); + const Status status = materialize_variant_rows(schema, std::move(physical), output); + if (!status.ok()) { + return status.to_string(); + } + try { + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + (void)variants.get_value_ref(0); + } catch (const Exception& exception) { + return exception.what(); + } + return {}; +} + +MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls) { + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(root_nulls.begin(), root_nulls.end()); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), std::move(null_map)); +} + +MutableColumnPtr nullable_int64(const std::vector& values, + const std::vector& nulls) { + auto data = ColumnInt64::create(); + data->get_data().assign(values.begin(), values.end()); + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(nulls.begin(), nulls.end()); + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +template +MutableColumnPtr nullable_fixed(std::initializer_list values, + std::initializer_list nulls) { + auto data = ColumnType::create(); + for (const Value& value : values) { + data->insert_value(value); + } + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(nulls.begin(), nulls.end()); + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +template +MutableColumnPtr nullable_decimal(uint32_t scale, std::initializer_list values) { + auto data = ColumnType::create(0, scale); + for (const Value& value : values) { + data->insert_value(value); + } + return ColumnNullable::create(std::move(data), ColumnUInt8::create(values.size(), 0)); +} + } // namespace TEST(VariantColumnReaderTest, UnshreddedRowsPreserveSqlNullAndVariantNull) { @@ -304,6 +441,177 @@ TEST(VariantColumnReaderTest, ShreddedIntegerKeepsDeclaredPhysicalWidth) { EXPECT_EQ(variants.get_value_ref(0).primitive_id(), VariantPrimitiveId::INT64); } +TEST(VariantColumnReaderTest, ReconstructsShreddedPrimitiveTypeMatrix) { + auto decode = [](ParquetColumnSchema schema, MutableColumnPtr typed, + const std::function& verify) { + auto output = make_nullable(std::make_shared())->create_column(); + const Status status = materialize_variant_rows( + schema, shredded_primitive_physical(std::move(typed)), output); + ASSERT_TRUE(status.ok()) << status; + verify(assert_cast( + assert_cast(*output).get_nested_column())); + }; + + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({0, 1}, {0, 0}), [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).primitive_id(), VariantPrimitiveId::FALSE_VALUE); + EXPECT_EQ(values.get_value_ref(1).primitive_id(), VariantPrimitiveId::TRUE_VALUE); + }); + + auto verify_integer = [&](DataTypePtr type, MutableColumnPtr typed, int width, int64_t first, + int64_t second) { + auto schema = shredded_primitive_schema(std::move(type)); + schema.children.back()->type_descriptor.integer_bit_width = width; + decode(std::move(schema), std::move(typed), [&](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_int(), first); + EXPECT_EQ(values.get_value_ref(1).get_int(), second); + }); + }; + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 8, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 16, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 32, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 64, std::numeric_limits::min(), std::numeric_limits::max()); + + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({std::numeric_limits::quiet_NaN(), + std::numeric_limits::infinity()}, + {0, 0}), + [](const auto& values) { + EXPECT_TRUE(std::isnan(values.get_value_ref(0).get_float())); + EXPECT_TRUE(std::isinf(values.get_value_ref(1).get_float())); + }); + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({-std::numeric_limits::infinity(), 1.25}, + {0, 0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_double(), + -std::numeric_limits::infinity()); + EXPECT_EQ(values.get_value_ref(1).get_double(), 1.25); + }); + + { + auto schema = shredded_primitive_schema(std::make_shared(9, 2)); + schema.children.back()->type_descriptor.decimal_precision = 9; + schema.children.back()->type_descriptor.decimal_scale = 2; + decode(std::move(schema), + nullable_decimal(2, {Decimal32 {12345}, Decimal32 {-1}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), (VariantDecimal {12345, 2, 4})); + EXPECT_EQ(values.get_value_ref(1).get_decimal(), (VariantDecimal {-1, 2, 4})); + }); + } + { + auto schema = shredded_primitive_schema(std::make_shared(18, 3)); + schema.children.back()->type_descriptor.decimal_precision = 18; + schema.children.back()->type_descriptor.decimal_scale = 3; + decode(std::move(schema), + nullable_decimal( + 3, {Decimal64 {123456789}, Decimal64 {-123456789}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), + (VariantDecimal {123456789, 3, 8})); + EXPECT_EQ(values.get_value_ref(1).get_decimal(), + (VariantDecimal {-123456789, 3, 8})); + }); + } + { + auto schema = shredded_primitive_schema(std::make_shared(38, 4)); + schema.children.back()->type_descriptor.decimal_precision = 38; + schema.children.back()->type_descriptor.decimal_scale = 4; + decode(std::move(schema), + nullable_decimal( + 4, {Decimal128V3 {static_cast(1234567890123456789LL)}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), + (VariantDecimal {1234567890123456789LL, 4, 16})); + }); + } + + const auto date = DateV2Value::create_from_olap_date( + (static_cast(1970) << 9) | (static_cast(1) << 5) | 2); + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed>({date}, {0}), + [](const auto& values) { EXPECT_EQ(values.get_value_ref(0).get_date(), 1); }); + + auto datetime = DateV2Value::create_from_olap_datetime(19700101000001ULL); + datetime.set_microsecond(234567); + { + auto schema = shredded_primitive_schema(std::make_shared(6)); + schema.children.back()->type_descriptor.time_unit = ParquetTimeUnit::MICROS; + schema.children.back()->type_descriptor.timestamp_is_adjusted_to_utc = false; + decode(std::move(schema), + nullable_fixed>({datetime}, {0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_timestamp_ntz_micros(), 1234567); + }); + } + TimestampTzValue timestamp; + timestamp.unchecked_set_time(1970, 1, 1, 0, 0, 2, 345678); + { + auto schema = shredded_primitive_schema(std::make_shared(6)); + schema.children.back()->type_descriptor.time_unit = ParquetTimeUnit::MICROS; + schema.children.back()->type_descriptor.timestamp_is_adjusted_to_utc = true; + decode(std::move(schema), + nullable_fixed({timestamp}, {0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_timestamp_micros(), 2345678); + }); + } + + auto verify_bytes = [&](bool string_annotation, bool uuid) { + const std::array bytes {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}; + auto schema = shredded_primitive_schema(std::make_shared()); + schema.children.back()->type_descriptor.is_string_annotation = string_annotation; + schema.children.back()->type_descriptor.is_uuid = uuid; + auto strings = ColumnString::create(); + if (uuid) { + strings->insert_data(reinterpret_cast(bytes.data()), bytes.size()); + } else { + strings->insert_data("bytes", 5); + } + auto typed = ColumnNullable::create(std::move(strings), ColumnUInt8::create(1, 0)); + decode(std::move(schema), std::move(typed), [&](const auto& values) { + if (uuid) { + EXPECT_EQ(values.get_value_ref(0).get_uuid(), bytes); + } else if (string_annotation) { + EXPECT_EQ(values.get_value_ref(0).get_string(), StringRef("bytes")); + } else { + EXPECT_EQ(values.get_value_ref(0).get_binary(), StringRef("bytes")); + } + }); + }; + verify_bytes(false, false); + verify_bytes(true, false); + verify_bytes(false, true); +} + +TEST(VariantColumnReaderTest, RejectsInvalidShreddedUuidWidth) { + auto schema = shredded_primitive_schema(std::make_shared()); + schema.children.back()->type_descriptor.is_uuid = true; + auto strings = ColumnString::create(); + strings->insert_data("short", 5); + auto typed = ColumnNullable::create(std::move(strings), ColumnUInt8::create(1, 0)); + const std::string error = + materialization_error(schema, shredded_primitive_physical(std::move(typed))); + EXPECT_NE(error.find("UUID has 5 bytes instead of 16"), std::string::npos) << error; +} + TEST(VariantColumnReaderTest, DifferentMetadataDictionariesRemainIndependent) { VariantBatchBuilder first_builder; auto first_row = first_builder.begin_row(); @@ -349,6 +657,31 @@ TEST(VariantColumnReaderTest, DifferentMetadataDictionariesRemainIndependent) { EXPECT_EQ(field.get_int(), 2); } +TEST(VariantColumnReaderTest, AppendsCompleteShreddedStatesWithDifferentSchemasAndMetadata) { + auto output = make_nullable(std::make_shared())->create_column(); + auto first_schema = shredded_named_object_schema("a"); + ASSERT_TRUE(materialize_variant_rows(first_schema, + complete_shredded_object_physical("left", 1, 11), output) + .ok()); + auto second_schema = shredded_named_object_schema("b"); + ASSERT_TRUE(materialize_variant_rows(second_schema, + complete_shredded_object_physical("right", 2, 22), output) + .ok()); + + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_EQ(variants.size(), 2); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("left"), &field)); + EXPECT_EQ(field.get_int(), 1); + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_int(), 11); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("right"), &field)); + EXPECT_EQ(field.get_int(), 2); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("b"), &field)); + EXPECT_EQ(field.get_int(), 22); +} + TEST(VariantColumnReaderTest, ShreddedObjectFieldMayOmitResidualValueColumn) { const std::array ignored {0}; const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); @@ -776,6 +1109,273 @@ TEST(VariantColumnReaderTest, MaterializesShreddedArrayElements) { EXPECT_EQ(value.array_at(1).get_int(), 4); } +TEST(VariantColumnReaderTest, RejectsCorruptShreddedWrappersWithoutCrashing) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const std::array invalid_value {static_cast(0xff)}; + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + const StringRef residual_int(int_seven.data(), int_seven.size()); + auto expect_error = [](const std::string& error, std::string_view expected) { + EXPECT_NE(error.find(expected), std::string::npos) << error; + }; + std::string_view current_case; + + try { + { + current_case = "null metadata"; + SCOPED_TRACE("null metadata"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {1})); + fields.push_back(nullable_strings({residual_int}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "null metadata"); + } + { + current_case = "wrapper without carriers"; + SCOPED_TRACE("wrapper without carriers"); + auto schema = unshredded_schema(); + schema.children.pop_back(); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + expect_error(materialization_error(schema, root_wrapper(std::move(fields))), + "neither value nor typed_value"); + } + { + current_case = "scalar with residual"; + SCOPED_TRACE("scalar with residual"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(nullable_int64({8}, {0})); + expect_error( + materialization_error(shredded_int64_schema(), root_wrapper(std::move(fields))), + "scalar typed_value cannot have residual"); + } + { + current_case = "object with scalar residual"; + SCOPED_TRACE("object with scalar residual"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({9}, {0})); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(wrapper_fields)), ColumnUInt8::create(1, 0))); + fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_object_schema(), + root_wrapper(std::move(fields))), + "non-object residual"); + } + { + current_case = "object field count mismatch"; + SCOPED_TRACE("object field count mismatch"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + MutableColumns unexpected_object_fields; + unexpected_object_fields.push_back(nullable_int64({1}, {0})); + unexpected_object_fields.push_back(nullable_int64({2}, {0})); + fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(unexpected_object_fields)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_object_schema(), + root_wrapper(std::move(fields))), + "physical field count mismatch"); + } + { + current_case = "array with residual"; + SCOPED_TRACE("array with residual"); + MutableColumns empty_wrapper_fields; + empty_wrapper_fields.push_back(nullable_int64({}, {})); + auto empty_elements = ColumnNullable::create( + ColumnStruct::create(std::move(empty_wrapper_fields)), ColumnUInt8::create()); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(0); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(empty_elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error( + materialization_error(shredded_array_schema(), root_wrapper(std::move(fields))), + "array typed_value cannot have residual"); + } + { + current_case = "null array element wrapper"; + SCOPED_TRACE("null array element wrapper"); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({0}, {1})); + auto wrappers = ColumnStruct::create(std::move(wrapper_fields)); + auto elements = ColumnNullable::create(std::move(wrappers), ColumnUInt8::create(1, 1)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error( + materialization_error(shredded_array_schema(), root_wrapper(std::move(fields))), + "array element wrapper is null"); + } + { + current_case = "missing array element"; + SCOPED_TRACE("missing array element"); + MutableColumns element_fields; + element_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + element_fields.push_back(nullable_int64({0}, {1})); + auto elements = ColumnNullable::create(ColumnStruct::create(std::move(element_fields)), + ColumnUInt8::create(1, 0)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_mixed_array_schema(), + root_wrapper(std::move(fields))), + "array element is missing"); + } + { + current_case = "root field count mismatch"; + SCOPED_TRACE("root field count mismatch"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(nullable_int64({8}, {0})); + fields.push_back(nullable_int64({9}, {0})); + expect_error( + materialization_error(shredded_int64_schema(), root_wrapper(std::move(fields))), + "physical field count mismatch"); + } + { + current_case = "invalid metadata"; + SCOPED_TRACE("invalid metadata"); + MutableColumns fields; + fields.push_back(nullable_strings({StringRef("bad")}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "metadata"); + } + { + current_case = "invalid residual value"; + SCOPED_TRACE("invalid residual value"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{invalid_value.data(), invalid_value.size()}}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "Variant"); + } + } catch (const std::exception& error) { + FAIL() << "Unexpected exception in " << current_case << ": " << error.what(); + } +} + +TEST(VariantColumnReaderTest, ImmediateCorruptionLeavesDestinationUnchanged) { + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE( + materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({7}), output) + .ok()); + MutableColumns invalid_fields; + invalid_fields.push_back(nullable_strings( + {{VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()}}, {0})); + const Status status = materialize_variant_rows(shredded_int64_schema(), + root_wrapper(std::move(invalid_fields)), output); + EXPECT_FALSE(status.ok()); + ASSERT_EQ(output->size(), 1); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, MaterializesMixedRootArraysAndNullKinds) { + VariantBatchBuilder residual_builder; + { + auto row = residual_builder.begin_row(); + row.add_null(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef("x")); + row.add_int(2); + object.finish(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + auto array = row.start_array(); + row.add_int(3); + row.add_int(4); + array.finish(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + row.add_string(StringRef("tail")); + row.finish(); + } + VariantBatchBuilder residuals = residual_builder.finish_batch(); + const VariantRef first = residuals.value_at(0); + std::vector residual_values; + for (size_t row = 0; row < residuals.num_rows(); ++row) { + residual_values.push_back(residuals.value_at(row).value); + } + residual_values.insert(residual_values.begin() + 1, StringRef {}); + + MutableColumns element_fields; + element_fields.push_back(nullable_strings(residual_values, {0, 1, 0, 0, 0})); + element_fields.push_back(nullable_int64({0, 1, 0, 0, 0}, {1, 0, 1, 1, 1})); + auto elements = ColumnNullable::create(ColumnStruct::create(std::move(element_fields)), + ColumnUInt8::create(5, 0)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->get_data().assign({0, 5, 5, 5}); + auto arrays = ColumnArray::create(std::move(elements), std::move(offsets)); + + const StringRef metadata(first.metadata.data, first.metadata.size); + const std::array ignored {0}; + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata, metadata, metadata, metadata}, {0, 0, 0, 0})); + root_fields.push_back(nullable_strings( + {{ignored.data(), 0}, {ignored.data(), 0}, {ignored.data(), 0}, {ignored.data(), 0}}, + {1, 1, 1, 1})); + auto typed_nulls = ColumnUInt8::create(4, 0); + typed_nulls->get_data()[2] = 1; + typed_nulls->get_data()[3] = 1; + root_fields.push_back(ColumnNullable::create(std::move(arrays), std::move(typed_nulls))); + auto physical = root_wrapper(std::move(root_fields), {0, 0, 0, 1}); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_mixed_array_schema(), *physical, output).ok()); + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0, 0, 0, 1})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).num_elements(), 0); + const VariantRef mixed = variants.get_value_ref(1); + ASSERT_EQ(mixed.num_elements(), 5); + EXPECT_TRUE(mixed.array_at(0).is_null()); + EXPECT_EQ(mixed.array_at(1).get_int(), 1); + VariantRef object_field; + ASSERT_TRUE(mixed.array_at(2).object_find(StringRef("x"), &object_field)); + EXPECT_EQ(object_field.get_int(), 2); + EXPECT_EQ(mixed.array_at(3).array_at(1).get_int(), 4); + EXPECT_EQ(mixed.array_at(4).get_string(), StringRef("tail")); + EXPECT_TRUE(variants.get_value_ref(2).is_null()); +} + TEST(VariantColumnReaderTest, MaterializesVariantNestedInStruct) { const std::array int_seven { static_cast(static_cast(VariantPrimitiveId::INT8) @@ -817,6 +1417,124 @@ TEST(VariantColumnReaderTest, MaterializesVariantNestedInStruct) { EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); } +TEST(VariantColumnReaderTest, MaterializesPhysicallyShreddedVariantInStructArrayAndMap) { + auto make_plan_child = [](const ParquetColumnSchema* schema) { + auto child = std::make_unique(); + child->schema = schema; + child->contains_variant = schema->kind == ParquetColumnSchemaKind::VARIANT; + return child; + }; + + { + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + MutableColumns physical_fields; + physical_fields.push_back(shredded_int64_physical({11})); + auto physical = ColumnStruct::create(std::move(physical_fields)); + auto output = std::make_shared( + DataTypes {make_nullable(std::make_shared())}, + Strings {"v"}) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_column(0)) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 11); + } + + { + ParquetColumnSchema root_schema; + root_schema.name = "items"; + root_schema.kind = ParquetColumnSchemaKind::LIST; + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto physical = ColumnArray::create(shredded_int64_physical({12, 13}), std::move(offsets)); + auto output = std::make_shared( + make_nullable(std::make_shared())) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_data()) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 12); + EXPECT_EQ(variants.get_value_ref(1).get_int(), 13); + } + + { + ParquetColumnSchema root_schema; + root_schema.name = "entries"; + root_schema.kind = ParquetColumnSchemaKind::MAP; + auto key_schema = std::make_unique(); + key_schema->name = "key"; + key_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + key_schema->type = std::make_shared(); + root_schema.children.push_back(std::move(key_schema)); + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + plan.children.push_back(make_plan_child(root_schema.children[1].get())); + auto keys = ColumnString::create(); + keys->insert_data("a", 1); + keys->insert_data("b", 1); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto physical = ColumnMap::create(std::move(keys), shredded_int64_physical({14, 15}), + std::move(offsets)); + auto output = + std::make_shared(std::make_shared(), + make_nullable(std::make_shared())) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_values()) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 14); + EXPECT_EQ(variants.get_value_ref(1).get_int(), 15); + } +} + +TEST(VariantColumnReaderTest, ProjectedShreddedStateRejectsRootMaterialization) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({17}), output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + EXPECT_THROW((void)variants.get_value_ref(0), Exception); +} + TEST(VariantColumnReaderTest, AlignsNestedPrimitiveNullabilityAroundVariant) { const std::array int_seven { static_cast(static_cast(VariantPrimitiveId::INT8) diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out index 0157263d8a83fc..6abb509f237f85 100644 --- a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -21,6 +21,13 @@ 8 false {"n":30,"name":"same","ok":true} 9 false {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} +-- !variant_root_array_projection -- +1 false [] \N \N \N \N +2 false [null,1,{"x":2},[3,4],"tail"] null 1 2 4 +3 false [{"nested":[null,{"y":5}]}] {"nested":[null,{"y":5}]} \N \N \N +4 false null \N \N \N \N +5 true \N \N \N \N \N + -- !variant_path_expressions -- 1 ALICE 11 1.5 true 3 hz 10 DAVE 51 5.5 false 15 sz @@ -47,13 +54,52 @@ 8 30 9 40 --- !variant_implicit_shredded_filter -- +-- !variant_multi_file_serial -- +2 20 \N 2 \N {"b":2,"shared":20,"z":200} +3 30 3 \N \N {"a":3,"shared":30,"z":300} +4 40 \N \N \N {"c":4,"shared":40} +5 50 \N 5 500 {"b":5,"new_field":{"k":500},"shared":50} + +-- !variant_multi_file_parallel -- +2 20 \N 2 \N {"b":2,"shared":20,"z":200} +3 30 3 \N \N {"a":3,"shared":30,"z":300} +4 40 \N \N \N {"c":4,"shared":40} +5 50 \N 5 500 {"b":5,"new_field":{"k":500},"shared":50} + +-- !variant_type_matrix -- +true -128 -32768 2147483647 -9223372036854775808 true true -1234567890.1234 1970-01-02 1970-01-01T00:00:01.234567 "YmluYXJ5" false + +-- !variant_multi_row_group_result -- +192 8000 8191 1554336 + +-- !variant_deletion_vector_current -- +2048 0 4094 4192256 + +-- !variant_deletion_vector_before_delete -- +4096 0 4095 8386560 + +-- !variant_equality_delete_current -- +1 10 keep-one {"label":"keep-one","n":10} +3 30 keep-three {"label":"keep-three","n":30} + +-- !variant_equality_delete_before_delete -- +1 10 keep-one {"label":"keep-one","n":10} +2 20 delete {"label":"delete","n":20} +3 30 keep-three {"label":"keep-three","n":30} + +-- !variant_implicit_filter -- 10 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} 11 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} 9 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} +-- !variant_shredded_only_time_travel -- +1095 3001 4095 3885060 + +-- !variant_mixed_before_delete -- +1096 3001 5000 3890060 + -- !variant_page_pruning_result -- -1095 3001 4095 +1094 3001 4094 -- !variant_aggregate -- false 2 70 4 @@ -115,6 +161,15 @@ true 5 170 4.17 -- !variant_delete_only_merge -- 0 +-- !variant_position_delete_alignment -- +10 dave 50 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +9 carol 40 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_before_position_delete -- +10 dave 50 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 null 60 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +9 carol 40 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + -- !variant_orc_missing_column -- 1 \N diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index 43c6735a6dad74..65857764eee781 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -56,6 +56,39 @@ suite("test_iceberg_variant_read", .withPathStyleAccessEnabled(true) .withCredentials(new AWSStaticCredentialsProvider(credentials)) .build() + def executeCommand = { String command, int timeoutSeconds = 300 -> + StringBuilder stdout = new StringBuilder() + StringBuilder stderr = new StringBuilder() + def process = new ProcessBuilder("/bin/bash", "-c", command).start() + process.consumeProcessOutput(stdout, stderr) + process.waitForOrKill(timeoutSeconds * 1000) + assertEquals(0, process.exitValue(), + "Command failed\nstdout:\n${stdout}\nstderr:\n${stderr}") + return stdout.toString() + } + String dockerCommand = context.config.otherConfigs.get("externalDockerCommand") ?: "docker" + String sparkContainer = context.config.otherConfigs.get("icebergSparkContainer") + if (sparkContainer == null || sparkContainer.isEmpty()) { + String containers = executeCommand( + "${dockerCommand} ps --format '{{.ID}}\t{{.Names}}'", 30) + def matches = [] + containers.readLines().each { String line -> + String containerId = line.split(/\t/, 2)[0] + String probe = "${dockerCommand} exec ${containerId} bash -lc " + + "'test -f /mnt/SUCCESS && command -v spark-sql >/dev/null'" + try { + executeCommand(probe, 30) + matches.add(containerId) + } catch (Throwable ignored) { + // Only the Spark service contains the Iceberg writer dependencies. + } + } + assertEquals(1, matches.size(), "Expected exactly one usable Spark Iceberg container") + sparkContainer = matches[0] + } + def runInSparkContainer = { String command -> + executeCommand("${dockerCommand} exec ${sparkContainer} bash -lc '${command}'", 300) + } def latestSnapshotId = { String tableName -> List> rows = spark_iceberg """ @@ -100,6 +133,115 @@ suite("test_iceberg_variant_read", (10, parse_json('{"name":"dave","n":50,"ratio":5.5,"ok":false,"arr":[7,8],"nested":{"city":"sz"}}')), (11, parse_json('{"name":null,"n":60,"ratio":6.5,"ok":true,"arr":[9,10],"nested":{"city":null}}')); + DROP TABLE IF EXISTS demo.${dbName}.variant_root_arrays; + CREATE TABLE demo.${dbName}.variant_root_arrays (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_root_arrays VALUES + (1, parse_json('[]')), + (2, parse_json('[null,1,{"x":2},[3,4],"tail"]')), + (3, parse_json('[{"nested":[null,{"y":5}]}]')), + (4, parse_json('null')), + (5, NULL); + + DROP TABLE IF EXISTS demo.${dbName}.variant_multi_file; + CREATE TABLE demo.${dbName}.variant_multi_file (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false' + ); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (1, parse_json('{"a":1,"shared":10}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES ( + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='1' + ); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (2, parse_json('{"b":2,"shared":20,"z":200}')); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (3, parse_json('{"z":300,"shared":30,"a":3}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES + ('write.parquet.shred-variants'='false'); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (4, parse_json('{"c":4,"shared":40}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES + ('write.parquet.shred-variants'='true'); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (5, parse_json('{"shared":50,"b":5,"new_field":{"k":500}}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_type_matrix; + CREATE TABLE demo.${dbName}.variant_type_matrix (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_type_matrix SELECT 1, to_variant_object(named_struct( + 'bool_value', true, + 'tiny_value', CAST(-128 AS TINYINT), + 'small_value', CAST(-32768 AS SMALLINT), + 'int_value', CAST(2147483647 AS INT), + 'big_value', CAST('-9223372036854775808' AS BIGINT), + 'float_value', CAST('NaN' AS FLOAT), + 'double_value', CAST('Infinity' AS DOUBLE), + 'decimal_value', CAST('-1234567890.1234' AS DECIMAL(20, 4)), + 'date_value', CAST('1970-01-02' AS DATE), + 'timestamp_value', TIMESTAMP'1970-01-01 00:00:01.234567', + 'binary_value', CAST('binary' AS BINARY), + 'null_value', CAST(NULL AS INT) + )); + + DROP TABLE IF EXISTS demo.${dbName}.variant_multi_row_group; + CREATE TABLE demo.${dbName}.variant_multi_row_group (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100', + 'write.parquet.row-group-size-bytes'='4096' + ); + SET spark.sql.shuffle.partitions=1; + INSERT INTO demo.${dbName}.variant_multi_row_group + SELECT /*+ COALESCE(1) */ CAST(id AS INT), parse_json(concat( + '{"n":', id, ',"padding":"', repeat('x', 256), '"}')) + FROM range(0, 8192); + + DROP TABLE IF EXISTS demo.${dbName}.variant_deletion_vector; + CREATE TABLE demo.${dbName}.variant_deletion_vector (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100', + 'write.delete.mode'='merge-on-read', + 'read.parquet.vectorization.enabled'='false', + 'write.parquet.row-group-size-bytes'='4096' + ); + INSERT INTO demo.${dbName}.variant_deletion_vector + SELECT /*+ COALESCE(1) */ CAST(id AS INT), parse_json(concat('{"n":', id, ',"keep":', + IF(id % 2 = 0, 'true', 'false'), '}')) + FROM range(0, 4096); + RESET spark.sql.shuffle.partitions; + + DROP TABLE IF EXISTS demo.${dbName}.variant_equality_delete; + CREATE TABLE demo.${dbName}.variant_equality_delete (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_equality_delete VALUES + (1, parse_json('{"n":10,"label":"keep-one"}')), + (2, parse_json('{"n":20,"label":"delete"}')), + (3, parse_json('{"n":30,"label":"keep-three"}')); + DROP TABLE IF EXISTS demo.${dbName}.variant_page_pruning; DROP TABLE IF EXISTS demo.${dbName}.variant_nested; @@ -156,10 +298,83 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_write_guard VALUES (1); """ + List> multiRowGroupFiles = spark_iceberg """ + SELECT COUNT(*) FROM demo.${dbName}.variant_multi_row_group.files WHERE content = 0 + """ + assertEquals(1, multiRowGroupFiles.size()) + assertEquals("1", multiRowGroupFiles[0][0].toString(), + "The multi-row-group fixture must contain exactly one data file") + + String equalityDeleteBaseSnapshot = latestSnapshotId("variant_equality_delete") + String equalityDeleteJava = ''' +import java.util.HashMap; +import java.util.Map; +import org.apache.iceberg.CatalogUtil; +import org.apache.iceberg.DeleteFile; +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.Catalog; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.data.GenericRecord; +import org.apache.iceberg.data.Record; +import org.apache.iceberg.data.parquet.GenericParquetWriter; +import org.apache.iceberg.deletes.EqualityDeleteWriter; +import org.apache.iceberg.io.OutputFile; +import org.apache.iceberg.parquet.Parquet; + +public class AppendVariantEqualityDelete { + public static void main(String[] args) throws Exception { + Map props = new HashMap<>(); + props.put("type", "rest"); + props.put("uri", "http://rest:8181"); + props.put("warehouse", "s3://warehouse/wh/"); + props.put("io-impl", "org.apache.iceberg.aws.s3.S3FileIO"); + props.put("s3.endpoint", "http://minio:9000"); + props.put("s3.path-style-access", "true"); + props.put("s3.region", "us-east-1"); + Catalog catalog = CatalogUtil.buildIcebergCatalog("demo", props, null); + Table table = catalog.loadTable(TableIdentifier.of(args[0], args[1])); + Schema equalitySchema = table.schema().select("id"); + int fieldId = table.schema().findField("id").fieldId(); + OutputFile output = table.io().newOutputFile( + table.location() + "/data/variant-equality-delete-" + + System.currentTimeMillis() + ".parquet"); + EqualityDeleteWriter writer = Parquet.writeDeletes(output) + .forTable(table) + .rowSchema(equalitySchema) + .withSpec(PartitionSpec.unpartitioned()) + .createWriterFunc(GenericParquetWriter::create) + .equalityFieldIds(fieldId) + .overwrite() + .buildEqualityWriter(); + GenericRecord record = GenericRecord.create(equalitySchema); + record.setField("id", Integer.valueOf(args[2])); + writer.write(record); + writer.close(); + DeleteFile deleteFile = writer.toDeleteFile(); + table.newRowDelta().addDeletes(deleteFile).commit(); + } +} +''' + String encodedEqualityDeleteJava = + equalityDeleteJava.getBytes("UTF-8").encodeBase64().toString() + runInSparkContainer( + "echo ${encodedEqualityDeleteJava} | base64 -d " + + ">/tmp/AppendVariantEqualityDelete.java && " + + "javac -cp \"/opt/spark/jars/*\" " + + "/tmp/AppendVariantEqualityDelete.java && " + + "java -cp \"/tmp:/opt/spark/jars/*\" AppendVariantEqualityDelete " + + "${dbName} variant_equality_delete 2") + String writeGuardSourceSnapshot = latestSnapshotId("variant_write_guard") + String deletionVectorBaseSnapshot = latestSnapshotId("variant_deletion_vector") spark_iceberg """ ALTER TABLE demo.${dbName}.variant_write_guard ADD COLUMN payload VARIANT """ + spark_iceberg """ + DELETE FROM demo.${dbName}.variant_deletion_vector WHERE id % 2 = 1 + """ // Register a stable Iceberg metadata fixture so the page-pruning case always uses a // standards-compliant shredded Variant file, independent of the Spark writer version. @@ -175,6 +390,21 @@ suite("test_iceberg_variant_read", metadata_file => 's3a://warehouse/wh/${dbName}/variant_page_pruning/metadata/${shreddedMetadataName}') """ + String shreddedOnlySnapshot = latestSnapshotId("variant_page_pruning") + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_page_pruning SET TBLPROPERTIES ( + 'read.parquet.vectorization.enabled'='false', + 'write.delete.mode'='merge-on-read' + ); + INSERT INTO demo.${dbName}.variant_page_pruning VALUES + (5000, parse_json('{"n":5000,"padding":"mixed-unshredded"}')); + """ + String mixedBeforeDeleteSnapshot = latestSnapshotId("variant_page_pruning") + // One deletion vector targets the shredded fixture and another targets the appended + // unshredded file, forcing both physical states through the same scan and delete alignment. + spark_iceberg """ + DELETE FROM demo.${dbName}.variant_page_pruning WHERE id IN (4095, 5000) + """ sql """drop catalog if exists ${catalogName}""" sql """ @@ -198,18 +428,6 @@ suite("test_iceberg_variant_read", sql """set profile_level=2""" def profileAction = new ProfileAction(context) - def getProfileByToken = { String token -> - for (int retry = 0; retry < 20; ++retry) { - List profileData = profileAction.getProfileList() - for (final def profileItem in profileData) { - if (profileItem["Sql Statement"].toString().contains(token)) { - return profileAction.getProfile(profileItem["Profile ID"].toString()) - } - } - Thread.sleep(500) - } - throw new IllegalStateException("Missing profile for token: " + token) - } def counterSum = { String profile, String counterName -> Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)") Matcher matcher = pattern.matcher(profile) @@ -219,6 +437,25 @@ suite("test_iceberg_variant_read", } return sum } + def getProfileByToken = { String token, List positiveCounters = [] -> + String lastProfile = "" + for (int retry = 0; retry < 20; ++retry) { + List profileData = profileAction.getProfileList() + for (final def profileItem in profileData) { + if (profileItem["Sql Statement"].toString().contains(token)) { + lastProfile = profileAction.getProfile( + profileItem["Profile ID"].toString()).toString() + if (positiveCounters.every { counterSum(lastProfile, it) > 0 }) { + return lastProfile + } + } + } + Thread.sleep(500) + } + throw new IllegalStateException( + "Profile did not expose positive counters ${positiveCounters} for token ${token}: " + + lastProfile) + } String evolutionInitial = latestSnapshotId("variant_evolution") sql """ALTER TABLE variant_evolution CREATE TAG variant_initial""" @@ -306,6 +543,18 @@ suite("test_iceberg_variant_read", ORDER BY id """ + order_qt_variant_root_array_projection """ + SELECT id, + v IS NULL, + CAST(v AS STRING), + CAST(v[1] AS STRING), + CAST(v[2] AS INT), + CAST(v[3]['x'] AS INT), + CAST(v[4][2] AS INT) + FROM variant_root_arrays + ORDER BY id + """ + order_qt_variant_path_expressions """ SELECT id, UPPER(CAST(v['name'] AS STRING)), @@ -327,8 +576,8 @@ suite("test_iceberg_variant_read", ORDER BY id """ - // The first INSERT is unshredded while the second is shredded. Keep both small files on one - // scanner so their complete and leaf-only physical states must be projected before batching. + // Keep the independent Spark writes on one scanner to exercise metadata dictionaries and + // complete Variant state transitions across file boundaries before batching. sql "set parallel_pipeline_task_num=1" sql "set max_file_scanners_concurrency=1" order_qt_variant_cross_file_leaf_projection """ @@ -337,44 +586,167 @@ suite("test_iceberg_variant_read", ORDER BY id """ - // Keep the root Variant as output while the implicit scalar comparison drives the shredded - // typed_value statistics/page-index path. - order_qt_variant_implicit_shredded_filter """ + order_qt_variant_multi_file_serial """ + SELECT id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + sql "set parallel_pipeline_task_num=4" + sql "set max_file_scanners_concurrency=8" + order_qt_variant_multi_file_parallel """ + SELECT id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + + order_qt_variant_type_matrix """ + SELECT CAST(v['bool_value'] AS BOOLEAN), + CAST(v['tiny_value'] AS TINYINT), + CAST(v['small_value'] AS SMALLINT), + CAST(v['int_value'] AS INT), + CAST(v['big_value'] AS BIGINT), + ISNAN(CAST(v['float_value'] AS FLOAT)), + ISINF(CAST(v['double_value'] AS DOUBLE)), + CAST(v['decimal_value'] AS DECIMAL(20, 4)), + CAST(v['date_value'] AS DATE), + CAST(v['timestamp_value'] AS DATETIMEV2(6)), + CAST(v['binary_value'] AS STRING), + v['null_value'] IS NULL + FROM variant_type_matrix + """ + + String multiRowGroupColdToken = + "iceberg_variant_multi_row_group_cold_" + UUID.randomUUID().toString() + sql """ + SELECT '${multiRowGroupColdToken}', COUNT(*), MIN(id), MAX(id) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + String multiRowGroupColdProfile = getProfileByToken(multiRowGroupColdToken, + ["RowGroupsTotalNum", "VariantDirectLeafPathMisses", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() + assertTrue(counterSum(multiRowGroupColdProfile, "RowGroupsTotalNum") > 1, + "The generated Variant file did not contain multiple Parquet row groups") + assertTrue(counterSum(multiRowGroupColdProfile, "VariantDirectLeafPathMisses") > 0, + "The unshredded scan did not record its direct-leaf fallback") + assertTrue(counterSum(multiRowGroupColdProfile, "VariantReconstructedRows") > 0, + "The unshredded scan did not reconstruct Variant rows") + assertTrue(counterSum(multiRowGroupColdProfile, "FilteredRowsByLazyRead") > 0, + "The unshredded Variant predicate did not defer non-predicate columns") + String multiRowGroupWarmToken = + "iceberg_variant_multi_row_group_warm_" + UUID.randomUUID().toString() + sql """ + SELECT '${multiRowGroupWarmToken}', COUNT(*), MIN(id), MAX(id) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + String multiRowGroupWarmProfile = getProfileByToken(multiRowGroupWarmToken, + ["VariantDirectLeafPathMisses"]).toString() + assertTrue(counterSum(multiRowGroupWarmProfile, "VariantDirectLeafPathMisses") > 0, + "The warm unshredded scan did not preserve its direct-leaf fallback") + qt_variant_multi_row_group_result """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + + qt_variant_deletion_vector_current """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_deletion_vector + WHERE v['keep'] = true + """ + qt_variant_deletion_vector_before_delete """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_deletion_vector FOR VERSION AS OF ${deletionVectorBaseSnapshot} + WHERE v['n'] >= 0 + """ + order_qt_variant_equality_delete_current """ + SELECT id, CAST(v['n'] AS INT), CAST(v['label'] AS STRING), CAST(v AS STRING) + FROM variant_equality_delete + WHERE v['n'] >= 0 + ORDER BY id + """ + order_qt_variant_equality_delete_before_delete """ + SELECT id, CAST(v['n'] AS INT), CAST(v['label'] AS STRING), CAST(v AS STRING) + FROM variant_equality_delete FOR VERSION AS OF ${equalityDeleteBaseSnapshot} + WHERE v['n'] >= 0 + ORDER BY id + """ + + // Keep the root Variant as output while the scalar comparison exercises the fallback path for + // the unshredded Spark files. + order_qt_variant_implicit_filter """ SELECT id, CAST(v AS STRING) FROM variant_values WHERE v['n'] > 35 ORDER BY id """ - // The query projects the root Variant, while the predicate uses typed_value page metadata. + qt_variant_shredded_only_time_travel """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_page_pruning FOR VERSION AS OF ${shreddedOnlySnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + qt_variant_mixed_before_delete """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_page_pruning FOR VERSION AS OF ${mixedBeforeDeleteSnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + + // The query projects the complete Variant while its predicate reads the shredded typed leaf. + // The appended unshredded file must fall back independently in the same scan. String pagePruningToken = "iceberg_variant_page_pruning_" + UUID.randomUUID().toString() sql """ SELECT '${pagePruningToken}', id, CAST(v AS STRING) FROM variant_page_pruning - WHERE v['n'] > 3000 + WHERE CAST(v['n'] AS INT) > 3000 ORDER BY id """ - String pagePruningProfile = getProfileByToken(pagePruningToken).toString() + String pagePruningProfile = getProfileByToken(pagePruningToken, + ["FilteredRowsByPage", "VariantLeafProjections", "VariantDirectLeafPathMisses", + "VariantDirectLeafRows", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() assertTrue(counterSum(pagePruningProfile, "FilteredRowsByPage") > 0, "Shredded Variant typed_value did not filter any Parquet page") // The predicate_access_paths contract keeps the typed leaf eager while the complete Variant // root is read through the independent deferred-output projection. assertTrue(counterSum(pagePruningProfile, "VariantLeafProjections") > 0, "A root Variant output query did not retain its typed predicate leaf projection") + assertTrue(counterSum(pagePruningProfile, "VariantDirectLeafPathMisses") > 0, + "The mixed scan did not fall back for its unshredded Variant file") + assertTrue(counterSum(pagePruningProfile, "VariantDirectLeafRows") > 0, + "The mixed scan did not evaluate rows from the shredded typed leaf") + assertTrue(counterSum(pagePruningProfile, "VariantReconstructedRows") > 0, + "The mixed scan did not reconstruct complete Variant output") + assertTrue(counterSum(pagePruningProfile, "FilteredRowsByLazyRead") > 0, + "The mixed Variant scan did not delay output materialization") String leafProjectionToken = "iceberg_variant_leaf_projection_" + UUID.randomUUID().toString() sql """ SELECT '${leafProjectionToken}', COUNT(*) FROM variant_page_pruning - WHERE v['n'] > 3000 + WHERE CAST(v['n'] AS INT) > 3000 """ - String leafProjectionProfile = getProfileByToken(leafProjectionToken).toString() + String leafProjectionProfile = getProfileByToken(leafProjectionToken, + ["VariantLeafProjections"]).toString() assertTrue(counterSum(leafProjectionProfile, "VariantLeafProjections") > 0, "Variant typed predicate did not retain a physical leaf projection") qt_variant_page_pruning_result """ SELECT COUNT(*), MIN(id), MAX(id) FROM variant_page_pruning - WHERE v['n'] > 3000 + WHERE CAST(v['n'] AS INT) > 3000 """ // A later Variant metadata predicate must not prune away an earlier error-producing conjunct. @@ -529,6 +901,7 @@ suite("test_iceberg_variant_read", // A delete-only MERGE emits only position deletes. It must remain available even though // update/insert actions would route the unchanged Variant through the unsupported data writer. + String beforePositionDeleteSnapshot = latestSnapshotId("variant_values") sql """ MERGE INTO variant_values t USING (SELECT 11 AS id) s @@ -536,6 +909,31 @@ suite("test_iceberg_variant_read", WHEN MATCHED THEN DELETE """ qt_variant_delete_only_merge "SELECT COUNT(*) FROM variant_values WHERE id = 11" + order_qt_variant_position_delete_alignment """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT), CAST(v AS STRING) + FROM variant_values + WHERE v['n'] >= 40 + ORDER BY id + """ + order_qt_variant_before_position_delete """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT), CAST(v AS STRING) + FROM variant_values FOR VERSION AS OF ${beforePositionDeleteSnapshot} + WHERE v['n'] >= 40 + ORDER BY id + """ + String positionDeleteToken = + "iceberg_variant_position_delete_" + UUID.randomUUID().toString() + sql """ + SELECT '${positionDeleteToken}', COUNT(*) + FROM variant_values + WHERE v['n'] >= 40 + """ + String positionDeleteProfile = getProfileByToken(positionDeleteToken, + ["VariantDirectLeafPathMisses", "VariantReconstructedRows"]).toString() + assertTrue(counterSum(positionDeleteProfile, "VariantDirectLeafPathMisses") > 0, + "Position-delete filtering did not preserve the unshredded Variant fallback") + assertTrue(counterSum(positionDeleteProfile, "VariantReconstructedRows") > 0, + "Position-delete filtering did not reconstruct its Variant rows") // Files written before the Variant field existed have no physical Variant payload. Schema // evolution must synthesize NULL instead of rejecting their non-Parquet file format. From 8c78f007fcc2cafe42357bd25852b5407126177b Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 16:47:16 +0800 Subject: [PATCH 06/11] fix: preserve nested Variant append atomicity --- .../parquet/reader/variant_column_reader.cpp | 101 +++++++--- .../parquet/variant_column_reader_test.cpp | 176 ++++++++++++++++++ .../iceberg/test_iceberg_variant_read.groovy | 170 +++++++++++++---- 3 files changed, 390 insertions(+), 57 deletions(-) diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index 4827f951c94fe5..b51358a4f0649c 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -26,6 +26,7 @@ #include #include #include +#include #include "common/exception.h" #include "core/assert_cast.h" @@ -852,16 +853,30 @@ ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physi void append_compatible_column(IColumn& output, const IColumn& converted) { if (auto* output_nullable = check_and_get_column(output)) { - if (const auto* converted_nullable = check_and_get_column(converted)) { - append_compatible_column(output_nullable->get_nested_column(), - converted_nullable->get_nested_column()); - output_nullable->get_null_map_column().insert_range_from( - converted_nullable->get_null_map_column(), 0, converted.size()); - } else { - append_compatible_column(output_nullable->get_nested_column(), converted); - // External slots and nested Iceberg fields may remain nullable even when one file's - // physical node is required. Preserve that destination invariant with non-null bits. - output_nullable->push_false_to_nullmap(converted.size()); + auto& nested = output_nullable->get_nested_column(); + auto& null_map = output_nullable->get_null_map_column(); + const size_t nested_size = nested.size(); + const size_t null_map_size = null_map.size(); + try { + if (const auto* converted_nullable = check_and_get_column(converted)) { + append_compatible_column(nested, converted_nullable->get_nested_column()); + null_map.insert_range_from(converted_nullable->get_null_map_column(), 0, + converted.size()); + } else { + append_compatible_column(nested, converted); + // External slots and nested Iceberg fields may remain nullable even when one + // file's physical node is required. Preserve that destination invariant with + // non-null bits. + output_nullable->push_false_to_nullmap(converted.size()); + } + } catch (...) { + if (nested.size() > nested_size) { + nested.pop_back(nested.size() - nested_size); + } + if (null_map.size() > null_map_size) { + null_map.pop_back(null_map.size() - null_map_size); + } + throw; } return; } @@ -885,8 +900,25 @@ void append_compatible_column(IColumn& output, const IColumn& converted) { throw Exception(ErrorCode::CORRUPTION, "Parquet Variant materialization produced an incompatible STRUCT"); } + std::vector original_sizes(output_struct->tuple_size()); for (size_t i = 0; i < output_struct->tuple_size(); ++i) { - append_compatible_column(output_struct->get_column(i), converted_struct->get_column(i)); + original_sizes[i] = output_struct->get_column(i).size(); + } + try { + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + append_compatible_column(output_struct->get_column(i), + converted_struct->get_column(i)); + } + } catch (...) { + // Variant corruption can surface only during lazy fallback after earlier siblings + // were appended. Roll every child back to preserve the failed-append invariant. + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + auto& child = output_struct->get_column(i); + if (child.size() > original_sizes[i]) { + child.pop_back(child.size() - original_sizes[i]); + } + } + throw; } return; } @@ -897,12 +929,22 @@ void append_compatible_column(IColumn& output, const IColumn& converted) { throw Exception(ErrorCode::CORRUPTION, "Parquet Variant materialization produced an incompatible ARRAY"); } - const size_t element_base = output_array->get_data().size(); - append_compatible_column(output_array->get_data(), converted_array->get_data()); + auto& output_data = output_array->get_data(); auto& output_offsets = output_array->get_offsets(); - output_offsets.reserve(output_offsets.size() + converted_array->size()); - for (const auto offset : converted_array->get_offsets()) { - output_offsets.push_back(element_base + offset); + const size_t element_base = output_data.size(); + const size_t offsets_size = output_offsets.size(); + try { + append_compatible_column(output_data, converted_array->get_data()); + output_offsets.reserve(output_offsets.size() + converted_array->size()); + for (const auto offset : converted_array->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + } catch (...) { + if (output_data.size() > element_base) { + output_data.pop_back(output_data.size() - element_base); + } + output_offsets.resize(offsets_size); + throw; } return; } @@ -913,13 +955,28 @@ void append_compatible_column(IColumn& output, const IColumn& converted) { throw Exception(ErrorCode::CORRUPTION, "Parquet Variant materialization produced an incompatible MAP"); } - const size_t element_base = output_map->get_keys().size(); - append_compatible_column(output_map->get_keys(), converted_map->get_keys()); - append_compatible_column(output_map->get_values(), converted_map->get_values()); + auto& output_keys = output_map->get_keys(); + auto& output_values = output_map->get_values(); auto& output_offsets = output_map->get_offsets(); - output_offsets.reserve(output_offsets.size() + converted_map->size()); - for (const auto offset : converted_map->get_offsets()) { - output_offsets.push_back(element_base + offset); + const size_t element_base = output_keys.size(); + const size_t values_size = output_values.size(); + const size_t offsets_size = output_offsets.size(); + try { + append_compatible_column(output_keys, converted_map->get_keys()); + append_compatible_column(output_values, converted_map->get_values()); + output_offsets.reserve(output_offsets.size() + converted_map->size()); + for (const auto offset : converted_map->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + } catch (...) { + if (output_keys.size() > element_base) { + output_keys.pop_back(output_keys.size() - element_base); + } + if (output_values.size() > values_size) { + output_values.pop_back(output_values.size() - values_size); + } + output_offsets.resize(offsets_size); + throw; } return; } diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 5bf2d33c10f1af..24ace262faba37 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -1300,6 +1300,182 @@ TEST(VariantColumnReaderTest, ImmediateCorruptionLeavesDestinationUnchanged) { EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); } +TEST(VariantColumnReaderTest, LazyNestedCorruptionLeavesDestinationUnchanged) { + const std::array invalid_value {static_cast(0xff)}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + auto corrupt_variant = [&]() { + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{invalid_value.data(), invalid_value.size()}}, {0})); + return root_wrapper(std::move(fields)); + }; + auto label_schema = []() { + auto schema = std::make_unique(); + schema->name = "label"; + schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + schema->type = make_nullable(std::make_shared()); + return schema; + }; + auto make_plan = [](const ParquetColumnSchema& root) { + auto build = [&](auto&& self, const ParquetColumnSchema* schema) + -> std::unique_ptr { + auto node = std::make_unique(); + node->schema = schema; + node->contains_variant = schema->kind == ParquetColumnSchemaKind::VARIANT; + for (const auto& child_schema : schema->children) { + auto child = self(self, child_schema.get()); + node->contains_variant = node->contains_variant || child->contains_variant; + node->children.push_back(std::move(child)); + } + return node; + }; + return build(build, &root); + }; + auto make_struct_schema = [&](ParquetColumnSchema variant_schema) { + ParquetColumnSchema root; + root.name = "row"; + root.kind = ParquetColumnSchemaKind::STRUCT; + root.children.push_back(label_schema()); + root.children.push_back(std::make_unique(std::move(variant_schema))); + return root; + }; + auto make_struct_physical = [&](std::string_view label, MutableColumnPtr variant) { + MutableColumns fields; + fields.push_back(nullable_strings({StringRef(label.data(), label.size())}, {0})); + fields.push_back(std::move(variant)); + return ColumnStruct::create(std::move(fields)); + }; + const auto element_type = std::make_shared( + DataTypes {make_nullable(std::make_shared()), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}); + + { + auto output = element_type->create_column(); + auto valid_schema = make_struct_schema(shredded_int64_schema()); + auto valid_plan = make_plan(valid_schema); + ASSERT_TRUE(materialize_variant_columns( + *valid_plan, + *make_struct_physical("before", shredded_int64_physical({7})), output) + .ok()); + + auto corrupt_schema = make_struct_schema(unshredded_schema()); + auto corrupt_plan = make_plan(corrupt_schema); + const Status status = materialize_variant_columns( + *corrupt_plan, *make_struct_physical("after", corrupt_variant()), output); + EXPECT_FALSE(status.ok()); + + const auto& structure = assert_cast(*output); + const auto& label = assert_cast(structure.get_column(0)); + EXPECT_EQ(label.size(), 1); + EXPECT_EQ(label.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(label.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& payload = assert_cast(structure.get_column(1)); + EXPECT_EQ(payload.size(), 1); + EXPECT_EQ(payload.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(payload.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } + + { + auto output = std::make_shared(element_type)->create_column(); + auto valid_element_schema = make_struct_schema(shredded_int64_schema()); + ParquetColumnSchema valid_schema; + valid_schema.name = "rows"; + valid_schema.kind = ParquetColumnSchemaKind::LIST; + valid_schema.children.push_back( + std::make_unique(std::move(valid_element_schema))); + auto valid_plan = make_plan(valid_schema); + auto valid_offsets = ColumnArray::ColumnOffsets::create(); + valid_offsets->insert_value(1); + auto valid_physical = + ColumnArray::create(make_struct_physical("before", shredded_int64_physical({7})), + std::move(valid_offsets)); + ASSERT_TRUE(materialize_variant_columns(*valid_plan, *valid_physical, output).ok()); + + auto corrupt_element_schema = make_struct_schema(unshredded_schema()); + ParquetColumnSchema corrupt_schema; + corrupt_schema.name = "rows"; + corrupt_schema.kind = ParquetColumnSchemaKind::LIST; + corrupt_schema.children.push_back( + std::make_unique(std::move(corrupt_element_schema))); + auto corrupt_plan = make_plan(corrupt_schema); + auto corrupt_offsets = ColumnArray::ColumnOffsets::create(); + corrupt_offsets->insert_value(1); + auto corrupt_physical = ColumnArray::create( + make_struct_physical("after", corrupt_variant()), std::move(corrupt_offsets)); + const Status status = materialize_variant_columns(*corrupt_plan, *corrupt_physical, output); + EXPECT_FALSE(status.ok()); + + const auto& array = assert_cast(*output); + EXPECT_EQ(array.get_offsets(), (ColumnArray::Offsets64 {1})); + const auto& element = assert_cast(array.get_data()); + EXPECT_EQ(element.get_null_map_data(), (NullMap {0})); + const auto& structure = assert_cast(element.get_nested_column()); + const auto& label = assert_cast(structure.get_column(0)); + EXPECT_EQ(label.size(), 1); + EXPECT_EQ(label.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(label.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& payload = assert_cast(structure.get_column(1)); + EXPECT_EQ(payload.size(), 1); + EXPECT_EQ(payload.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(payload.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } + + { + auto output = + std::make_shared(make_nullable(std::make_shared()), + make_nullable(std::make_shared())) + ->create_column(); + auto make_map_schema = [&](ParquetColumnSchema variant_schema) { + ParquetColumnSchema root; + root.name = "entries"; + root.kind = ParquetColumnSchemaKind::MAP; + root.children.push_back(label_schema()); + root.children.push_back( + std::make_unique(std::move(variant_schema))); + return root; + }; + auto make_map_physical = [&](std::string_view key, MutableColumnPtr variant) { + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + return ColumnMap::create(nullable_strings({StringRef(key.data(), key.size())}, {0}), + std::move(variant), std::move(offsets)); + }; + + auto valid_schema = make_map_schema(shredded_int64_schema()); + auto valid_plan = make_plan(valid_schema); + ASSERT_TRUE(materialize_variant_columns( + *valid_plan, *make_map_physical("before", shredded_int64_physical({7})), + output) + .ok()); + auto corrupt_schema = make_map_schema(unshredded_schema()); + auto corrupt_plan = make_plan(corrupt_schema); + const Status status = materialize_variant_columns( + *corrupt_plan, *make_map_physical("after", corrupt_variant()), output); + EXPECT_FALSE(status.ok()); + + const auto& map = assert_cast(*output); + EXPECT_EQ(map.get_offsets(), (ColumnArray::Offsets64 {1})); + const auto& keys = assert_cast(map.get_keys()); + EXPECT_EQ(keys.size(), 1); + EXPECT_EQ(keys.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(keys.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& values = assert_cast(map.get_values()); + EXPECT_EQ(values.size(), 1); + EXPECT_EQ(values.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(values.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } +} + TEST(VariantColumnReaderTest, MaterializesMixedRootArraysAndNullKinds) { VariantBatchBuilder residual_builder; { diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index 65857764eee781..ffc99236a1976f 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -123,11 +123,11 @@ suite("test_iceberg_variant_read", (6, parse_json('42')), (7, parse_json('"root-string"')); ALTER TABLE demo.${dbName}.variant_values SET TBLPROPERTIES ( - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_values - WITH (`shred-variants`=true, `variant-inference-buffer-size`=100) VALUES + VALUES (8, parse_json('{"ok":true,"n":30,"name":"same"}')), (9, parse_json('{"name":"carol","n":40,"ratio":4.5,"ok":true,"arr":[5,6],"nested":{"city":"bj"},"new_key":"new"}')), (10, parse_json('{"name":"dave","n":50,"ratio":5.5,"ok":false,"arr":[7,8],"nested":{"city":"sz"}}')), @@ -138,7 +138,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_root_arrays VALUES @@ -158,7 +158,7 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_multi_file VALUES (1, parse_json('{"a":1,"shared":10}')); ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES ( - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='1' ); INSERT INTO demo.${dbName}.variant_multi_file @@ -170,7 +170,7 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_multi_file VALUES (4, parse_json('{"c":4,"shared":40}')); ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES - ('write.parquet.shred-variants'='true'); + ('write.parquet.shred-variants'='false'); INSERT INTO demo.${dbName}.variant_multi_file VALUES (5, parse_json('{"shared":50,"b":5,"new_field":{"k":500}}')); @@ -179,7 +179,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_type_matrix SELECT 1, to_variant_object(named_struct( @@ -202,7 +202,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100', 'write.parquet.row-group-size-bytes'='4096' ); @@ -217,7 +217,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100', 'write.delete.mode'='merge-on-read', 'read.parquet.vectorization.enabled'='false', @@ -234,7 +234,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_equality_delete VALUES @@ -254,7 +254,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_nested SELECT @@ -276,7 +276,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_signed_selector @@ -298,6 +298,13 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_write_guard VALUES (1); """ + List> multiFileDataFiles = spark_iceberg """ + SELECT COUNT(*) FROM demo.${dbName}.variant_multi_file.files WHERE content = 0 + """ + assertEquals(1, multiFileDataFiles.size()) + assertTrue(Long.parseLong(multiFileDataFiles[0][0].toString()) > 1, + "The parallel Variant fixture must contain multiple data files") + List> multiRowGroupFiles = spark_iceberg """ SELECT COUNT(*) FROM demo.${dbName}.variant_multi_row_group.files WHERE content = 0 """ @@ -375,6 +382,21 @@ public class AppendVariantEqualityDelete { spark_iceberg """ DELETE FROM demo.${dbName}.variant_deletion_vector WHERE id % 2 = 1 """ + List> deletionVectorFiles = spark_iceberg """ + SELECT file_format, content_offset, content_size_in_bytes + FROM demo.${dbName}.variant_deletion_vector.files + WHERE content = 1 + """ + assertFalse(deletionVectorFiles.isEmpty(), + "The Variant deletion fixture must expose a live delete file") + deletionVectorFiles.each { List deleteFile -> + assertEquals("PUFFIN", deleteFile[0].toString().toUpperCase(), + "The format-v3 Variant fixture must use PUFFIN deletion vectors") + assertTrue(Long.parseLong(deleteFile[1].toString()) >= 0, + "A PUFFIN deletion vector must expose its content offset") + assertTrue(Long.parseLong(deleteFile[2].toString()) > 0, + "A PUFFIN deletion vector must expose its content size") + } // Register a stable Iceberg metadata fixture so the page-pruning case always uses a // standards-compliant shredded Variant file, independent of the Spark writer version. @@ -428,33 +450,60 @@ public class AppendVariantEqualityDelete { sql """set profile_level=2""" def profileAction = new ProfileAction(context) + def mergedProfile = { String profile -> + if (!profile.contains("MergedProfile:")) { + return profile + } + String merged = profile.substring(profile.indexOf("MergedProfile:")) + int end = merged.length() + ["DetailProfile(", "Execution Profile:", "Appendix:"].each { String sectionName -> + int sectionIndex = merged.indexOf(sectionName) + if (sectionIndex > 0) { + end = Math.min(end, sectionIndex) + } + } + return merged.substring(0, end) + } def counterSum = { String profile, String counterName -> - Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)") - Matcher matcher = pattern.matcher(profile) + Pattern pattern = Pattern.compile("(?m)^\\s*(?:-\\s*)?" + + Pattern.quote(counterName) + ":\\s+([^\\n]+)") + Matcher matcher = pattern.matcher(mergedProfile(profile)) long sum = 0 while (matcher.find()) { - sum += Long.parseLong(matcher.group(1).replace(",", "")) + String valueText = matcher.group(1) + // Merged counters may be human-readable; the parenthesized value is the exact sum. + Matcher exact = Pattern.compile("\\(([0-9,]+)\\)").matcher(valueText) + Matcher number = Pattern.compile("([0-9,]+)").matcher(valueText) + if (exact.find()) { + sum += Long.parseLong(exact.group(1).replace(",", "")) + } else if (number.find()) { + sum += Long.parseLong(number.group(1).replace(",", "")) + } } return sum } + def profileInfoValues = { String profile, String infoName -> + Pattern pattern = Pattern.compile( + Pattern.quote(infoName) + ":\\s*\\[([^\\]]*)\\]") + Matcher matcher = pattern.matcher(profile) + if (!matcher.find()) { + return [] + } + return matcher.group(1).split(",").collect { String value -> value.trim() } + .findAll { String value -> !value.isEmpty() } + .collect { String value -> Long.parseLong(value.replace(",", "")) } + } def getProfileByToken = { String token, List positiveCounters = [] -> - String lastProfile = "" - for (int retry = 0; retry < 20; ++retry) { - List profileData = profileAction.getProfileList() - for (final def profileItem in profileData) { - if (profileItem["Sql Statement"].toString().contains(token)) { - lastProfile = profileAction.getProfile( - profileItem["Profile ID"].toString()).toString() - if (positiveCounters.every { counterSum(lastProfile, it) > 0 }) { - return lastProfile - } - } - } - Thread.sleep(500) + String lastProfile = profileAction.getProfileBySql(token, positiveCounters) + if (positiveCounters.every { String counter -> counterSum(lastProfile, counter) > 0 }) { + return lastProfile } - throw new IllegalStateException( - "Profile did not expose positive counters ${positiveCounters} for token ${token}: " + - lastProfile) + return profileAction.waitProfile({ + lastProfile = profileAction.getProfileBySql(token, positiveCounters) + return positiveCounters.every { + String counter -> counterSum(lastProfile, counter) > 0 + } ? lastProfile : "" + }, [], "Completed profile with positive counters ${positiveCounters} for ${token}") } String evolutionInitial = latestSnapshotId("variant_evolution") @@ -599,6 +648,7 @@ public class AppendVariantEqualityDelete { """ sql "set parallel_pipeline_task_num=4" sql "set max_file_scanners_concurrency=8" + sql "set min_file_scanners_concurrency=4" order_qt_variant_multi_file_parallel """ SELECT id, CAST(v['shared'] AS INT), @@ -610,6 +660,36 @@ public class AppendVariantEqualityDelete { WHERE v['shared'] >= 20 ORDER BY id """ + String parallelScanToken = + "iceberg_variant_parallel_scan_" + UUID.randomUUID().toString() + List> parallelScanRows = sql """ + SELECT '${parallelScanToken}', id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + assertEquals(4, parallelScanRows.size(), + "The parallel Variant query must read rows from multiple data files") + String parallelScanProfile = profileAction.getProfileBySql( + parallelScanToken, ["PerScannerRowsRead"]) + if (profileInfoValues(parallelScanProfile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } <= 1) { + parallelScanProfile = profileAction.waitProfile({ + String profile = profileAction.getProfileBySql( + parallelScanToken, ["PerScannerRowsRead"]) + return profileInfoValues(profile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } > 1 ? profile : "" + }, [], "Completed parallel Variant profile with multiple non-empty scanners") + } + assertTrue(profileInfoValues(parallelScanProfile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } > 1, + "The parallel Variant query did not use multiple non-empty scanners") + sql "set min_file_scanners_concurrency=1" order_qt_variant_type_matrix """ SELECT CAST(v['bool_value'] AS BOOLEAN), @@ -665,7 +745,7 @@ public class AppendVariantEqualityDelete { qt_variant_deletion_vector_current """ SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) FROM variant_deletion_vector - WHERE v['keep'] = true + WHERE v['n'] >= 0 """ qt_variant_deletion_vector_before_delete """ SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) @@ -705,6 +785,29 @@ public class AppendVariantEqualityDelete { WHERE CAST(v['n'] AS INT) > 3000 """ + // The complete Variant is the only scanned output column outside the predicate. A positive + // lazy-read count therefore proves Variant output deferral rather than deferral of an id + // sibling, while the row relationship proves reconstruction happens after filtering. + String lazyVariantToken = + "iceberg_variant_lazy_materialization_" + UUID.randomUUID().toString() + List> lazyVariantRows = sql """ + SELECT '${lazyVariantToken}', CAST(v AS STRING) + FROM variant_page_pruning FOR VERSION AS OF ${shreddedOnlySnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + String lazyVariantProfile = getProfileByToken(lazyVariantToken, + ["VariantDirectLeafRows", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() + long reconstructedVariantRows = + counterSum(lazyVariantProfile, "VariantReconstructedRows") + assertEquals((long) lazyVariantRows.size(), reconstructedVariantRows, + "Complete Variant reconstruction must be limited to selected output rows") + assertTrue(counterSum(lazyVariantProfile, "VariantDirectLeafRows") > + reconstructedVariantRows, + "Variant output was not deferred until after its shredded-leaf predicate") + assertTrue(counterSum(lazyVariantProfile, "FilteredRowsByLazyRead") > 0, + "The shredded predicate did not defer complete Variant output") + // The query projects the complete Variant while its predicate reads the shredded typed leaf. // The appended unshredded file must fall back independently in the same scan. String pagePruningToken = "iceberg_variant_page_pruning_" + UUID.randomUUID().toString() @@ -716,8 +819,7 @@ public class AppendVariantEqualityDelete { """ String pagePruningProfile = getProfileByToken(pagePruningToken, ["FilteredRowsByPage", "VariantLeafProjections", "VariantDirectLeafPathMisses", - "VariantDirectLeafRows", "VariantReconstructedRows", - "FilteredRowsByLazyRead"]).toString() + "VariantDirectLeafRows", "VariantReconstructedRows"]).toString() assertTrue(counterSum(pagePruningProfile, "FilteredRowsByPage") > 0, "Shredded Variant typed_value did not filter any Parquet page") // The predicate_access_paths contract keeps the typed leaf eager while the complete Variant @@ -730,8 +832,6 @@ public class AppendVariantEqualityDelete { "The mixed scan did not evaluate rows from the shredded typed leaf") assertTrue(counterSum(pagePruningProfile, "VariantReconstructedRows") > 0, "The mixed scan did not reconstruct complete Variant output") - assertTrue(counterSum(pagePruningProfile, "FilteredRowsByLazyRead") > 0, - "The mixed Variant scan did not delay output materialization") String leafProjectionToken = "iceberg_variant_leaf_projection_" + UUID.randomUUID().toString() sql """ From 469931c3a898367e4a9d9017d6747f8292646f25 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 16:13:32 +0800 Subject: [PATCH 07/11] [fix](parquet) Isolate Variant planning from ordinary scans --- .../parquet/parquet_column_schema.cpp | 14 +++-------- .../format_v2/parquet/parquet_column_schema.h | 4 +++ .../parquet/parquet_file_context.cpp | 2 ++ .../format_v2/parquet/parquet_file_context.h | 3 +++ be/src/format_v2/parquet/parquet_reader.cpp | 25 +++++++++++++------ .../format_v2/parquet/parquet_statistics.cpp | 10 ++++++-- .../parquet/reader/native_column_reader.cpp | 21 ++++++++++------ .../parquet/reader/variant_column_reader.cpp | 1 + .../format_v2/parquet/parquet_schema_test.cpp | 8 ++++++ .../parquet/parquet_statistics_test.cpp | 1 + .../parquet/variant_column_reader_test.cpp | 1 + 11 files changed, 62 insertions(+), 28 deletions(-) diff --git a/be/src/format_v2/parquet/parquet_column_schema.cpp b/be/src/format_v2/parquet/parquet_column_schema.cpp index 623f1ece3dd409..71416e17dc9209 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.cpp +++ b/be/src/format_v2/parquet/parquet_column_schema.cpp @@ -178,16 +178,6 @@ void propagate_native_max_levels(ParquetColumnSchema* schema) { } } -bool contains_variant_node(const ParquetColumnSchema& schema) { - if (schema.kind == ParquetColumnSchemaKind::VARIANT) { - return true; - } - return std::ranges::any_of(schema.children, [](const auto& child) { - DORIS_CHECK(child != nullptr); - return contains_variant_node(*child); - }); -} - std::unique_ptr build_native_node_schema(const NativeFieldSchema& field, int32_t local_id) { auto result = std::make_unique(); @@ -222,6 +212,7 @@ std::unique_ptr build_native_node_schema(const NativeFieldS } if (field.variant_physical_type != nullptr) { result->kind = ParquetColumnSchemaKind::VARIANT; + result->contains_variant = true; } else if (primitive_type == TYPE_ARRAY) { result->kind = ParquetColumnSchemaKind::LIST; } else if (primitive_type == TYPE_MAP) { @@ -233,10 +224,11 @@ std::unique_ptr build_native_node_schema(const NativeFieldS for (size_t child_idx = 0; child_idx < field.children.size(); ++child_idx) { result->children.push_back( build_native_node_schema(field.children[child_idx], cast_set(child_idx))); + result->contains_variant |= result->children.back()->contains_variant; } // A nested Variant changes its public child type from the physical STRUCT carrier. Rebuild // every enclosing complex type so file-block columns keep the same logical shape as readers. - if (result->kind != ParquetColumnSchemaKind::VARIANT && contains_variant_node(*result)) { + if (result->kind != ParquetColumnSchemaKind::VARIANT && result->contains_variant) { DataTypePtr logical_type; if (result->kind == ParquetColumnSchemaKind::LIST) { DORIS_CHECK(result->children.size() == 1); diff --git a/be/src/format_v2/parquet/parquet_column_schema.h b/be/src/format_v2/parquet/parquet_column_schema.h index 4cb295018332fe..3103b97cfc253c 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.h +++ b/be/src/format_v2/parquet/parquet_column_schema.h @@ -57,6 +57,10 @@ struct ParquetColumnSchema { ParquetColumnSchemaKind kind = ParquetColumnSchemaKind::PRIMITIVE; + // Cached during schema construction so readers created per row group do not repeatedly walk + // ordinary nested schemas to discover whether Variant-specific planning is needed. + bool contains_variant = false; + // ======== Dremel Levels ======== int16_t max_definition_level = 0; diff --git a/be/src/format_v2/parquet/parquet_file_context.cpp b/be/src/format_v2/parquet/parquet_file_context.cpp index c178e491b8ac21..8ba8cf94662f9a 100644 --- a/be/src/format_v2/parquet/parquet_file_context.cpp +++ b/be/src/format_v2/parquet/parquet_file_context.cpp @@ -273,6 +273,7 @@ Status ParquetFileContext::open(io::FileReaderSPtr input_file_reader, io::IOCont bool enable_page_cache, const io::FileDescription& file_description, bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary) { DORIS_CHECK(input_file_reader != nullptr); + contains_variant = false; if (detail::should_stage_small_http_file(input_file_reader->path().native(), input_file_reader->size(), config::in_memory_file_size)) { @@ -602,6 +603,7 @@ Status ParquetFileContext::close() { native_io_ctx = nullptr; native_page_cache_enabled = false; native_page_cache_file_key.clear(); + contains_variant = false; return Status::OK(); } diff --git a/be/src/format_v2/parquet/parquet_file_context.h b/be/src/format_v2/parquet/parquet_file_context.h index 0cd413e10557ac..38e78438e39b29 100644 --- a/be/src/format_v2/parquet/parquet_file_context.h +++ b/be/src/format_v2/parquet/parquet_file_context.h @@ -139,6 +139,9 @@ struct ParquetFileContext { int64_t native_footer_cache_hits = 0; bool native_page_cache_enabled = false; std::string native_page_cache_file_key; + // Set once after the logical file schema is built. Per-request planning uses this guard so + // ordinary files never enter Variant projection or shredded-statistics paths. + bool contains_variant = false; Status open(io::FileReaderSPtr input_file_reader, io::IOContext* io_ctx, bool enable_page_cache, const io::FileDescription& file_description, diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index bd44316463f6b1..8c6efe27558bbc 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -163,6 +163,9 @@ size_t finalize_variant_leaf_projections( if (local_id < 0 || local_id >= static_cast(file_schema.size())) { continue; } + if (!file_schema[local_id]->contains_variant) { + continue; + } retained += detail::finalize_variant_leaf_projection(metadata.to_thrift(), *file_schema[local_id], &projection); } @@ -467,6 +470,11 @@ Status ParquetReader::init(RuntimeState* state) { SCOPED_TIMER(_parquet_profile.parse_meta_time); RETURN_IF_ERROR(build_parquet_column_schema(_state->file_context.native_metadata->schema(), &_state->file_schema)); + _state->file_context.contains_variant = + std::ranges::any_of(_state->file_schema, [](const auto& column) { + DORIS_CHECK(column != nullptr); + return column->contains_variant; + }); if (_enable_mapping_timestamp_tz) { for (auto& column_schema : _state->file_schema) { apply_timestamp_tz_mapping(column_schema.get()); @@ -518,13 +526,16 @@ Status ParquetReader::open(std::shared_ptr request) { } auto request_snapshot = request; DORIS_CHECK(request_snapshot != nullptr); - const size_t retained_variant_leaf_projections = - finalize_variant_leaf_projections(*_state->file_context.native_metadata, - _state->file_schema, - &request_snapshot->predicate_columns) + - finalize_variant_leaf_projections(*_state->file_context.native_metadata, - _state->file_schema, - &request_snapshot->non_predicate_columns); + size_t retained_variant_leaf_projections = 0; + if (_state->file_context.contains_variant) { + retained_variant_leaf_projections = + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->predicate_columns) + + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->non_predicate_columns); + } if (_parquet_profile.variant_leaf_projections != nullptr) { COUNTER_UPDATE(_parquet_profile.variant_leaf_projections, retained_variant_leaf_projections); diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 20c72e8c7360ae..08779056c73ee6 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -1283,6 +1283,12 @@ Status select_row_groups_by_metadata( if (pruning_stats != nullptr) { pruning_stats->total_row_groups = cast_set(candidate_size); } + const bool contains_variant = + file_context != nullptr ? file_context->contains_variant + : std::ranges::any_of(file_schema, [](const auto& column) { + DORIS_CHECK(column != nullptr); + return column->contains_variant; + }); selected_row_groups->reserve(candidate_size); for (size_t candidate_idx = 0; candidate_idx < candidate_size; ++candidate_idx) { const int row_group_idx = candidate_row_groups == nullptr @@ -1309,8 +1315,8 @@ Status select_row_groups_by_metadata( has_expr_zonemap_filter(request, runtime_state) && (check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, timezone) || - check_shredded_variant_statistics(metadata, row_group, file_schema, request, - timezone))) { + (contains_variant && check_shredded_variant_statistics( + metadata, row_group, file_schema, request, timezone)))) { prune_reason = ParquetRowGroupPruneReason::STATISTICS; } if (probe_mode != ParquetMetadataProbeMode::FOOTER_ONLY && diff --git a/be/src/format_v2/parquet/reader/native_column_reader.cpp b/be/src/format_v2/parquet/reader/native_column_reader.cpp index 618eca6f6a50ce..273964e5d1f451 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native_column_reader.cpp @@ -292,13 +292,18 @@ Status NativeColumnReader::create( } auto logical_type = projected_type(column_schema, projection, false); - auto native_type = projected_type(column_schema, projection, true); + auto native_type = logical_type; + std::unique_ptr variant_plan; + if (column_schema.contains_variant) { + // Native readers are instantiated per projected column and row group. Keep Variant tree + // construction out of ordinary scans instead of charging that setup cost at every split. + native_type = projected_type(column_schema, projection, true); + variant_plan = build_variant_plan(column_schema, projection); + } std::shared_ptr schema_node; RETURN_IF_ERROR(build_native_schema_node(native_type, column_schema, &schema_node)); std::set projected_ids; collect_projected_ids(column_schema, projection, *field, &projected_ids); - auto variant_plan = build_variant_plan(column_schema, projection); - auto native_reader = std::unique_ptr( new NativeColumnReader(column_schema, std::move(logical_type), std::move(native_type), std::move(variant_plan), profile)); @@ -364,7 +369,7 @@ Status NativeColumnReader::init( runtime_state != nullptr && runtime_state->enable_strict_mode())); DORIS_CHECK(_native_reader != nullptr); _skip_column = _native_type->create_column(); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { _variant_physical_column = _native_type->create_column(); } return Status::OK(); @@ -387,7 +392,7 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ RETURN_IF_ERROR(filter.init(filter_data, static_cast(rows), filter_all)); _native_reader->reset_filter_map_index(); const bool materialize_variant = - !dictionary_ids && _variant_plan->contains_variant && output_type->equals(*_type); + !dictionary_ids && _variant_plan != nullptr && output_type->equals(*_type); if (materialize_variant) { _variant_physical_column->clear(); } @@ -715,7 +720,7 @@ Status NativeColumnReader::select_with_dictionary_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(survivor_count != nullptr); DORIS_CHECK(used_filter != nullptr); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { row_filter->clear(); *used_filter = false; return Status::OK(); @@ -871,7 +876,7 @@ Status NativeColumnReader::select_with_fixed_width_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(used_filter != nullptr); DORIS_CHECK(execution_kind != nullptr); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { // Direct fixed-width evaluation cannot preserve a Variant physical subtree's row shape. row_filter->clear(); *used_filter = false; @@ -1008,7 +1013,7 @@ bool NativeColumnReader::crossed_page_since_last_batch() { Result NativeColumnReader::dictionary_values() { DORIS_CHECK(_native_reader != nullptr); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { return ResultError( Status::NotSupported("Parquet Variant columns do not expose dictionary values")); } diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index b51358a4f0649c..da5ecbb8310a20 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -470,6 +470,7 @@ std::unique_ptr clone_schema( result->leaf_column_id = source.leaf_column_id; result->type_descriptor = source.type_descriptor; result->kind = source.kind; + result->contains_variant = source.contains_variant; result->max_definition_level = source.max_definition_level; result->max_repetition_level = source.max_repetition_level; result->nullable_definition_level = source.nullable_definition_level; diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp b/be/test/format_v2/parquet/parquet_schema_test.cpp index 9ecc1edf6fc582..acd5e8600789ec 100644 --- a/be/test/format_v2/parquet/parquet_schema_test.cpp +++ b/be/test/format_v2/parquet/parquet_schema_test.cpp @@ -217,6 +217,7 @@ TEST(ParquetSchemaTest, NativeSchemaRecognizesVariantLogicalGroup) { ASSERT_TRUE(status.ok()) << status; ASSERT_EQ(fields.size(), 1); EXPECT_EQ(fields[0]->kind, ParquetColumnSchemaKind::VARIANT); + EXPECT_TRUE(fields[0]->contains_variant); EXPECT_EQ(remove_nullable(fields[0]->type)->get_primitive_type(), TYPE_VARIANT); EXPECT_NE(typeid_cast(remove_nullable(fields[0]->type).get()), nullptr); @@ -247,6 +248,10 @@ TEST(ParquetSchemaTest, NestedVariantPropagatesIntoParentLogicalType) { const auto status = build_parquet_column_schema(descriptor, &fields); ASSERT_TRUE(status.ok()) << status; ASSERT_EQ(fields.size(), 1); + EXPECT_TRUE(fields[0]->contains_variant); + ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_FALSE(fields[0]->children[0]->contains_variant); + EXPECT_TRUE(fields[0]->children[1]->contains_variant); const auto* info_type = assert_cast(remove_nullable(fields[0]->type).get()); ASSERT_EQ(info_type->get_elements().size(), 2); @@ -481,9 +486,12 @@ TEST(ParquetSchemaTest, NativeMetadataTreePreservesNestedFieldNamesAndIds) { std::vector> fields; ASSERT_TRUE(build_parquet_column_schema(native_schema, &fields).ok()); ASSERT_EQ(fields.size(), 1); + EXPECT_FALSE(fields[0]->contains_variant); EXPECT_EQ(fields[0]->name, "protocol"); EXPECT_EQ(fields[0]->parquet_field_id, 10); ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_FALSE(fields[0]->children[0]->contains_variant); + EXPECT_FALSE(fields[0]->children[1]->contains_variant); EXPECT_EQ(fields[0]->children[0]->name, "minReaderVersion"); EXPECT_EQ(fields[0]->children[0]->leaf_column_id, 0); EXPECT_EQ(fields[0]->children[1]->name, "minWriterVersion"); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 4c3fdcafa06ffe..533b526d613d9f 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -924,6 +924,7 @@ TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) variant->name = "v"; variant->local_id = 0; variant->kind = format::parquet::ParquetColumnSchemaKind::VARIANT; + variant->contains_variant = true; variant->type = make_nullable(std::make_shared()); variant->children.push_back(bytes("metadata", 0, 0)); variant->children.push_back(bytes("value", 1, 1)); diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 24ace262faba37..51a50319285c5b 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -70,6 +70,7 @@ ParquetColumnSchema unshredded_schema() { ParquetColumnSchema schema; schema.name = "payload"; schema.kind = ParquetColumnSchemaKind::VARIANT; + schema.contains_variant = true; schema.type = make_nullable(std::make_shared()); const auto binary = make_nullable(std::make_shared()); schema.variant_physical_type = make_nullable(std::make_shared( From 52dc650c902d82774c8a8e4ac96608c849862eff Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 20:32:30 +0800 Subject: [PATCH 08/11] [fix](variant) Preserve projected shredded states across exchange --- .../column/variant_v2/column_variant_v2.cpp | 293 +++++++++++++++++- .../column/variant_v2/column_variant_v2.h | 5 + .../function/function_variant_element_v2.cpp | 7 +- .../parquet/reader/variant_column_reader.cpp | 41 ++- .../parquet/variant_column_reader_test.cpp | 141 +++++++++ 5 files changed, 480 insertions(+), 7 deletions(-) diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index 0d5a7fb74bb6df..e0ba9bd048cb97 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -22,9 +22,11 @@ #include #include #include +#include #include #include #include +#include #include "common/check.h" #include "common/exception.h" @@ -313,6 +315,266 @@ ValidatedTypedInput validate_typed_input(ColumnPtr column, DataTypePtr scalar_ty "ColumnVariantV2::{} is intentionally unsupported for Variant values", method); } +class CompositeVariantShreddedState final : public VariantShreddedState { +public: + explicit CompositeVariantShreddedState( + std::vector> segments) + : _segments(std::move(segments)) { + DORIS_CHECK(std::ranges::all_of(_segments, [](const auto& segment) { + return segment != nullptr; + })) << "composite Variant shredded segments must not be null"; + } + + size_t size() const override { + size_t rows = 0; + for (const auto& segment : _segments) { + DORIS_CHECK_LE(segment->size(), std::numeric_limits::max() - rows) + << "composite Variant shredded row count overflows size_t"; + rows += segment->size(); + } + return rows; + } + + size_t byte_size() const override { + size_t bytes = 0; + for (const auto& segment : _segments) { + bytes += segment->byte_size(); + } + std::lock_guard lock(_materialization_lock); + return bytes + (_materialized ? _materialized->byte_size() : 0); + } + + size_t allocated_bytes() const override { + size_t bytes = 0; + for (const auto& segment : _segments) { + bytes += segment->allocated_bytes(); + } + std::lock_guard lock(_materialization_lock); + return bytes + (_materialized ? _materialized->allocated_bytes() : 0); + } + + void sanity_check() const override { + for (const auto& segment : _segments) { + segment->sanity_check(); + } + } + + void for_each_subcolumn(const IColumn::ImutableColumnCallback& callback) const override { + for (const auto& segment : _segments) { + segment->for_each_subcolumn(callback); + } + } + + std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t /*result_size_hint*/) const override { + DORIS_CHECK_EQ(filter.size(), size()) + << "composite Variant shredded filter size does not match row count"; + std::vector> selected; + selected.reserve(_segments.size()); + size_t offset = 0; + for (const auto& segment : _segments) { + IColumn::Filter segment_filter; + segment_filter.insert(filter.begin() + offset, + filter.begin() + offset + segment->size()); + auto filtered = segment->filter(segment_filter, -1); + if (filtered->size() != 0) { + selected.push_back(std::move(filtered)); + } + offset += segment->size(); + } + return pack(std::move(selected)); + } + + std::shared_ptr select_range(size_t start, size_t length) const override { + DORIS_CHECK_LE(start, size()) << "composite Variant range starts past source size"; + DORIS_CHECK_LE(length, size() - start) << "composite Variant range exceeds source size"; + std::vector> selected; + if (length == 0) { + return pack(std::move(selected)); + } + const size_t end = start + length; + size_t offset = 0; + for (const auto& segment : _segments) { + const size_t segment_end = offset + segment->size(); + const size_t overlap_begin = std::max(start, offset); + const size_t overlap_end = std::min(end, segment_end); + if (overlap_begin < overlap_end) { + selected.push_back( + segment->select_range(overlap_begin - offset, overlap_end - overlap_begin)); + } + offset = segment_end; + if (offset >= end) { + break; + } + } + return pack(std::move(selected)); + } + + std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const override { + if (indices_begin == indices_end) { + return pack({}); + } + DORIS_CHECK(indices_begin != nullptr && indices_end != nullptr && + indices_begin < indices_end) + << "composite Variant indices are invalid"; + + std::vector segment_ends; + segment_ends.reserve(_segments.size()); + size_t rows = 0; + for (const auto& segment : _segments) { + rows += segment->size(); + segment_ends.push_back(rows); + } + + std::vector> selected; + const uint32_t* cursor = indices_begin; + while (cursor != indices_end) { + DORIS_CHECK_LT(*cursor, rows) << "composite Variant source index is out of range"; + const size_t segment_index = + std::upper_bound(segment_ends.begin(), segment_ends.end(), *cursor) - + segment_ends.begin(); + const size_t segment_begin = segment_index == 0 ? 0 : segment_ends[segment_index - 1]; + DorisVector local_indices; + while (cursor != indices_end && *cursor >= segment_begin && + *cursor < segment_ends[segment_index]) { + local_indices.push_back(static_cast(*cursor - segment_begin)); + ++cursor; + } + selected.push_back(_segments[segment_index]->select_indices( + local_indices.data(), local_indices.data() + local_indices.size())); + } + return pack(std::move(selected)); + } + + bool can_materialize() const override { + return std::ranges::all_of(_segments, + [](const auto& segment) { return segment->can_materialize(); }); + } + + bool try_append(const VariantShreddedState& source) override { + if (const auto* composite = dynamic_cast(&source)) { + for (const auto& segment : composite->_segments) { + append(segment); + } + } else { + append(source.select_range(0, source.size())); + } + std::lock_guard lock(_materialization_lock); + _materialized.reset(); + return true; + } + + std::optional find_typed_value( + std::span path) const override { + if (_segments.empty()) { + return std::nullopt; + } + std::vector matches; + matches.reserve(_segments.size()); + for (const auto& segment : _segments) { + auto match = segment->find_typed_value(path); + if (!match.has_value()) { + return std::nullopt; + } + matches.push_back(std::move(*match)); + } + + const bool homogeneous = matches.front().column && matches.front().type && + std::ranges::all_of(matches, [&](const auto& match) { + return match.column && match.type && !match.normalized && + exact_typed_identity(matches.front().type, match.type); + }); + if (homogeneous) { + MutableColumnPtr combined = matches.front().column->clone_empty(); + for (const auto& match : matches) { + combined->insert_range_from(*match.column, 0, match.column->size()); + } + return VariantShreddedTypedValue {.column = std::move(combined), + .type = matches.front().type, + .normalized = nullptr}; + } + + auto values = ColumnVariantV2::create(); + auto nulls = ColumnUInt8::create(); + nulls->reserve(size()); + for (const auto& match : matches) { + if (match.normalized) { + const auto& nullable = assert_cast(*match.normalized); + const auto& variants = + assert_cast(nullable.get_nested_column()); + values->insert_range_from(variants, 0, variants.size()); + nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); + continue; + } + auto typed = ColumnVariantV2::create_typed(match.column, match.type); + values->insert_range_from(*typed, 0, typed->size()); + const auto& nullable = assert_cast(*match.column); + nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); + } + return VariantShreddedTypedValue { + .column = nullptr, + .type = nullptr, + .normalized = ColumnNullable::create(std::move(values), std::move(nulls))}; + } + + const ColumnVariantV2& materialized_column() const override { + std::lock_guard lock(_materialization_lock); + if (!_materialized) { + auto materialized = ColumnVariantV2::create(); + for (const auto& segment : _segments) { + const ColumnVariantV2& source = segment->materialized_column(); + materialized->insert_range_from(source, 0, source.size()); + } + _materialized = std::move(materialized); + } + return *_materialized; + } + +private: + static std::shared_ptr pack( + std::vector> segments) { + if (segments.size() == 1) { + return std::move(segments.front()); + } + return std::make_shared(std::move(segments)); + } + + void append(std::shared_ptr source) { + if (source->size() == 0) { + return; + } + if (const auto* composite = + dynamic_cast(source.get())) { + _segments.insert(_segments.end(), composite->_segments.begin(), + composite->_segments.end()); + return; + } + if (!_segments.empty()) { + auto& tail = _segments.back(); + if (tail.use_count() != 1) { + tail = tail->select_range(0, tail->size()); + } + if (tail->try_append(*source)) { + return; + } + } + _segments.push_back(std::move(source)); + } + + std::vector> _segments; + mutable std::mutex _materialization_lock; + mutable ColumnVariantV2::MutablePtr _materialized; +}; + +std::shared_ptr combine_shredded_states( + std::shared_ptr left, std::shared_ptr right) { + auto combined = std::make_shared( + std::vector> {std::move(left)}); + combined->try_append(*right); + return combined; +} + } // namespace #ifdef BE_TEST @@ -818,6 +1080,14 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) _check_invariants(); return; } + if (!_shredded->can_materialize() && !selected_source->can_materialize()) { + // Different files may shred the same projected path with different physical + // identities. Keep both incomplete states ordered because neither can reconstruct the + // root value. + _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); + _check_invariants(); + return; + } } if (_shredded) { ensure_encoded(); @@ -911,9 +1181,6 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } - if (_shredded) { - ensure_encoded(); - } if (!_typed && empty() && _metadatas->empty() && source._shredded) { // Gather into the native shredded representation for the same reason as range selection: // row selection does not require, and may not have, a complete logical Variant value. @@ -921,6 +1188,26 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) _check_invariants(); return; } + if (_shredded && source._shredded) { + auto selected_source = source._shredded->select_indices(indices_begin, indices_end); + if (_shredded.use_count() != 1) { + _shredded = _shredded->select_range(0, size()); + } + if (_shredded->try_append(*selected_source)) { + _check_invariants(); + return; + } + if (!_shredded->can_materialize() && !selected_source->can_materialize()) { + // Exchange channels can gather projected rows from files whose shredded leaf types + // differ. Preserve the segments instead of encoding an incomplete logical Variant. + _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); + _check_invariants(); + return; + } + } + if (_shredded) { + ensure_encoded(); + } if (source._shredded) { insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); return; diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index abe4df4b8223d4..e5be6e67e88b44 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -53,6 +53,9 @@ struct VariantShreddedTypedValue { // retain the decoded leaf without copying it or depending on scanner lifetime. ColumnPtr column; DataTypePtr type; + // Physical identities such as binary annotations cannot use the typed scalar state. In that + // case the format reader may return an exact Nullable leaf instead. + ColumnPtr normalized; }; // Format readers keep their native shredded representation behind this interface. Core Variant @@ -76,6 +79,8 @@ class VariantShreddedState { size_t length) const = 0; virtual std::shared_ptr select_indices( const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; + // False means the state contains only projected leaves and cannot reconstruct root values. + virtual bool can_materialize() const = 0; // Appends another state only when both format-owned physical layouts have identical semantics. // An incompatible source must leave this state unchanged and return false. virtual bool try_append(const VariantShreddedState& source) = 0; diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 90863fe86c5c72..2c506c05377e65 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -152,7 +152,8 @@ std::optional extract_shredded_typed_variant_element( if (!match.has_value()) { return std::nullopt; } - const auto& leaf = assert_cast(*match->column); + const ColumnPtr& matched_column = match->normalized ? match->normalized : match->column; + const auto& leaf = assert_cast(*matched_column); auto nulls = leaf.get_null_map_column().clone_resized(source.size()); auto& null_data = assert_cast(*nulls).get_data(); for (size_t row = 0; row < source.size(); ++row) { @@ -160,6 +161,10 @@ std::optional extract_shredded_typed_variant_element( static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); } + if (match->normalized) { + return ColumnNullable::create(leaf.get_nested_column_ptr(), std::move(nulls)); + } + // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null // map is produced here, so predicates and casts can consume the leaf without reconstructing // canonical Variant rows. diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index da5ecbb8310a20..09ec88841e459e 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -553,6 +553,25 @@ bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { } } +ColumnPtr normalize_projected_primitive_leaf(const ParquetColumnSchema& schema, + const ColumnPtr& typed) { + const auto& nullable = assert_cast(*typed); + VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = nullable.size()}); + for (size_t row = 0; row < nullable.size(); ++row) { + auto output_row = builder.begin_row(); + if (nullable.get_null_map_data()[row] != 0) { + output_row.add_null(); + } else { + append_typed_scalar(schema, nullable.get_nested_column(), row, output_row); + } + output_row.finish(); + } + auto values = ColumnVariantV2::create(); + values->insert_encoded_batch(builder.finish_batch()); + auto nulls = nullable.get_null_map_column().clone_resized(nullable.size()); + return ColumnNullable::create(std::move(values), std::move(nulls)); +} + bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { return (!left && !right) || (left && right && left->equals(*right)); } @@ -648,6 +667,8 @@ class ParquetVariantShreddedState final : public VariantShreddedState { _complete, _profile); } + bool can_materialize() const override { return _complete; } + bool try_append(const VariantShreddedState& source) override { const auto* parquet_source = dynamic_cast(&source); if (parquet_source == nullptr || _complete != parquet_source->_complete || @@ -705,15 +726,29 @@ class ParquetVariantShreddedState final : public VariantShreddedState { } if (position + 1 == path.size()) { if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || - check_and_get_column(*typed) == nullptr || - !supports_direct_typed_variant_state(*typed_schema)) { + check_and_get_column(*typed) == nullptr) { update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); return std::nullopt; } + if (!supports_direct_typed_variant_state(*typed_schema)) { + if (_complete) { + update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); + return std::nullopt; + } + // A partial projection cannot reconstruct its root. Normalize only the exact + // requested leaf so Parquet annotations survive heterogeneous file schemas. + update_counter(_profile.variant_direct_leaf_rows, + static_cast(typed->size())); + return VariantShreddedTypedValue { + .column = nullptr, + .type = nullptr, + .normalized = normalize_projected_primitive_leaf(*typed_schema, typed)}; + } update_counter(_profile.variant_direct_leaf_rows, static_cast(typed->size())); return VariantShreddedTypedValue {.column = std::move(typed), - .type = remove_nullable(typed_schema->type)}; + .type = remove_nullable(typed_schema->type), + .normalized = nullptr}; } if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { return path_miss(); diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 51a50319285c5b..8dd080f6387253 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -237,6 +237,27 @@ MutableColumnPtr projected_shredded_object_physical(const std::vector& return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); } +MutableColumnPtr projected_shredded_binary_object_physical( + const std::vector& values) { + std::vector refs; + refs.reserve(values.size()); + for (const auto value : values) { + refs.emplace_back(value.data(), value.size()); + } + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_strings(refs, std::vector(values.size(), 0))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back( + ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); +} + MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); MutableColumnPtr nullable_int64(const std::vector& values, const std::vector& nulls); @@ -840,6 +861,126 @@ TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializin EXPECT_EQ(plan.variant_state_schema.use_count(), 3); } +TEST(VariantColumnReaderTest, GathersConsecutiveProjectedShreddedBatches) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto first = make_nullable(std::make_shared())->create_column(); + auto second = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE( + materialize_variant_columns(plan, projected_shredded_object_physical({10, 20}), first) + .ok()); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({30}), second) + .ok()); + + auto gathered = make_nullable(std::make_shared())->create_column(); + const std::array first_indices {1, 0}; + const std::array second_indices {0}; + gathered->insert_indices_from(*first, first_indices.begin(), first_indices.end()); + gathered->insert_indices_from(*second, second_indices.begin(), second_indices.end()); + + const auto& variants = assert_cast( + assert_cast(*gathered).get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + const auto& values = assert_cast( + assert_cast(*match->column).get_nested_column()); + EXPECT_EQ(values.get_data(), ColumnInt64::Container({20, 10, 30})); +} + +TEST(VariantColumnReaderTest, GathersProjectedShreddedBatchesWithDifferentLeafTypes) { + auto integer_schema = shredded_object_schema(); + auto string_schema = shredded_binary_object_schema(); + for (auto* schema : {&integer_schema, &string_schema}) { + schema->local_id = 0; + schema->children[2]->local_id = 2; + schema->children[2]->children[0]->local_id = 0; + schema->children[2]->children[0]->children[0]->local_id = 0; + } + auto make_plan = [](const ParquetColumnSchema& schema) { + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + return plan; + }; + auto integer_plan = make_plan(integer_schema); + auto string_plan = make_plan(string_schema); + auto integers = make_nullable(std::make_shared())->create_column(); + auto strings = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(integer_plan, projected_shredded_object_physical({7}), + integers) + .ok()); + ASSERT_TRUE(materialize_variant_columns( + string_plan, projected_shredded_binary_object_physical({"seven"}), strings) + .ok()); + + auto gathered = make_nullable(std::make_shared())->create_column(); + const std::array selected {0}; + gathered->insert_indices_from(*integers, selected.begin(), selected.end()); + gathered->insert_indices_from(*strings, selected.begin(), selected.end()); + + const auto& nullable = assert_cast(*gathered); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + ColumnPtr extracted; + ASSERT_TRUE( + extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) + .ok()); + const auto& extracted_variants = assert_cast( + assert_cast(*extracted).get_nested_column()); + EXPECT_EQ(extracted_variants.get_value_ref(0).get_int(), 7); + EXPECT_EQ(extracted_variants.get_value_ref(1).get_binary(), StringRef("seven")); + + const std::array shredded_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + auto reordered = ColumnVariantV2::create(); + const std::array reversed {1, 0}; + reordered->insert_indices_from(variants, reversed.begin(), reversed.end()); + const auto reordered_match = reordered->find_shredded_typed_value(shredded_path); + ASSERT_TRUE(reordered_match.has_value()); + ASSERT_TRUE(reordered_match->normalized); + const auto& reordered_values = assert_cast( + assert_cast(*reordered_match->normalized).get_nested_column()); + EXPECT_EQ(reordered_values.get_value_ref(0).get_binary(), StringRef("seven")); + EXPECT_EQ(reordered_values.get_value_ref(1).get_int(), 7); + + IColumn::Filter keep_integer {1, 0}; + const auto filtered = variants.filter(keep_integer, 1); + const auto filtered_match = + assert_cast(*filtered).find_shredded_typed_value(shredded_path); + ASSERT_TRUE(filtered_match.has_value()); + ASSERT_TRUE(filtered_match->column); + EXPECT_EQ( + assert_cast( + assert_cast(*filtered_match->column).get_nested_column()) + .get_data()[0], + 7); +} + TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { constexpr size_t width = 64; constexpr size_t batch_count = 16; From d5f0e2c6f1824003a4a9993614229309fde2a3a1 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 22:09:34 +0800 Subject: [PATCH 09/11] [fix](variant) Adapt shredded callback to master MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What problem does this PR solve? Issue Number: None Related PR: #66446 Problem Summary: Master exposes immutable subcolumn traversal through IColumn::ColumnCallback, while branch-4.1 still uses ImutableColumnCallback. Adapt the forward-port implementation to master’s callback contract so the composite shredded state overrides the interface and compiles. ### Release note None ### Check List (For Author) - Test: Unit Test - Behavior changed: No - Does this need documentation: No --- be/src/core/column/variant_v2/column_variant_v2.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index e0ba9bd048cb97..cb16694dd7f7cc 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -359,7 +359,8 @@ class CompositeVariantShreddedState final : public VariantShreddedState { } } - void for_each_subcolumn(const IColumn::ImutableColumnCallback& callback) const override { + // Preserve master's immutable traversal contract while forwarding into each segment. + void for_each_subcolumn(IColumn::ColumnCallback callback) const override { for (const auto& segment : _segments) { segment->for_each_subcolumn(callback); } From 5c8528a0324fd4140cceff2c5abe8095513260e4 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Wed, 5 Aug 2026 20:40:48 +0800 Subject: [PATCH 10/11] [fix](variant) Revert projected shredded state preservation ### What problem does this PR solve? Issue Number: None Related PR: #66446 Problem Summary: Remove the forward port of projected shredded Variant state preservation and its master-only callback adaptation so the branch returns exactly to the state before that pull request. ### Release note None ### Check List (For Author) - Test: No need to test (the resulting tree exactly matches the pre-#66446 state) - Behavior changed: Yes (removes the #66446 forward port) - Does this need documentation: No --- .../column/variant_v2/column_variant_v2.cpp | 294 +----------------- .../column/variant_v2/column_variant_v2.h | 5 - .../function/function_variant_element_v2.cpp | 7 +- .../parquet/reader/variant_column_reader.cpp | 41 +-- .../parquet/variant_column_reader_test.cpp | 141 --------- 5 files changed, 7 insertions(+), 481 deletions(-) diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index cb16694dd7f7cc..0d5a7fb74bb6df 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -22,11 +22,9 @@ #include #include #include -#include #include #include #include -#include #include "common/check.h" #include "common/exception.h" @@ -315,267 +313,6 @@ ValidatedTypedInput validate_typed_input(ColumnPtr column, DataTypePtr scalar_ty "ColumnVariantV2::{} is intentionally unsupported for Variant values", method); } -class CompositeVariantShreddedState final : public VariantShreddedState { -public: - explicit CompositeVariantShreddedState( - std::vector> segments) - : _segments(std::move(segments)) { - DORIS_CHECK(std::ranges::all_of(_segments, [](const auto& segment) { - return segment != nullptr; - })) << "composite Variant shredded segments must not be null"; - } - - size_t size() const override { - size_t rows = 0; - for (const auto& segment : _segments) { - DORIS_CHECK_LE(segment->size(), std::numeric_limits::max() - rows) - << "composite Variant shredded row count overflows size_t"; - rows += segment->size(); - } - return rows; - } - - size_t byte_size() const override { - size_t bytes = 0; - for (const auto& segment : _segments) { - bytes += segment->byte_size(); - } - std::lock_guard lock(_materialization_lock); - return bytes + (_materialized ? _materialized->byte_size() : 0); - } - - size_t allocated_bytes() const override { - size_t bytes = 0; - for (const auto& segment : _segments) { - bytes += segment->allocated_bytes(); - } - std::lock_guard lock(_materialization_lock); - return bytes + (_materialized ? _materialized->allocated_bytes() : 0); - } - - void sanity_check() const override { - for (const auto& segment : _segments) { - segment->sanity_check(); - } - } - - // Preserve master's immutable traversal contract while forwarding into each segment. - void for_each_subcolumn(IColumn::ColumnCallback callback) const override { - for (const auto& segment : _segments) { - segment->for_each_subcolumn(callback); - } - } - - std::shared_ptr filter(const IColumn::Filter& filter, - ssize_t /*result_size_hint*/) const override { - DORIS_CHECK_EQ(filter.size(), size()) - << "composite Variant shredded filter size does not match row count"; - std::vector> selected; - selected.reserve(_segments.size()); - size_t offset = 0; - for (const auto& segment : _segments) { - IColumn::Filter segment_filter; - segment_filter.insert(filter.begin() + offset, - filter.begin() + offset + segment->size()); - auto filtered = segment->filter(segment_filter, -1); - if (filtered->size() != 0) { - selected.push_back(std::move(filtered)); - } - offset += segment->size(); - } - return pack(std::move(selected)); - } - - std::shared_ptr select_range(size_t start, size_t length) const override { - DORIS_CHECK_LE(start, size()) << "composite Variant range starts past source size"; - DORIS_CHECK_LE(length, size() - start) << "composite Variant range exceeds source size"; - std::vector> selected; - if (length == 0) { - return pack(std::move(selected)); - } - const size_t end = start + length; - size_t offset = 0; - for (const auto& segment : _segments) { - const size_t segment_end = offset + segment->size(); - const size_t overlap_begin = std::max(start, offset); - const size_t overlap_end = std::min(end, segment_end); - if (overlap_begin < overlap_end) { - selected.push_back( - segment->select_range(overlap_begin - offset, overlap_end - overlap_begin)); - } - offset = segment_end; - if (offset >= end) { - break; - } - } - return pack(std::move(selected)); - } - - std::shared_ptr select_indices( - const uint32_t* indices_begin, const uint32_t* indices_end) const override { - if (indices_begin == indices_end) { - return pack({}); - } - DORIS_CHECK(indices_begin != nullptr && indices_end != nullptr && - indices_begin < indices_end) - << "composite Variant indices are invalid"; - - std::vector segment_ends; - segment_ends.reserve(_segments.size()); - size_t rows = 0; - for (const auto& segment : _segments) { - rows += segment->size(); - segment_ends.push_back(rows); - } - - std::vector> selected; - const uint32_t* cursor = indices_begin; - while (cursor != indices_end) { - DORIS_CHECK_LT(*cursor, rows) << "composite Variant source index is out of range"; - const size_t segment_index = - std::upper_bound(segment_ends.begin(), segment_ends.end(), *cursor) - - segment_ends.begin(); - const size_t segment_begin = segment_index == 0 ? 0 : segment_ends[segment_index - 1]; - DorisVector local_indices; - while (cursor != indices_end && *cursor >= segment_begin && - *cursor < segment_ends[segment_index]) { - local_indices.push_back(static_cast(*cursor - segment_begin)); - ++cursor; - } - selected.push_back(_segments[segment_index]->select_indices( - local_indices.data(), local_indices.data() + local_indices.size())); - } - return pack(std::move(selected)); - } - - bool can_materialize() const override { - return std::ranges::all_of(_segments, - [](const auto& segment) { return segment->can_materialize(); }); - } - - bool try_append(const VariantShreddedState& source) override { - if (const auto* composite = dynamic_cast(&source)) { - for (const auto& segment : composite->_segments) { - append(segment); - } - } else { - append(source.select_range(0, source.size())); - } - std::lock_guard lock(_materialization_lock); - _materialized.reset(); - return true; - } - - std::optional find_typed_value( - std::span path) const override { - if (_segments.empty()) { - return std::nullopt; - } - std::vector matches; - matches.reserve(_segments.size()); - for (const auto& segment : _segments) { - auto match = segment->find_typed_value(path); - if (!match.has_value()) { - return std::nullopt; - } - matches.push_back(std::move(*match)); - } - - const bool homogeneous = matches.front().column && matches.front().type && - std::ranges::all_of(matches, [&](const auto& match) { - return match.column && match.type && !match.normalized && - exact_typed_identity(matches.front().type, match.type); - }); - if (homogeneous) { - MutableColumnPtr combined = matches.front().column->clone_empty(); - for (const auto& match : matches) { - combined->insert_range_from(*match.column, 0, match.column->size()); - } - return VariantShreddedTypedValue {.column = std::move(combined), - .type = matches.front().type, - .normalized = nullptr}; - } - - auto values = ColumnVariantV2::create(); - auto nulls = ColumnUInt8::create(); - nulls->reserve(size()); - for (const auto& match : matches) { - if (match.normalized) { - const auto& nullable = assert_cast(*match.normalized); - const auto& variants = - assert_cast(nullable.get_nested_column()); - values->insert_range_from(variants, 0, variants.size()); - nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); - continue; - } - auto typed = ColumnVariantV2::create_typed(match.column, match.type); - values->insert_range_from(*typed, 0, typed->size()); - const auto& nullable = assert_cast(*match.column); - nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); - } - return VariantShreddedTypedValue { - .column = nullptr, - .type = nullptr, - .normalized = ColumnNullable::create(std::move(values), std::move(nulls))}; - } - - const ColumnVariantV2& materialized_column() const override { - std::lock_guard lock(_materialization_lock); - if (!_materialized) { - auto materialized = ColumnVariantV2::create(); - for (const auto& segment : _segments) { - const ColumnVariantV2& source = segment->materialized_column(); - materialized->insert_range_from(source, 0, source.size()); - } - _materialized = std::move(materialized); - } - return *_materialized; - } - -private: - static std::shared_ptr pack( - std::vector> segments) { - if (segments.size() == 1) { - return std::move(segments.front()); - } - return std::make_shared(std::move(segments)); - } - - void append(std::shared_ptr source) { - if (source->size() == 0) { - return; - } - if (const auto* composite = - dynamic_cast(source.get())) { - _segments.insert(_segments.end(), composite->_segments.begin(), - composite->_segments.end()); - return; - } - if (!_segments.empty()) { - auto& tail = _segments.back(); - if (tail.use_count() != 1) { - tail = tail->select_range(0, tail->size()); - } - if (tail->try_append(*source)) { - return; - } - } - _segments.push_back(std::move(source)); - } - - std::vector> _segments; - mutable std::mutex _materialization_lock; - mutable ColumnVariantV2::MutablePtr _materialized; -}; - -std::shared_ptr combine_shredded_states( - std::shared_ptr left, std::shared_ptr right) { - auto combined = std::make_shared( - std::vector> {std::move(left)}); - combined->try_append(*right); - return combined; -} - } // namespace #ifdef BE_TEST @@ -1081,14 +818,6 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) _check_invariants(); return; } - if (!_shredded->can_materialize() && !selected_source->can_materialize()) { - // Different files may shred the same projected path with different physical - // identities. Keep both incomplete states ordered because neither can reconstruct the - // root value. - _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); - _check_invariants(); - return; - } } if (_shredded) { ensure_encoded(); @@ -1182,6 +911,9 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } + if (_shredded) { + ensure_encoded(); + } if (!_typed && empty() && _metadatas->empty() && source._shredded) { // Gather into the native shredded representation for the same reason as range selection: // row selection does not require, and may not have, a complete logical Variant value. @@ -1189,26 +921,6 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) _check_invariants(); return; } - if (_shredded && source._shredded) { - auto selected_source = source._shredded->select_indices(indices_begin, indices_end); - if (_shredded.use_count() != 1) { - _shredded = _shredded->select_range(0, size()); - } - if (_shredded->try_append(*selected_source)) { - _check_invariants(); - return; - } - if (!_shredded->can_materialize() && !selected_source->can_materialize()) { - // Exchange channels can gather projected rows from files whose shredded leaf types - // differ. Preserve the segments instead of encoding an incomplete logical Variant. - _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); - _check_invariants(); - return; - } - } - if (_shredded) { - ensure_encoded(); - } if (source._shredded) { insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); return; diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index e5be6e67e88b44..abe4df4b8223d4 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -53,9 +53,6 @@ struct VariantShreddedTypedValue { // retain the decoded leaf without copying it or depending on scanner lifetime. ColumnPtr column; DataTypePtr type; - // Physical identities such as binary annotations cannot use the typed scalar state. In that - // case the format reader may return an exact Nullable leaf instead. - ColumnPtr normalized; }; // Format readers keep their native shredded representation behind this interface. Core Variant @@ -79,8 +76,6 @@ class VariantShreddedState { size_t length) const = 0; virtual std::shared_ptr select_indices( const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; - // False means the state contains only projected leaves and cannot reconstruct root values. - virtual bool can_materialize() const = 0; // Appends another state only when both format-owned physical layouts have identical semantics. // An incompatible source must leave this state unchanged and return false. virtual bool try_append(const VariantShreddedState& source) = 0; diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 2c506c05377e65..90863fe86c5c72 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -152,8 +152,7 @@ std::optional extract_shredded_typed_variant_element( if (!match.has_value()) { return std::nullopt; } - const ColumnPtr& matched_column = match->normalized ? match->normalized : match->column; - const auto& leaf = assert_cast(*matched_column); + const auto& leaf = assert_cast(*match->column); auto nulls = leaf.get_null_map_column().clone_resized(source.size()); auto& null_data = assert_cast(*nulls).get_data(); for (size_t row = 0; row < source.size(); ++row) { @@ -161,10 +160,6 @@ std::optional extract_shredded_typed_variant_element( static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); } - if (match->normalized) { - return ColumnNullable::create(leaf.get_nested_column_ptr(), std::move(nulls)); - } - // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null // map is produced here, so predicates and casts can consume the leaf without reconstructing // canonical Variant rows. diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index 09ec88841e459e..da5ecbb8310a20 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -553,25 +553,6 @@ bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { } } -ColumnPtr normalize_projected_primitive_leaf(const ParquetColumnSchema& schema, - const ColumnPtr& typed) { - const auto& nullable = assert_cast(*typed); - VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = nullable.size()}); - for (size_t row = 0; row < nullable.size(); ++row) { - auto output_row = builder.begin_row(); - if (nullable.get_null_map_data()[row] != 0) { - output_row.add_null(); - } else { - append_typed_scalar(schema, nullable.get_nested_column(), row, output_row); - } - output_row.finish(); - } - auto values = ColumnVariantV2::create(); - values->insert_encoded_batch(builder.finish_batch()); - auto nulls = nullable.get_null_map_column().clone_resized(nullable.size()); - return ColumnNullable::create(std::move(values), std::move(nulls)); -} - bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { return (!left && !right) || (left && right && left->equals(*right)); } @@ -667,8 +648,6 @@ class ParquetVariantShreddedState final : public VariantShreddedState { _complete, _profile); } - bool can_materialize() const override { return _complete; } - bool try_append(const VariantShreddedState& source) override { const auto* parquet_source = dynamic_cast(&source); if (parquet_source == nullptr || _complete != parquet_source->_complete || @@ -726,29 +705,15 @@ class ParquetVariantShreddedState final : public VariantShreddedState { } if (position + 1 == path.size()) { if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || - check_and_get_column(*typed) == nullptr) { + check_and_get_column(*typed) == nullptr || + !supports_direct_typed_variant_state(*typed_schema)) { update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); return std::nullopt; } - if (!supports_direct_typed_variant_state(*typed_schema)) { - if (_complete) { - update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); - return std::nullopt; - } - // A partial projection cannot reconstruct its root. Normalize only the exact - // requested leaf so Parquet annotations survive heterogeneous file schemas. - update_counter(_profile.variant_direct_leaf_rows, - static_cast(typed->size())); - return VariantShreddedTypedValue { - .column = nullptr, - .type = nullptr, - .normalized = normalize_projected_primitive_leaf(*typed_schema, typed)}; - } update_counter(_profile.variant_direct_leaf_rows, static_cast(typed->size())); return VariantShreddedTypedValue {.column = std::move(typed), - .type = remove_nullable(typed_schema->type), - .normalized = nullptr}; + .type = remove_nullable(typed_schema->type)}; } if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { return path_miss(); diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 8dd080f6387253..51a50319285c5b 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -237,27 +237,6 @@ MutableColumnPtr projected_shredded_object_physical(const std::vector& return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); } -MutableColumnPtr projected_shredded_binary_object_physical( - const std::vector& values) { - std::vector refs; - refs.reserve(values.size()); - for (const auto value : values) { - refs.emplace_back(value.data(), value.size()); - } - MutableColumns wrapper_fields; - wrapper_fields.push_back(nullable_strings(refs, std::vector(values.size(), 0))); - auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); - MutableColumns object_fields; - object_fields.push_back( - ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); - auto object = ColumnStruct::create(std::move(object_fields)); - MutableColumns root_fields; - root_fields.push_back( - ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); - auto root = ColumnStruct::create(std::move(root_fields)); - return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); -} - MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); MutableColumnPtr nullable_int64(const std::vector& values, const std::vector& nulls); @@ -861,126 +840,6 @@ TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializin EXPECT_EQ(plan.variant_state_schema.use_count(), 3); } -TEST(VariantColumnReaderTest, GathersConsecutiveProjectedShreddedBatches) { - auto schema = shredded_object_schema(); - schema.local_id = 0; - schema.children[2]->local_id = 2; - schema.children[2]->children[0]->local_id = 0; - schema.children[2]->children[0]->children[0]->local_id = 0; - auto projection = format::LocalColumnIndex::partial_local(0); - projection.children.push_back(format::LocalColumnIndex::partial_local(2)); - projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); - projection.children.back().children.back().children.push_back( - format::LocalColumnIndex::local(0)); - VariantMaterializationNode plan; - plan.schema = &schema; - plan.contains_variant = true; - plan.variant_projection = std::move(projection); - plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); - - auto first = make_nullable(std::make_shared())->create_column(); - auto second = make_nullable(std::make_shared())->create_column(); - ASSERT_TRUE( - materialize_variant_columns(plan, projected_shredded_object_physical({10, 20}), first) - .ok()); - ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({30}), second) - .ok()); - - auto gathered = make_nullable(std::make_shared())->create_column(); - const std::array first_indices {1, 0}; - const std::array second_indices {0}; - gathered->insert_indices_from(*first, first_indices.begin(), first_indices.end()); - gathered->insert_indices_from(*second, second_indices.begin(), second_indices.end()); - - const auto& variants = assert_cast( - assert_cast(*gathered).get_nested_column()); - const std::array path {VariantShreddedPathSegment { - .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; - const auto match = variants.find_shredded_typed_value(path); - ASSERT_TRUE(match.has_value()); - const auto& values = assert_cast( - assert_cast(*match->column).get_nested_column()); - EXPECT_EQ(values.get_data(), ColumnInt64::Container({20, 10, 30})); -} - -TEST(VariantColumnReaderTest, GathersProjectedShreddedBatchesWithDifferentLeafTypes) { - auto integer_schema = shredded_object_schema(); - auto string_schema = shredded_binary_object_schema(); - for (auto* schema : {&integer_schema, &string_schema}) { - schema->local_id = 0; - schema->children[2]->local_id = 2; - schema->children[2]->children[0]->local_id = 0; - schema->children[2]->children[0]->children[0]->local_id = 0; - } - auto make_plan = [](const ParquetColumnSchema& schema) { - auto projection = format::LocalColumnIndex::partial_local(0); - projection.children.push_back(format::LocalColumnIndex::partial_local(2)); - projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); - projection.children.back().children.back().children.push_back( - format::LocalColumnIndex::local(0)); - VariantMaterializationNode plan; - plan.schema = &schema; - plan.contains_variant = true; - plan.variant_projection = std::move(projection); - plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); - return plan; - }; - auto integer_plan = make_plan(integer_schema); - auto string_plan = make_plan(string_schema); - auto integers = make_nullable(std::make_shared())->create_column(); - auto strings = make_nullable(std::make_shared())->create_column(); - ASSERT_TRUE(materialize_variant_columns(integer_plan, projected_shredded_object_physical({7}), - integers) - .ok()); - ASSERT_TRUE(materialize_variant_columns( - string_plan, projected_shredded_binary_object_physical({"seven"}), strings) - .ok()); - - auto gathered = make_nullable(std::make_shared())->create_column(); - const std::array selected {0}; - gathered->insert_indices_from(*integers, selected.begin(), selected.end()); - gathered->insert_indices_from(*strings, selected.begin(), selected.end()); - - const auto& nullable = assert_cast(*gathered); - const auto& variants = assert_cast(nullable.get_nested_column()); - const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; - std::unique_ptr path; - ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); - ColumnPtr extracted; - ASSERT_TRUE( - extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) - .ok()); - const auto& extracted_variants = assert_cast( - assert_cast(*extracted).get_nested_column()); - EXPECT_EQ(extracted_variants.get_value_ref(0).get_int(), 7); - EXPECT_EQ(extracted_variants.get_value_ref(1).get_binary(), StringRef("seven")); - - const std::array shredded_path {VariantShreddedPathSegment { - .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; - auto reordered = ColumnVariantV2::create(); - const std::array reversed {1, 0}; - reordered->insert_indices_from(variants, reversed.begin(), reversed.end()); - const auto reordered_match = reordered->find_shredded_typed_value(shredded_path); - ASSERT_TRUE(reordered_match.has_value()); - ASSERT_TRUE(reordered_match->normalized); - const auto& reordered_values = assert_cast( - assert_cast(*reordered_match->normalized).get_nested_column()); - EXPECT_EQ(reordered_values.get_value_ref(0).get_binary(), StringRef("seven")); - EXPECT_EQ(reordered_values.get_value_ref(1).get_int(), 7); - - IColumn::Filter keep_integer {1, 0}; - const auto filtered = variants.filter(keep_integer, 1); - const auto filtered_match = - assert_cast(*filtered).find_shredded_typed_value(shredded_path); - ASSERT_TRUE(filtered_match.has_value()); - ASSERT_TRUE(filtered_match->column); - EXPECT_EQ( - assert_cast( - assert_cast(*filtered_match->column).get_nested_column()) - .get_data()[0], - 7); -} - TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { constexpr size_t width = 64; constexpr size_t batch_count = 16; From 478fa646179dbd581e67fe55a68e974bc4b81ffc Mon Sep 17 00:00:00 2001 From: Gabriel Date: Wed, 5 Aug 2026 20:42:08 +0800 Subject: [PATCH 11/11] [improvement](parquet) Use predicate tree for compound page pruning ### What problem does this PR solve? Issue Number: None Related PR: #66412 Problem Summary: Forward-port compound Parquet Page Index pruning to master. AND nodes intersect conservative candidate ranges, OR nodes union them, and unavailable OR branches retain the complete range. The master adaptation also keeps compound predicates behind the existing metadata-pruning safety fence. ### Release note Support compound Parquet Page Index pruning in File Scanner V2. ### Check List (For Author) - Test: Unit Test (2 focused BE ASAN tests passed) - Behavior changed: Yes (enables conservative compound Page Index pruning) - Does this need documentation: Yes (included in this commit) --- be/benchmark/parquet/AGENTS.md | 8 +- be/benchmark/parquet/README.md | 12 + .../parquet/benchmark_parquet_reader.hpp | 169 ++++++++++++++ .../format_v2/parquet/parquet_statistics.cpp | 209 +++++++++++++++++- .../format_v2/parquet/parquet_scan_test.cpp | 57 +++++ .../parquet/parquet_statistics_test.cpp | 171 ++++++++++++++ docs/file-scanner-v2-parquet-scan-design.md | 17 +- 7 files changed, 628 insertions(+), 15 deletions(-) diff --git a/be/benchmark/parquet/AGENTS.md b/be/benchmark/parquet/AGENTS.md index 4d8c0610f17b5d..da51348852002e 100644 --- a/be/benchmark/parquet/AGENTS.md +++ b/be/benchmark/parquet/AGENTS.md @@ -57,7 +57,7 @@ be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetSelection/' # currently 25 be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetReader/' # currently 167 + | grep -c '^ParquetReader/' # currently 169 be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^FileScannerExpr/' # currently 8 @@ -189,6 +189,10 @@ Except for the axis being varied, reader cases inherit the baseline: nullable IN alternating 10% nulls, 10% selectivity, 32 columns, predicate at column zero, and predicate plus payload projection. +Two dedicated multi-column OR cases scan the same Page Index fixture and change only the Doris Page +Index switch. They retain the complete residual expression and validate the same selected row +count before reporting throughput. + ## How decoder data is generated Decoder pages are constructed in memory before the timed loop. There is no Parquet file, Python @@ -346,7 +350,7 @@ be simulated by silently changing the local reader benchmark. ## Current validation record -The current expected registration counts are 228 decoder, 92 kernel, 25 selection, 167 reader, and +The current expected registration counts are 228 decoder, 92 kernel, 25 selection, 169 reader, and 8 expression-lifecycle cases. A smoke run is an execution record only, not a reviewed performance baseline, because repetitions, host isolation, warmups, cache control, `perf` data, variance, and before/after comparison are not collected. diff --git a/be/benchmark/parquet/README.md b/be/benchmark/parquet/README.md index 891fd02d8375ed..b4761855f5ceae 100644 --- a/be/benchmark/parquet/README.md +++ b/be/benchmark/parquet/README.md @@ -137,6 +137,18 @@ be/output/lib/benchmark_test \ --benchmark_min_time=1s ``` +The multi-column OR pair scans the same ColumnIndex/OffsetIndex fixture and changes only the Doris +Page Index switch. Both variants retain the full residual expression, so the comparison measures +metadata pruning without changing result semantics: + +```shell +be/output/lib/benchmark_test \ + --benchmark_filter='^ParquetReader/multi_column_or/page_index_(off|on)$' \ + --benchmark_min_time=1s \ + --benchmark_repetitions=10 \ + --benchmark_report_aggregates_only=true +``` + Every result reports throughput plus `raw_rows`, `selected_rows`, `fixture_bytes`, `ns/raw_row`, and (when at least one row survives) `ns/selected_row`. Keep CPU frequency, build type, compiler, machine placement, and benchmark filters fixed when comparing two commits. diff --git a/be/benchmark/parquet/benchmark_parquet_reader.hpp b/be/benchmark/parquet/benchmark_parquet_reader.hpp index b456244ba0e3ba..4763ffe4c9867e 100644 --- a/be/benchmark/parquet/benchmark_parquet_reader.hpp +++ b/be/benchmark/parquet/benchmark_parquet_reader.hpp @@ -34,6 +34,7 @@ #include #include +#include "common/config.h" #include "core/assert_cast.h" #include "core/block/block.h" #include "core/column/column_nullable.h" @@ -61,6 +62,19 @@ namespace reader_detail { constexpr size_t READER_ROWS = 1UL << 14; constexpr size_t READER_ROW_GROUP_ROWS = 1UL << 12; +constexpr size_t MULTI_COLUMN_OR_ROWS = 1UL << 20; +constexpr size_t MULTI_COLUMN_OR_ROW_GROUP_ROWS = 1UL << 18; + +class ScopedPageIndexConfig { +public: + explicit ScopedPageIndexConfig(bool enabled) : _previous(config::enable_parquet_page_index) { + config::enable_parquet_page_index = enabled; + } + ~ScopedPageIndexConfig() { config::enable_parquet_page_index = _previous; } + +private: + bool _previous; +}; inline void throw_if_error(const Status& status) { if (!status.ok()) { @@ -256,6 +270,8 @@ class Int32LessThanExpr final : public VExpr { _column_id(column_id), _upper_bound(upper_bound) {} + bool is_constant() const override { return false; } + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, size_t count, ColumnPtr& result_column) const override { DORIS_CHECK(block != nullptr); @@ -612,6 +628,151 @@ inline void run_reader(benchmark::State& state, ReaderScenario scenario) { } } +inline std::filesystem::path ensure_multi_column_or_fixture() { + static std::mutex fixture_mutex; + const auto directory = + std::filesystem::temp_directory_path() / "doris_parquet_reader_benchmark"; + const auto path = directory / "v2_multi_column_or_page_index_v2.parquet"; + std::lock_guard guard(fixture_mutex); + if (std::filesystem::exists(path)) { + return path; + } + + arrow::Int32Builder ascending_builder; + arrow::Int32Builder descending_builder; + arrow::Int32Builder payload_builder; + PARQUET_THROW_NOT_OK(ascending_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + PARQUET_THROW_NOT_OK(descending_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + PARQUET_THROW_NOT_OK(payload_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + for (size_t row = 0; row < MULTI_COLUMN_OR_ROWS; ++row) { + const int32_t row_in_group = static_cast(row % MULTI_COLUMN_OR_ROW_GROUP_ROWS); + PARQUET_THROW_NOT_OK(ascending_builder.Append(row_in_group)); + PARQUET_THROW_NOT_OK(descending_builder.Append( + static_cast(MULTI_COLUMN_OR_ROW_GROUP_ROWS - 1) - row_in_group)); + PARQUET_THROW_NOT_OK(payload_builder.Append(static_cast(row))); + } + auto table = arrow::Table::Make( + arrow::schema({arrow::field("ascending", arrow::int32(), true), + arrow::field("descending", arrow::int32(), true), + arrow::field("payload", arrow::int32(), true)}), + {ascending_builder.Finish().ValueOrDie(), descending_builder.Finish().ValueOrDie(), + payload_builder.Finish().ValueOrDie()}); + + std::filesystem::create_directories(directory); + const auto temporary_path = path.string() + ".tmp"; + std::filesystem::remove(temporary_path); + const auto output_result = arrow::io::FileOutputStream::Open(temporary_path); + if (!output_result.ok()) { + throw std::runtime_error(output_result.status().ToString()); + } + const auto output = *output_result; + ::parquet::WriterProperties::Builder properties; + properties.version(::parquet::ParquetVersion::PARQUET_2_6); + properties.data_page_version(::parquet::ParquetDataPageVersion::V2); + properties.compression(::parquet::Compression::UNCOMPRESSED); + properties.disable_dictionary(); + properties.encoding(::parquet::Encoding::PLAIN); + properties.enable_write_page_index(); + properties.write_batch_size(8192); + properties.data_pagesize(64 * 1024); + PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), output, + MULTI_COLUMN_OR_ROW_GROUP_ROWS, + properties.build())); + PARQUET_THROW_NOT_OK(output->Close()); + std::filesystem::rename(temporary_path, path); + return path; +} + +inline std::unique_ptr open_multi_column_or_reader( + const std::filesystem::path& path) { + auto session = std::make_unique(); + auto properties = std::make_shared(); + properties->system_type = TFileType::FILE_LOCAL; + auto description = std::make_unique(); + description->path = path.string(); + description->file_size = static_cast(std::filesystem::file_size(path)); + description->range_start_offset = 0; + description->range_size = -1; + session->reader = std::make_unique(properties, description, + nullptr, nullptr); + throw_if_error(session->reader->init(&session->runtime_state)); + throw_if_error(session->reader->get_schema(&session->schema)); + + session->request = std::make_shared(); + format::FileScanRequestBuilder request_builder(session->request.get()); + std::array predicate_positions {}; + for (int column = 0; column < 2; ++column) { + const auto column_id = format::LocalColumnId(column); + throw_if_error(request_builder.add_predicate_column(column_id)); + session->request->predicate_only_columns.push_back(column_id); + predicate_positions[column] = + static_cast(session->request->local_positions.at(column_id).value()); + } + throw_if_error(request_builder.add_non_predicate_column(format::LocalColumnId(2))); + + TExprNode node; + node.__set_node_type(TExprNodeType::COMPOUND_PRED); + node.__set_opcode(TExprOpcode::COMPOUND_OR); + node.__set_type(std::make_shared()->to_thrift()); + node.__set_num_children(2); + node.__set_is_nullable(false); + auto compound = VCompoundPred::create_shared(node); + constexpr int32_t UPPER_BOUND = static_cast(MULTI_COLUMN_OR_ROW_GROUP_ROWS / 10); + compound->add_child(std::make_shared(predicate_positions[0], UPPER_BOUND)); + compound->add_child(std::make_shared(predicate_positions[1], UPPER_BOUND)); + auto context = VExprContext::create_shared(std::move(compound)); + throw_if_error(context->prepare(&session->runtime_state, RowDescriptor())); + throw_if_error(context->open(&session->runtime_state)); + session->request->conjuncts.push_back(context); + session->opened_conjuncts.push_back(std::move(context)); + throw_if_error(session->reader->open(session->request)); + return session; +} + +inline void run_multi_column_or_reader(benchmark::State& state, bool enable_page_index) { + try { + const auto fixture = ensure_multi_column_or_fixture(); + ScopedPageIndexConfig page_index_config(enable_page_index); + size_t selected_rows = 0; + for (auto _ : state) { + state.PauseTiming(); + auto session = open_multi_column_or_reader(fixture); + state.ResumeTiming(); + const ReaderScenario scenario {.operation = ReaderOperation::PREDICATE_SCAN, + .encoding = Encoding::PLAIN, + .null_percent = 0, + .null_pattern = Pattern::CLUSTERED, + .selectivity_percent = 20, + .projection = Projection::PREDICATE_ONLY, + .schema_width = 3, + .predicate_position = 0}; + selected_rows = scan_reader(session.get(), scenario); + state.PauseTiming(); + throw_if_error(session->reader->close()); + state.ResumeTiming(); + benchmark::ClobberMemory(); + } + constexpr size_t ROW_GROUPS = MULTI_COLUMN_OR_ROWS / MULTI_COLUMN_OR_ROW_GROUP_ROWS; + constexpr size_t EXPECTED_ROWS = ROW_GROUPS * 2 * (MULTI_COLUMN_OR_ROW_GROUP_ROWS / 10); + if (selected_rows != EXPECTED_ROWS) { + state.SkipWithError("multi-column OR benchmark returned unexpected rows"); + return; + } + state.SetItemsProcessed(static_cast(state.iterations() * selected_rows)); + state.counters["raw_rows"] = static_cast(MULTI_COLUMN_OR_ROWS); + state.counters["selected_rows"] = static_cast(selected_rows); + state.counters["fixture_bytes"] = static_cast(std::filesystem::file_size(fixture)); + state.counters["ns/raw_row"] = benchmark::Counter( + static_cast(MULTI_COLUMN_OR_ROWS), + benchmark::Counter::kIsIterationInvariantRate | benchmark::Counter::kInvert); + state.counters["ns/selected_row"] = benchmark::Counter( + static_cast(selected_rows), + benchmark::Counter::kIsIterationInvariantRate | benchmark::Counter::kInvert); + } catch (const std::exception& error) { + state.SkipWithError(error.what()); + } +} + inline bool register_reader_benchmarks() { for (const auto& scenario : reader_scenarios()) { std::string name = "ParquetReader/" + reader_scenario_name(scenario); @@ -619,6 +780,14 @@ inline bool register_reader_benchmarks() { run_reader(state, scenario); })->Unit(benchmark::kNanosecond); } + benchmark::RegisterBenchmark( + "ParquetReader/multi_column_or/page_index_off", + [](benchmark::State& state) { run_multi_column_or_reader(state, false); }) + ->Unit(benchmark::kNanosecond); + benchmark::RegisterBenchmark( + "ParquetReader/multi_column_or/page_index_on", + [](benchmark::State& state) { run_multi_column_or_reader(state, true); }) + ->Unit(benchmark::kNanosecond); return true; } diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 08779056c73ee6..38a7a03c194784 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -1376,6 +1376,38 @@ std::vector intersect_ranges(const std::vector& left, return result; } +std::vector union_ranges(const std::vector& left, + const std::vector& right) { + std::vector result; + result.reserve(left.size() + right.size()); + auto append = [&](const RowRange& range) { + if (range.length == 0) { + return; + } + if (!result.empty()) { + auto& previous = result.back(); + const int64_t previous_end = previous.start + previous.length; + if (range.start <= previous_end) { + previous.length = + std::max(previous_end, range.start + range.length) - previous.start; + return; + } + } + result.push_back(range); + }; + size_t left_idx = 0; + size_t right_idx = 0; + while (left_idx < left.size() || right_idx < right.size()) { + if (right_idx == right.size() || + (left_idx < left.size() && left[left_idx].start <= right[right_idx].start)) { + append(left[left_idx++]); + } else { + append(right[right_idx++]); + } + } + return result; +} + int64_t count_range_rows(const std::vector& ranges) { int64_t rows = 0; for (const auto& range : ranges) { @@ -1632,6 +1664,154 @@ RowRange native_page_row_range(const tparquet::OffsetIndex& offset_index, size_t return {.start = start, .length = end - start}; } +class NativePageIndexPredicateEvaluator { +public: + NativePageIndexPredicateEvaluator( + const tparquet::FileMetaData& metadata, + const std::unordered_map& page_indexes, + const std::vector>& file_schema, + const format::FileScanRequest& request, int64_t row_group_rows, + ParquetPruningStats* pruning_stats, const cctz::time_zone* timezone) + : _metadata(metadata), + _page_indexes(page_indexes), + _file_schema(file_schema), + _request(request), + _row_group_rows(row_group_rows), + _pruning_stats(pruning_stats), + _timezone(timezone) {} + + std::optional> evaluate(const VExprSPtr& expr) const { + if (expr == nullptr || !expr->can_evaluate_zonemap_filter()) { + return std::nullopt; + } + if (expr->op() == TExprOpcode::COMPOUND_AND) { + return evaluate_compound(expr, true); + } + if (expr->op() == TExprOpcode::COMPOUND_OR) { + return evaluate_compound(expr, false); + } + return evaluate_leaf(expr); + } + +private: + struct SlotPageZoneMaps { + DataTypePtr data_type; + std::vector ranges; + std::vector> zone_maps; + }; + + std::optional> evaluate_compound(const VExprSPtr& expr, + bool is_and) const { + std::optional> ranges; + for (const auto& child : expr->children()) { + if (!child->can_evaluate_zonemap_filter()) { + if (!is_and) { + return std::nullopt; + } + continue; + } + auto child_ranges = evaluate(child); + if (!child_ranges.has_value()) { + // An unavailable AND child can be ignored, while an unavailable OR branch must + // retain the complete range so metadata pruning cannot create a false negative. + if (!is_and) { + return std::nullopt; + } + continue; + } + if (!ranges.has_value()) { + ranges = std::move(*child_ranges); + } else if (is_and) { + ranges = intersect_ranges(*ranges, *child_ranges); + } else { + ranges = union_ranges(*ranges, *child_ranges); + } + if (is_and && ranges->empty()) { + return ranges; + } + } + return ranges; + } + + std::optional> evaluate_leaf(const VExprSPtr& expr) const { + std::set slot_indexes; + expr->collect_slot_column_ids(slot_indexes); + if (slot_indexes.size() != 1) { + return std::nullopt; + } + const int slot_index = *slot_indexes.begin(); + const auto* pages = load_slot_pages(slot_index); + if (pages == nullptr) { + return std::nullopt; + } + + std::vector ranges; + for (size_t page_idx = 0; page_idx < pages->ranges.size(); ++page_idx) { + ZoneMapEvalContext ctx; + add_slot_zonemap(&ctx, slot_index, pages->data_type, pages->zone_maps[page_idx]); + if (expr->evaluate_zonemap_filter(ctx) != ZoneMapFilterResult::kNoMatch) { + append_row_range(pages->ranges[page_idx], &ranges); + } + accumulate_zonemap_stats(ctx, _pruning_stats); + } + return ranges; + } + + const SlotPageZoneMaps* load_slot_pages(int slot_index) const { + const auto cached = _slot_page_zone_maps.find(slot_index); + if (cached != _slot_page_zone_maps.end()) { + return cached->second.has_value() ? &*cached->second : nullptr; + } + const auto file_column_id = file_column_id_by_block_position(_request, slot_index); + if (!file_column_id.has_value()) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + const auto* column_schema = resolve_local_leaf_schema(_file_schema, *file_column_id); + if (column_schema == nullptr || column_schema->type == nullptr || + !native_metadata_predicate_is_type_safe(*column_schema) || + !detail::has_supported_type_defined_order(_metadata, column_schema->leaf_column_id)) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + const auto index_it = _page_indexes.find(column_schema->leaf_column_id); + if (index_it == _page_indexes.end()) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + + const auto& indexes = index_it->second; + SlotPageZoneMaps pages; + pages.data_type = column_schema->type; + pages.ranges.reserve(indexes.offset_index.page_locations.size()); + pages.zone_maps.reserve(indexes.offset_index.page_locations.size()); + for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size(); + ++page_idx) { + const auto page_range = + native_page_row_range(indexes.offset_index, page_idx, _row_group_rows); + ParquetColumnStatistics statistics; + if (!build_native_page_statistics(indexes.column_index, *column_schema, page_idx, + page_range.length, &statistics, _timezone)) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + pages.ranges.push_back(page_range); + pages.zone_maps.push_back(ParquetStatisticsUtils::MakeZoneMap(statistics)); + } + const auto inserted = _slot_page_zone_maps.emplace(slot_index, std::move(pages)); + return &*inserted.first->second; + } + + const tparquet::FileMetaData& _metadata; + const std::unordered_map& _page_indexes; + const std::vector>& _file_schema; + const format::FileScanRequest& _request; + int64_t _row_group_rows; + ParquetPruningStats* _pruning_stats; + const cctz::time_zone* _timezone; + mutable std::unordered_map> _slot_page_zone_maps; +}; + } // namespace Status select_row_group_ranges_by_native_page_index( @@ -1660,10 +1840,15 @@ Status select_row_group_ranges_by_native_page_index( } std::map conjuncts_by_slot; + VExprContextSPtrs multi_slot_conjuncts; + // Compound predicates must honor the same metadata-pruning fence as single-slot predicates. for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto slot_index = expr_zonemap::single_slot_zonemap_index(conjunct); if (slot_index >= 0) { conjuncts_by_slot[slot_index].push_back(conjunct); + } else if (conjunct != nullptr && conjunct->root() != nullptr && + conjunct->root()->can_evaluate_zonemap_filter()) { + multi_slot_conjuncts.push_back(conjunct); } } for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { @@ -1701,12 +1886,7 @@ Status select_row_group_ranges_by_native_page_index( ZoneMapFilterResult::kNoMatch) { append_row_range(page_range, &filter_ranges); } - if (pruning_stats != nullptr) { - pruning_stats->expr_zonemap_unusable_evals += ctx.stats.unusable_zonemap_eval_count; - pruning_stats->in_zonemap_point_check_count += - ctx.stats.in_zonemap_point_check_count; - pruning_stats->in_zonemap_range_only_count += ctx.stats.in_zonemap_range_only_count; - } + accumulate_zonemap_stats(ctx, pruning_stats); } if (!usable) { continue; @@ -1721,6 +1901,23 @@ Status select_row_group_ranges_by_native_page_index( } } + NativePageIndexPredicateEvaluator evaluator(metadata, page_indexes, file_schema, request, + row_group_rows, pruning_stats, timezone); + for (const auto& conjunct : multi_slot_conjuncts) { + auto conjunct_ranges = evaluator.evaluate(conjunct->root()); + if (!conjunct_ranges.has_value()) { + continue; + } + *selected_ranges = intersect_ranges(*selected_ranges, *conjunct_ranges); + if (selected_ranges->empty()) { + if (pruning_stats != nullptr) { + pruning_stats->filtered_page_rows += row_group_rows; + ++pruning_stats->filtered_row_groups_by_page_index; + } + return Status::OK(); + } + } + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto predicate = extract_variant_shredded_predicate(conjunct); if (!predicate.has_value()) { diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp b/be/test/format_v2/parquet/parquet_scan_test.cpp index cc95b51cebf751..d63970c7bbf675 100644 --- a/be/test/format_v2/parquet/parquet_scan_test.cpp +++ b/be/test/format_v2/parquet/parquet_scan_test.cpp @@ -1678,6 +1678,17 @@ void write_page_index_parquet_file(const std::string& file_path) { write_table(file_path, table, ids.size(), false, true); } +void write_multi_column_page_index_parquet_file(const std::string& file_path) { + std::vector ascending(128); + std::iota(ascending.begin(), ascending.end(), 0); + std::vector descending(ascending.rbegin(), ascending.rend()); + auto schema = arrow::schema({arrow::field("ascending", arrow::int32(), false), + arrow::field("descending", arrow::int32(), false)}); + auto table = arrow::Table::Make(schema, + {build_int32_array(ascending), build_int32_array(descending)}); + write_table(file_path, table, ascending.size(), false, true); +} + void write_multi_row_group_page_index_parquet_file(const std::string& file_path) { std::vector ids(384); std::iota(ids.begin(), ids.end(), 0); @@ -4195,6 +4206,52 @@ TEST_F(ParquetScanTest, ProfileCountersReflectPageIndexAndRangeGapPruning) { EXPECT_GT(profile.get_counter("RangeGapSkippedRows")->value(), 0); } +TEST_F(ParquetScanTest, MultiColumnOrUsesPageIndexAndResidualExpression) { + write_multi_column_page_index_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(1)).ok()); + const auto low_ascending = create_int32_function_conjunct(0, "lt", TExprOpcode::LT, 13); + const auto low_descending = create_int32_function_conjunct(1, "lt", TExprOpcode::LT, 13); + auto disjunction = create_compound_conjunct(TExprOpcode::COMPOUND_OR, low_ascending->root(), + low_descending->root()); + ASSERT_TRUE(disjunction->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(disjunction->open(&state).ok()); + request->conjuncts.push_back(disjunction); + ASSERT_TRUE(reader->open(request).ok()); + + std::vector selected; + bool eof = false; + while (!eof) { + Block block = build_file_block(schema); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + const auto& values = int32_data_column(*block.get_by_position(0).column); + for (size_t row = 0; row < rows; ++row) { + selected.push_back(values.get_element(row)); + } + } + + std::vector expected(13); + std::iota(expected.begin(), expected.end(), 0); + for (int32_t value = 115; value < 128; ++value) { + expected.push_back(value); + } + EXPECT_EQ(selected, expected); + EXPECT_GT(counter_value(profile, "FilteredRowsByPage"), 0); + EXPECT_LT(counter_value(profile, "RawRowsRead"), 128); + EXPECT_GT(counter_value(profile, "RowsFilteredByConjunct"), 0); + disjunction->close(); +} + TEST_F(ParquetScanTest, OpenDefersPageIndexProbeToCurrentRowGroup) { write_multi_row_group_page_index_parquet_file(_file_path); RuntimeProfile profile("lazy_page_index_profile"); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 533b526d613d9f..6b48b05d61cd2f 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -40,6 +40,7 @@ #include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" +#include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" #include "exprs/vliteral.h" @@ -167,6 +168,41 @@ class MetadataInt32GreaterThanExpr final : public VExpr { const std::string _expr_name = "MetadataInt32GreaterThanExpr"; }; +class MetadataSlotInt32GreaterThanExpr final : public VExpr { +public: + MetadataSlotInt32GreaterThanExpr(int slot_index, int32_t value) + : VExpr(std::make_shared(), false), + _slot_index(slot_index), + _value(value) {} + + const std::string& expr_name() const override { return _expr_name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("MetadataSlotInt32GreaterThanExpr is metadata-only"); + } + bool can_evaluate_zonemap_filter() const override { return true; } + void collect_slot_column_ids(std::set& column_ids) const override { + column_ids.insert(_slot_index); + } + ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override { + const auto zone_map = ctx.zone_map(_slot_index); + if (zone_map == nullptr) { + return unsupported_zonemap_filter(ctx); + } + if (!zone_map->has_not_null) { + return ZoneMapFilterResult::kNoMatch; + } + return zone_map->max_value <= Field::create_field(_value) + ? ZoneMapFilterResult::kNoMatch + : ZoneMapFilterResult::kMayMatch; + } + +private: + int _slot_index; + int32_t _value; + const std::string _expr_name = "MetadataSlotInt32GreaterThanExpr"; +}; + class MetadataBoundsProbeExpr final : public VExpr { public: explicit MetadataBoundsProbeExpr(bool require_false_boolean = false) @@ -466,6 +502,141 @@ TEST(NativeParquetStatisticsTest, InvalidTimeAndPaddedBooleanPageBoundsCannotPru EXPECT_EQ(bool_ranges[0].start, 0); EXPECT_EQ(bool_ranges[0].length, 1); } + +TEST(NativeParquetStatisticsTest, MultiColumnOrUnionsPageIndexRanges) { + auto encode_int32 = [](int32_t value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto make_schema = [](int local_id, int leaf_column_id) { + auto column = std::make_unique(); + column->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + column->local_id = local_id; + column->leaf_column_id = leaf_column_id; + column->type = std::make_shared(); + column->type_descriptor.doris_type = column->type; + column->type_descriptor.physical_type = tparquet::Type::INT32; + return column; + }; + auto make_page_index = [&](const std::vector& values) { + format::parquet::NativeParquetPageIndex page_index; + std::vector encoded; + encoded.reserve(values.size()); + for (const auto value : values) { + encoded.push_back(encode_int32(value)); + } + page_index.column_index.__set_min_values(encoded); + page_index.column_index.__set_max_values(encoded); + page_index.column_index.__set_null_pages(std::vector(values.size(), false)); + page_index.column_index.__set_null_counts(std::vector(values.size(), 0)); + std::vector locations; + for (size_t page_idx = 0; page_idx < values.size(); ++page_idx) { + tparquet::PageLocation location; + location.__set_offset(static_cast(page_idx * 100)); + location.__set_compressed_page_size(100); + location.__set_first_row_index(static_cast(page_idx * 10)); + locations.push_back(location); + } + page_index.offset_index.__set_page_locations(std::move(locations)); + return page_index; + }; + + std::vector> schema; + schema.push_back(make_schema(0, 0)); + schema.push_back(make_schema(1, 1)); + tparquet::ColumnOrder order; + order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder()); + tparquet::FileMetaData metadata; + metadata.__set_column_orders({order, order}); + + auto make_compound_expr = [](TExprOpcode::type opcode, VExprSPtr left, VExprSPtr right) { + TExprNode compound_node; + compound_node.__set_node_type(TExprNodeType::COMPOUND_PRED); + compound_node.__set_opcode(opcode); + compound_node.__set_type(std::make_shared()->to_thrift()); + compound_node.__set_num_children(2); + compound_node.__set_is_nullable(false); + auto compound = VCompoundPred::create_shared(compound_node); + compound->add_child(std::move(left)); + compound->add_child(std::move(right)); + return compound; + }; + auto make_compound = [&](TExprOpcode::type opcode) { + return VExprContext::create_shared(make_compound_expr( + opcode, std::make_shared(0, 50), + std::make_shared(1, 50))); + }; + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.local_positions.emplace(format::LocalColumnId(1), format::LocalIndex(1)); + request.predicate_columns = {format::LocalColumnIndex::top_level(format::LocalColumnId(0)), + format::LocalColumnIndex::top_level(format::LocalColumnId(1))}; + request.conjuncts = {make_compound(TExprOpcode::COMPOUND_OR)}; + + std::unordered_map page_indexes; + page_indexes.emplace(0, make_page_index({100, 0, 0})); + page_indexes.emplace(1, make_page_index({0, 0, 100})); + std::vector selected_ranges; + std::map skip_plans; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 2); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 10); + EXPECT_EQ(selected_ranges[1].start, 20); + EXPECT_EQ(selected_ranges[1].length, 10); + + // A compound predicate after the safety fence must not participate in metadata pruning. + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 30); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + + page_indexes.erase(1); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 30); + + request.conjuncts = {make_compound(TExprOpcode::COMPOUND_AND)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 10); + + page_indexes.emplace(1, make_page_index({0, 0, 100})); + auto first_branch = make_compound_expr( + TExprOpcode::COMPOUND_AND, std::make_shared(0, 50), + std::make_shared(1, 50)); + auto second_branch = make_compound_expr( + TExprOpcode::COMPOUND_AND, std::make_shared(0, -1), + std::make_shared(1, 50)); + request.conjuncts = {VExprContext::create_shared(make_compound_expr( + TExprOpcode::COMPOUND_OR, std::move(first_branch), std::move(second_branch)))}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 20); + EXPECT_EQ(selected_ranges[0].length, 10); +} + TEST(ParquetBloomFilterPruningTest, NativeUint32BloomUsesPhysicalInt32Hash) { const auto column_schema = uint32_parquet_bloom_schema(); format::parquet::native::BlockSplitBloomFilter bloom_filter; diff --git a/docs/file-scanner-v2-parquet-scan-design.md b/docs/file-scanner-v2-parquet-scan-design.md index b81a779ac5edcc..29dfe66c6fac63 100644 --- a/docs/file-scanner-v2-parquet-scan-design.md +++ b/docs/file-scanner-v2-parquet-scan-design.md @@ -197,9 +197,9 @@ flowchart LR does not repeatedly interpret table-schema evolution. 3. **Capability checks:** ZoneMap, Dictionary, and Bloom use only expressions they can interpret safely. All others remain row-level residual predicates. -4. **Prefer safe single-column predicates:** Single-column predicates can drive indexes and staged - filtering. Multi-column, stateful, or error-sensitive expressions retain whole-expression - evaluation. +4. **Prefer safe single-column row filters:** Single-column predicates can drive staged dictionary + or raw filtering. Multi-column AND/OR trees may still combine conservative Row Group and Page + Index candidate ranges, but the complete expression remains in whole-expression row evaluation. 5. **Runtime Filters can refresh:** ScannerScheduler refreshes late Runtime Filters before reading. TableReader handles partition-range pruning during Split preparation, and passes file-pushable parts as localized conjuncts. @@ -285,9 +285,11 @@ sequenceDiagram ### How the plan drives physical skips ColumnIndex provides min/max/null semantics for each page. OffsetIndex maps pages to Row Group row -numbers and file offsets. Candidate ranges from multiple predicate columns are intersected into -`selected_ranges`; a `page_skip_plan` is then built for each leaf so its column reader can skip pages -that do not overlap surviving rows. +numbers and file offsets. Candidate ranges follow the predicate tree: AND nodes intersect child +ranges and OR nodes union them into `selected_ranges`. A missing or unusable AND child contributes +no pruning, while a missing or unusable OR branch retains the complete Row Group range. A +`page_skip_plan` is then built for each leaf so its column reader can skip pages that do not overlap +surviving rows. > `selected_ranges` represents logical row ranges, while `page_skip_plan` represents physical page > reads. Keeping them separate allows the scheduler to advance by row batch while each column skips @@ -826,7 +828,8 @@ split safely, or read anomalies must never change query semantics. | Bloom missing, disabled, or unreadable | Skip Bloom pruning and continue with later scan stages | | Incomplete dictionary page, mixed non-dictionary encoding, complex/repeated column | Disable dictionary pruning and Dictionary-ID Filter; use actual values | | Missing or inconsistent ColumnIndex/OffsetIndex | Disable fine-grained page pruning and read the full candidate range | -| Multi-column, OR, stateful, or error-order-sensitive expression | Preserve whole-expression evaluation to avoid changing SQL short-circuit or error semantics | +| Multi-column AND/OR expression | Combine only conservative metadata candidate ranges; preserve whole-expression row evaluation | +| Stateful or error-order-sensitive expression | Preserve whole-expression evaluation without metadata decomposition | | No stable file-version identity for Page Cache | Disable Parquet Page Cache to prevent stale-byte reads | | Incomplete Condition Cache coverage | Retain and recompute uncovered ranges |