Conversation
…t sub column (apache#62205) ### What problem does this PR solve? Optimized the calculation of length(str_col). Treat the string column as a combination of an offset sub column and a chars sub column. Prune the string column via NestedColumnPruning so that the BE only needs to read the offset sub column, thereby saving I/O for reading the chars sub column.
… non-SlotReference string expressions (apache#62315) … ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: StringEmptyToLengthRule only rewrites `str_col = ''` to `length(str_col) = 0` when the non-literal side is a direct SlotReference. This means expressions like `element_at(struct_col, 'f3') = ''` (which becomes `struct_element(struct_col, 'f3') = ''` after analysis) are not rewritten, preventing the OFFSET-only column reading optimization from applying to struct string fields. ### Release note StringEmptyToLengthRule now rewrites any string-typed expression compared against an empty string literal, not just direct column references. This enables the OFFSET optimization for struct field access patterns like `element_at(struct_col, 'f3') = ''`.
…array<struct> (apache#62631) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#62205 Problem Summary: PR apache#62205 introduced the `length(str)` OFFSET optimization for nested column pruning. When `length()` is applied to a string-like nested field, the pruner marks the access path with an OFFSET suffix so BE can read only the offset array instead of full element data. However, the OFFSET path dedup logic operates at **slot granularity**: if ANY non-OFFSET path exists for the same slot, ALL OFFSET paths are stripped. This is incorrect for `array<struct<...>>` columns where different struct fields have independent access patterns. **Bug 1 (crash):** Given a query like: ```sql SELECT array_match_all(x -> length(struct_element(x, 'str_field')) > 0, arr), struct_element(element_at(arr, 1), 'int_field') FROM t ``` The `int_field` non-OFFSET path causes the `str_field` OFFSET path to be stripped. BE never reads `str_field` data → crash or wrong results. **Fix:** Replace slot-level OFFSET dedup with per-field bidirectional prefix matching. An OFFSET path `P+["OFFSET"]` is only stripped when a non-OFFSET path Q shares a prefix relationship with P (covering the same container or ancestor), not when Q accesses a sibling struct field. **Bug 2 (wrong pruning):** When `isStringOffsetOnly=true` AND `accessPartialChild=true` (e.g., `cardinality(arr) + arr[*].f1`), `pruneDataType()` returns the full type instead of pruning unused fields. **Fix:** Add `!accessPartialChild` guard to the `isStringOffsetOnly` check. ### Release note Fix a BE crash caused by nested column pruning incorrectly stripping OFFSET access paths for array<struct> columns when different struct fields have independent access patterns (e.g., length() on one field + direct access on another).
…dicate (apache#62304) ### What problem does this PR solve? Treat nullable fields as a combination of a nullable flag and data. When evaluating the `col IS NULL` predicate, use the NestedColumnPruning rule to prune the col field to col.NULL, thereby saving I/O on the data.
…type when its children are accessed. (apache#63229) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#62205 Problem Summary: cardinality/map_size on element_at(map, key) was collected as an OFFSET-only access path. element_at(map, key) still needs map keys for lookup, and pushing a nested *.OFFSET predicate path can make BE route OFFSET to an array item child and fail with an invalid access path. Fall back to normal element access for these expressions while preserving OFFSET-only optimization for direct array/map cardinality. ### Release note None ### Check List (For Author) - Test: Unit Test - tools/fast-compile-fe.sh - FE_UT_PARALLEL=0 ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest#testCardinalityMapElementDoesNotUseOffsetPath - FE_UT_PARALLEL=0 ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest#testStructRootMapMixedAccessKeepsKeysPath+testCardinalityMapElementDoesNotUseOffsetPath - cd fe && mvn checkstyle:check -pl fe-core -q - ./build.sh --fe - Behavior changed: No - Does this need documentation: No ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…/variant nested column pruning (apache#63736) Extend TopN lazy materialization to defer reading complex-type base columns (struct, variant, map, array) until after TopN filtering, and expand the scope to all non-trivial projection expressions. ## Core Changes **PullUpProjectExprUnderTopN** (new CustomRewriter): - Two-pass design: Collector walks the plan tree top-down to find qualifying TopNs, then walks into descendants (through Join/Filter) to find Projects with pull-able expressions. Replacer simplifies found Projects bottom-up and adds upper Projects to restore pulled-up expressions. - Eligible expressions: Alias with non-trivial child (not Slot/Literal), not referenced by TopN order keys, no NoneMovableFunction. - Excludes: CTE Producers (output mapping safety), Join/Filter conditions that reference pulled-up outputs (cleared/removed). **LazyMaterializeTopN** (simplified): - Expression pull-up moved from physical PlanPostProcessor to logical CustomRewriter, eliminating hard-coded `MERGE_SORT→Distribute→LOCAL_SORT→Project` shape walking. Now only handles MaterializeNode insertion. **OperativeColumnDerive**: - Skip PreferPushDownProject input slots from operative propagation so complex-type base columns can be lazy. **Other**: - `PhysicalLazyMaterialize`: propagate access paths to lazy output slots for nested column/subPath pruning on BE. - `MaterializationNode`/`PlanNode`: fix nested column display in EXPLAIN. - `NoneMovableFunction`: fix missing interface name. - Session variable `enable_topn_expr_pullup` for rollback. **Tests**: - `topn_expr_pullup`: 15 test cases covering struct/variant/map/array, non-PPD expressions, joins, column order preservation, negative cases. - `topn_lazy_nested_column_pruning`: 17 test cases for struct/variant nested pruning + map/array lazy mat + multi-level variant nesting. - Updated 48 shape_check .out files to reflect new plan shapes. ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rUnderTopN with chained expressions (apache#64486) ### What problem does this PR solve? this pr refactor PullUpProjectExprUnderTopN to avoid slot-not-found error. in this version, pullup is done in bottom up algorithm. it makes pullup simpler than previous version Issue Number: close #xxx Related PR: apache#63736 Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
apache#61888) ### What problem does this PR solve? Problem Summary: This PR includes two main changes: Add offset-only read optimization support for string, array, and map types in column reader ### Release note - [Storage] Add offset-only read optimization for complex types (string, array, map) to improve read performance ### Check List - Test: BE unit tests passed - Behavior changed: No (materialization fix prevents silent failures, now returns error explicitly) - Does this need documentation: No --------- Co-authored-by: englefly <englefly@gmail.com>
… lazy nested pruning suite branch-4.2 prints the merged element_at function as `element_at` (StructElement was folded into ElementAt on this branch), so the expected materialize projections use `substring(element_at(struct_col[apache#2], 'city'), 1, 2147483647)`.
Contributor
Author
|
run buildall |
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #62205, #62315, #62631, #62304, #63229, #63736, #64486 to branch-4.2 (the meta path series, applied in merge order).
Adaptations for branch-4.2:
AccessPathInfoalready lives inorg.apache.doris.analysison this branch; the newACCESS_STRING_OFFSETconstant was ported there (aliasing the existingACCESS_OFFSET).ColumnAccessPath* -> TColumnAccessPath,ColumnAccessPathType -> TAccessPathType);NestedColumnPruningkeeps branch-4.2'scomparePathSegments/variant branch alongside upstream's new logic.LazyMaterializeTopNmerges upstream's restructure while preserving branch-4.2's Lance lazy-materialization gate;LogicalJoin.getRightConditionSlot()(a master-only helper) had to be added for the portedPushDownTopNThroughJoin.StringEmptyToLengthRuleTestusesElementAt(branch-4.2 foldedStructElementintoElementAt); the regression suite expectation for the same reason.REQUIRED EXTRA COMMIT: the FE now emits offset-only / null-only nested access paths, which branch-4.2's BE could not read —
null_column_pruning.groovyaborted the BE withstd::vector<...ColumnIterator>::operator[]: Assertion '__n < this->size()' failed. This PR therefore also cherry-picks the BE-side prerequisite #61888 "Exec Support offset prue column and null column in BE" (adapted to branch-4.2'sIColumn::mutate/Defer COW shape andColumnNullable::get_null_map_column_ptr()signature). Please review that extra commit carefully.Testing on branch-4.2 (FE + BE rebuilt from this branch):
MapAccessAllWithOffsetDoesNotPropagateOffsetToKeyunit test from [Exec](be) Support offset prue column and null column in BE #61888 was kept byte-identical to upstream although it may not hold (test binary not built with MAKE_TEST=ON).