refactor: add VirtualColumns support for getting VirtualColumn dependency tree structure#19281
Open
clintropolis wants to merge 1 commit intoapache:masterfrom
Open
refactor: add VirtualColumns support for getting VirtualColumn dependency tree structure#19281clintropolis wants to merge 1 commit intoapache:masterfrom
clintropolis wants to merge 1 commit intoapache:masterfrom
Conversation
…ency tree structure changes: * add class `VirtualColumns.Node` capturing a `VirtualColumn` and its transitive `VirtualColumn` dependencies * add `VirtualColumns.getNode` method which takes a virtual column name and returns a `VirtualColumns.Node` from a memoized map supplier * modified `VirtualColumns.findEquivalent` to take a `VirtualColumns.Node` as an argument, replacing the previous two-arg `findEquivalent(VirtualColumns, VirtualColumn)`, which iterates `node.getDependencies()` directly instead of calling `virtualColumn.requiredColumns()` + `virtualColumns.getVirtualColumn()` + null-checking, which simplifies both the implementation and all call sites * removed `ShardVirtualColumnCacheEntry` from `FilterSegmentPruner`, the shard equivalence cache now uses `VirtualColumns.Node` as the key instead of allocating a new tree-structure per call * `Projections` updated to use `getNode()` + `findEquivalent(Node)` * `SegmentGenerationStageSpec` method `addRequiredVirtualColumns(VirtualColumns, VirtualColumn, Map)` replaced by `addRequiredFromNode(Node, Map)` which walks `getDependencies()` of the node rather than manually calling `requiredColumns()` + `getVirtualColumn`
github-actions
bot
added
Area - Batch Ingestion
Area - Segment Format and Ser/De
Area - MSQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #19262, makes some stuff nicer.
changes:
VirtualColumns.Nodecapturing aVirtualColumnand its transitiveVirtualColumndependenciesVirtualColumns.getNodemethod which takes a virtual column name and returns aVirtualColumns.Nodefrom a memoized map supplierVirtualColumns.findEquivalentto take aVirtualColumns.Nodeas an argument, replacing the previous two-argfindEquivalent(VirtualColumns, VirtualColumn), which iteratesnode.getDependencies()directly instead of callingvirtualColumn.requiredColumns()+virtualColumns.getVirtualColumn()+ null-checking, which simplifies both the implementation and all call sitesShardVirtualColumnCacheEntryfromFilterSegmentPruner, the shard equivalence cache now usesVirtualColumns.Nodeas the key instead of allocating a new tree-structure per callProjectionsupdated to usegetNode()+findEquivalent(Node)SegmentGenerationStageSpecmethodaddRequiredVirtualColumns(VirtualColumns, VirtualColumn, Map)replaced byaddRequiredFromNode(Node, Map)which walksgetDependencies()of the node rather than manually callingrequiredColumns()+getVirtualColumn