Repository navigation
[python] Batch temporal alignment metadata and native matching - #10846
Merged
JingsongLi merged 5 commits intoOct 30, 2026
Merged
Conversation
XiaoHongbo-Hope
marked this pull request as ready for review
October 28, 2026 14:01
Contributor
|
+1 Please merge this after Rust PR #1238 merged and restore the dependency of Native CI on |
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Temporal alignment currently splits upstream batches without filling the requested batch size and materializes anchor keys as Python dictionaries. Coalesce metadata up to the requested size, pass Arrow tables directly, and reuse group encodings across aligned sources. No custom metadata container is introduced.
Use the optional Rust batch matcher for int64 temporal keys. As-of result mapping and scalar floating-point interpolation remain columnar; unsupported key types and older native packages retain the existing Python path. Integer payload interpolation retains exact-ratio arithmetic. Batch defaults, snapshots, authorization and projection semantics are unchanged.
This is separate from #10843, which optimizes row-ID fetching. Native kernel: apache/paimon-rust#1238. Prototype benchmark timings are not claimed for this narrower implementation.
Validation
Native CI dependency
paimon-rust#1238 is merged. Native CI now builds paimon-rust from
main, logs the installed commit, and requires the realtemporal_matchkernel. The differential test fails rather than skips in Native CI; old-package compatibility retains its fallback.Latest local temporal validation: native path 85 passed and 155 subtests passed; compatibility path 84 passed, 1 skipped and 155 subtests passed. Includes a million-row regression check ensuring interpolation time bounds are computed once during planning.
Native CI must pass on the restored
maindependency before merging.