Skip to content

[python] Batch temporal alignment metadata and native matching - #10846

Merged
JingsongLi merged 5 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-batch-native
Oct 30, 2026
Merged

JingsongLi merged 5 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-batch-native

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Temporal alignment currently splits upstream batches without filling the requested batch size and materializes anchor keys as Python dictionaries. Coalesce metadata up to the requested size, pass Arrow tables directly, and reuse group encodings across aligned sources. No custom metadata container is introduced.

Use the optional Rust batch matcher for int64 temporal keys. As-of result mapping and scalar floating-point interpolation remain columnar; unsupported key types and older native packages retain the existing Python path. Integer payload interpolation retains exact-ratio arithmetic. Batch defaults, snapshots, authorization and projection semantics are unchanged.

This is separate from #10843, which optimizes row-ID fetching. Native kernel: apache/paimon-rust#1238. Prototype benchmark timings are not claimed for this narrower implementation.

Validation

  • Compatibility path: 82 tests and 155 subtests pass; the native-only differential test requires the new extension.
  • Added checks for bounded coalescing, ordering, empty input batches, timestamp precision and old-package compatibility.
  • Full Rust extension plus integration suite: 101 passed, 155 subtests passed.
  • Full extension release build passes; all 300,000 output rows checked across 500 synthetic episodes (19 batches).
  • Flake8 with repository configuration and git diff --check pass.

Native CI dependency

paimon-rust#1238 is merged. Native CI now builds paimon-rust from main, logs the installed commit, and requires the real temporal_match kernel. The differential test fails rather than skips in Native CI; old-package compatibility retains its fallback.

Latest local temporal validation: native path 85 passed and 155 subtests passed; compatibility path 84 passed, 1 skipped and 155 subtests passed. Includes a million-row regression check ensuring interpolation time bounds are computed once during planning.

Native CI must pass on the restored main dependency before merging.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review October 28, 2026 14:01
@JingsongLi

JingsongLi commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

+1 Please merge this after Rust PR #1238 merged and restore the dependency of Native CI on main.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 1418eb5 into apache:master Oct 30, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants