Skip to content

[python] Read independent temporal sources concurrently - #10868

Open
XiaoHongbo-Hope wants to merge 2 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-parallel-reads
Open

XiaoHongbo-Hope wants to merge 2 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-parallel-reads

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Temporal alignment plans each input and reads each source payload serially, adding their remote I/O latency. Add opt-in to_arrow_batch_reader(parallelism=4) to overlap independent source planning and per-batch payload reads while preserving input/output order. The default remains serial.

Source-level concurrency is bounded by the requested parallelism and source count. Each batch finishes before the next is consumed. On failure, queued futures are cancelled and workers skip source operations that have not started; running tasks are joined before propagating the error. Existing snapshot, authorization and projection paths remain in use. Per-table reader concurrency still applies, and custom catalog/auth callbacks must support concurrent calls when enabled.

Validation

  • Temporal suite: 93 passed, 1 skipped, 176 subtests passed.
  • Added checks for serial/parallel output equality, bounded source concurrency, independent planning, worker cleanup, and argument validation.
  • Added a two-worker regression with a blocked first source, a failing second source, and a third source that must not execute, even if its wrapper is dequeued before caller-side cancellation.
  • Flake8 and git diff --check passed.

@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/temporal-parallel-reads branch from fcce942 to 403f925 Compare October 30, 2026 07:29
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/temporal-parallel-reads branch from 403f925 to 0f4e330 Compare October 30, 2026 08:06
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review October 30, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant