Skip to content

[python][ray] Incrementally commit update_by_row_id file groups#8826

Draft
XiaoHongbo-Hope wants to merge 8 commits into
apache:masterfrom
XiaoHongbo-Hope:update_by_row_id_incremental_commit
Draft

[python][ray] Incrementally commit update_by_row_id file groups#8826
XiaoHongbo-Hope wants to merge 8 commits into
apache:masterfrom
XiaoHongbo-Hope:update_by_row_id_incremental_commit

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Purpose

For large update_by_row_id jobs, a single final commit makes late failures expensive because all uncommitted file groups must be rewritten.

Add an optional max_groups_per_commit to make completed windows durable and reduce retry work. The default remains a single atomic commit; incremental mode preserves cleanup and conflict detection but may retain earlier windows after a later failure.

Tests

  • RayUpdateByRowIdTest
  • TestRowIdBaseEntriesCache
  • TestCheckRowIdFromSnapshot
  • TestFileStoreCommit
  • CommitCallbackTest
  • TestRESTCommit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant