Skip to content

[python] Support snapshot properties in table commits - #9692

Merged
JingsongLi merged 1 commit into
apache:masterfrom
YannByron:codex/pypaimon-snapshot-properties
Sep 8, 2026
Merged

[python] Support snapshot properties in table commits#9692
JingsongLi merged 1 commit into
apache:masterfrom
YannByron:codex/pypaimon-snapshot-properties

Conversation

@YannByron

Copy link
Copy Markdown
Contributor

Summary

PyPaimon can deserialize Snapshot.properties, but its table commit API has no way to populate them. Java supports the same capability through ManifestCommittable.addProperty. This PR closes that gap so Python applications can atomically publish recovery or checkpoint state with the data represented by the snapshot, instead of maintaining an inconsistent sidecar.

The direct LeRobot capture writer in #9665 is the first consumer: it uses the properties to resume global frame, episode, and task indices without scanning the table.

Changes

  • Add an optional snapshot_properties mapping to batch and stream table commits.
  • Preserve the properties through append, overwrite, conflict retry, and snapshot creation.
  • Keep existing callers unchanged when the argument is omitted and serialize an empty mapping as no snapshot properties, matching Java behavior.

Testing

  • Batch commits persist the supplied properties in the latest snapshot.
  • Stream commits persist checkpoint properties in the latest snapshot.
  • Batch, stream, and overwrite paths forward the properties to the file-store commit layer.
  • python -m pytest -q pypaimon/tests/table_commit_test.py pypaimon/tests/table/simple_table_test.py (28 passed).
  • Ruff and git diff --check pass for all changed files.

Notes

An empty append still does not create a snapshot solely for properties. This PR adds properties to snapshots produced by data commits; it does not add metadata-only commits.

Related: #9665

Allow batch and stream commits to atomically attach application state to the generated Paimon snapshot, matching the capability available through Java ManifestCommittable.

Co-Authored-By: Codex <noreply@anthropic.com>

AI-Model: gpt-5
Co-Authored-By: Codex <noreply@ai-tool.com>
Co-Authored-By: Codex <noreply@openai.com>
AI-Contributed/Feature: 76/76
AI-Contributed/UT: 101/101

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 5b18948 into apache:master Sep 8, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants