Skip to content

[rust] Support subscribing to primary-key table changelog#639

Merged
luoyuxia merged 4 commits into
apache:mainfrom
fresh-borzoni:feat/pk-table-changelog-scan
Jun 29, 2026
Merged

[rust] Support subscribing to primary-key table changelog#639
luoyuxia merged 4 commits into
apache:mainfrom
fresh-borzoni:feat/pk-table-changelog-scan

Conversation

@fresh-borzoni

@fresh-borzoni fresh-borzoni commented Jun 24, 2026

Copy link
Copy Markdown
Member

closes #380

Adds changelog (CDC) reads for primary-key tables. Non-append-only ARROW batches carry a per-record change-type vector before the Arrow IPC payload, so the decoder now reads it, and the record scanner no longer rejects PK tables. Create_log_scanner returns the changelog with per-record +I/-U/+U/-D.

The Arrow batch scanner still rejects PK tables (a record batch has no slot for a per-record op), the same as the Java client. A -U/+U pair stays within one poll, also matching Java.

@fresh-borzoni

Copy link
Copy Markdown
Member Author

@luoyuxia PTAL 🙏

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds CDC (changelog) subscription support for primary-key (KV) tables across the Rust client and language bindings by decoding a per-record ChangeType vector in non-append-only ARROW log batches, and updating scanner validation/docs/examples accordingly.

Changes:

  • Decode per-record ChangeType bytes for non-append-only ARROW log batches and propagate them through record-mode scanning.
  • Allow record-mode create_log_scanner() on primary-key tables while keeping Arrow batch scanners restricted to log tables; keep -U/+U pairs within a single poll batch.
  • Add end-to-end tests plus Rust/Python/C++ docs, examples, and API reference updates for KV changelog scanning.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
website/docs/user-guide/rust/example/primary-key-tables.md Adds Rust user-guide example for subscribing to PK-table CDC changelog.
website/docs/user-guide/rust/api-reference.md Documents create_log_scanner CDC behavior and adds ChangeType section.
website/docs/user-guide/python/example/primary-key-tables.md Adds Python user-guide example for PK-table changelog scanning.
website/docs/user-guide/python/api-reference.md Documents Python scanner behavior differences for PK vs log tables.
website/docs/user-guide/cpp/example/primary-key-tables.md Adds C++ user-guide example for PK-table changelog scanning.
website/docs/user-guide/cpp/api-reference.md Documents C++ scanner behavior differences for PK vs log tables.
crates/fluss/tests/test_fluss.rs Registers new KV changelog integration test module.
crates/fluss/tests/integration/kv_changelog.rs Adds end-to-end Rust integration test for PK-table CDC changelog.
crates/fluss/src/record/arrow.rs Implements changelog wire decoding (attributes bit + per-record change types) and adds unit tests.
crates/fluss/src/client/table/scanner.rs Allows record-mode log scanning for PK tables; improves validation and error messages.
crates/fluss/src/client/table/log_fetch_buffer.rs Ensures -U/+U update pairs are kept together within a single poll batch; adds test.
crates/examples/src/example_kv_changelog.rs Adds Rust example program demonstrating PK-table CDC changelog scanning.
crates/examples/Cargo.toml Registers the new Rust example target.
bindings/python/test/test_kv_changelog.py Adds Python integration tests for PK-table changelog scanning and batch-scan rejection.
bindings/python/fluss/init.pyi Updates Python type stubs/docs for PK-table changelog scanning.
bindings/python/example/kv_changelog.py Adds Python example for PK-table CDC changelog scanning.
bindings/cpp/test/test_kv_changelog.cpp Adds C++ integration tests for PK-table changelog scanning and batch-scan rejection.
bindings/cpp/include/fluss.hpp Updates C++ API docs for PK-table changelog scanning and batch-scan restriction.
bindings/cpp/examples/kv_changelog_example.cpp Adds C++ example for PK-table CDC changelog scanning.
bindings/cpp/CMakeLists.txt Builds the new C++ example executable.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/fluss/src/record/arrow.rs Outdated
Comment thread crates/fluss/src/record/arrow.rs
Comment thread crates/fluss/src/record/arrow.rs
Comment thread crates/fluss/src/client/table/log_fetch_buffer.rs
@fresh-borzoni

Copy link
Copy Markdown
Member Author

@luoyuxia Ty, addressed copilot comments, PTAL

@luoyuxia luoyuxia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fresh-borzoni Thanks for the pr. LGTM

@luoyuxia luoyuxia merged commit 81e9494 into apache:main Jun 29, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support subscribe to kv table change log

3 participants