[rust] Support subscribing to primary-key table changelog#639
Merged
luoyuxia merged 4 commits intoJun 29, 2026
Conversation
Member
Author
|
@luoyuxia PTAL 🙏 |
There was a problem hiding this comment.
Pull request overview
Adds CDC (changelog) subscription support for primary-key (KV) tables across the Rust client and language bindings by decoding a per-record ChangeType vector in non-append-only ARROW log batches, and updating scanner validation/docs/examples accordingly.
Changes:
- Decode per-record
ChangeTypebytes for non-append-only ARROW log batches and propagate them through record-mode scanning. - Allow record-mode
create_log_scanner()on primary-key tables while keeping Arrow batch scanners restricted to log tables; keep-U/+Upairs within a single poll batch. - Add end-to-end tests plus Rust/Python/C++ docs, examples, and API reference updates for KV changelog scanning.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/user-guide/rust/example/primary-key-tables.md | Adds Rust user-guide example for subscribing to PK-table CDC changelog. |
| website/docs/user-guide/rust/api-reference.md | Documents create_log_scanner CDC behavior and adds ChangeType section. |
| website/docs/user-guide/python/example/primary-key-tables.md | Adds Python user-guide example for PK-table changelog scanning. |
| website/docs/user-guide/python/api-reference.md | Documents Python scanner behavior differences for PK vs log tables. |
| website/docs/user-guide/cpp/example/primary-key-tables.md | Adds C++ user-guide example for PK-table changelog scanning. |
| website/docs/user-guide/cpp/api-reference.md | Documents C++ scanner behavior differences for PK vs log tables. |
| crates/fluss/tests/test_fluss.rs | Registers new KV changelog integration test module. |
| crates/fluss/tests/integration/kv_changelog.rs | Adds end-to-end Rust integration test for PK-table CDC changelog. |
| crates/fluss/src/record/arrow.rs | Implements changelog wire decoding (attributes bit + per-record change types) and adds unit tests. |
| crates/fluss/src/client/table/scanner.rs | Allows record-mode log scanning for PK tables; improves validation and error messages. |
| crates/fluss/src/client/table/log_fetch_buffer.rs | Ensures -U/+U update pairs are kept together within a single poll batch; adds test. |
| crates/examples/src/example_kv_changelog.rs | Adds Rust example program demonstrating PK-table CDC changelog scanning. |
| crates/examples/Cargo.toml | Registers the new Rust example target. |
| bindings/python/test/test_kv_changelog.py | Adds Python integration tests for PK-table changelog scanning and batch-scan rejection. |
| bindings/python/fluss/init.pyi | Updates Python type stubs/docs for PK-table changelog scanning. |
| bindings/python/example/kv_changelog.py | Adds Python example for PK-table CDC changelog scanning. |
| bindings/cpp/test/test_kv_changelog.cpp | Adds C++ integration tests for PK-table changelog scanning and batch-scan rejection. |
| bindings/cpp/include/fluss.hpp | Updates C++ API docs for PK-table changelog scanning and batch-scan restriction. |
| bindings/cpp/examples/kv_changelog_example.cpp | Adds C++ example for PK-table CDC changelog scanning. |
| bindings/cpp/CMakeLists.txt | Builds the new C++ example executable. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Member
Author
|
@luoyuxia Ty, addressed copilot comments, PTAL |
luoyuxia
approved these changes
Jun 29, 2026
luoyuxia
left a comment
Contributor
There was a problem hiding this comment.
@fresh-borzoni Thanks for the pr. LGTM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #380
Adds changelog (CDC) reads for primary-key tables. Non-append-only ARROW batches carry a per-record change-type vector before the Arrow IPC payload, so the decoder now reads it, and the record scanner no longer rejects PK tables. Create_log_scanner returns the changelog with per-record +I/-U/+U/-D.
The Arrow batch scanner still rejects PK tables (a record batch has no slot for a per-record op), the same as the Java client. A -U/+U pair stays within one poll, also matching Java.