[core] Align incremental scans and version selection with Java

[JingsongLi]

Purpose

Follow-up to #839. Align incremental planning and scan.version resolution with Java so the Python native planner can preserve the same read contract.

An APPEND window containing (id=1, value=10) followed by (id=1, value=20) currently reads only the newer value through the combined plan. Java packs this window as a batch but reads both physical events, including retracts. The current Rust planner also attaches endpoint deletion vectors and can prune historical files using endpoint global indexes.

Brief change log

• Carry Java's existing isStreaming flag through DataSplit, the binary codec and Python pickle payloads. Both combined and per-commit incremental plans retain L0 and read physical events without merging versions or dropping retracts.
• Keep endpoint DVs and automatic global indexes out of incremental planning. Preserve explicit row selections and conservative PK stats/limit handling. Readers still honor DVs explicitly present in supplied splits.
• Reject manifest DELETE entries in APPEND deltas instead of reconciling them into snapshot state; preserve Java's batch/per-commit split packing distinction.
• Adapt scan.version before selector validation: tags take precedence, same-kind selectors are overwritten, and different-kind conflicts still fail.

Tests

• cargo test --locked -p paimon --all-targets --features fulltext,vortex: 3,237 passed, 2 ignored.
• Integration coverage distinguishes snapshot state from incremental events for PK versions and value filters, DE endpoint deletion vectors, signed-zero keys, and independent branch histories.
• Python binding test_read.py: 79 passed.
• End-to-end validation with PyPaimon and a local wheel built from this branch, covering versions, retracts, first-row, DV, filters/limits, DE positions and split compatibility.
• cargo clippy --locked -p paimon --all-targets -- -D warnings and cargo fmt --all -- --check.

API and Format

No Java binary format version changes: the existing streaming bit is preserved, and batch golden bytes remain unchanged. Missing is_streaming fields in older JSON payloads default to false.

Python adds Split.is_streaming(). The existing no-argument Split.serialize() supports both batch and streaming splits and preserves the Java streaming bit. Batch serialization remains unchanged.

Documentation

Updated incremental-reading and Python-binding documentation and type stubs.

[leaves12138]

Reviewed 20e7926. No blocking issues found.

Compared the event/snapshot distinction and split packing with Java IncrementalDeltaStartingScanner and SnapshotReaderImpl, and selector adaptation with TimeTravelUtil. Incremental splits retain L0 and physical versions/retracts, avoid endpoint DVs and automatic global-index pruning, and preserve the streaming flag across the binary codec and pickle. Batch behavior and explicit split DVs remain covered by regression tests. The Python serialization opt-in correctly makes the existing PyPaimon adapter fall back instead of silently discarding the event contract.

Validation against this head:

• Rust table unit tests: 1,279 passed, 1 ignored; incremental batch and scan-planning integration suites: 51 passed.
• Built this branch's Python wheel; binding test_read.py: 79 passed.
• 23 independent interoperability checks passed: Java-written first-row, DV-enabled and retract/changelog tables across 21 incremental windows, value filters and planning-limit hints, numeric-tag precedence, same-kind selector overrides, and the existing PyPaimon adapter's safe fallback. Java and Rust returned the same physical event rows for these windows.
• Rust decoded and byte-for-byte reserialized 19 actual Java streaming frames. Java decoded and read 19 Rust-produced streaming splits with matching rows.
• cargo fmt --all -- --check and cargo clippy --locked -p paimon --lib -- -D warnings passed.

GitHub CI is still partially queued/running; this is not a claim that the full CI matrix has finished.