[core] Align dynamic PK events and first-row scans with Java

[JingsongLi]

Purpose

Continue Java parity for dynamic primary-key and first-row tables after #841. Two gaps were hidden by snapshot-only or empty first-row reads:

• Cross-partition updates wrote UPDATE_BEFORE with the new partition's physical values into the old partition directory. Java DeleteExistingProcessor writes DELETE with the old partition values. Incremental Rust reads therefore exposed the wrong row kind and partition.
• Explicit first-row all-files scans packed overlapping L0 runs separately and pruned non-key values before merging. Reading versions (1, 10) and (1, 99) could return both rows, or incorrectly return (1, 99) for value = 99.

Brief change log

• Restore old partition columns before writing migration DELETEs, sharing typed partition-to-Arrow conversion with format-table reads.
• Merge first-row versions by earliest sequence, with Java's retract validation and ignore-delete handling.
• Preserve overlapping first-row versions and apply value predicates after merging when L0 is included. Ordinary first-row batch scans continue to skip L0; incremental scans preserve physical events.
• Match Java's DV precedence for L0 visibility, without changing the existing materialized DV index pruning path.

Tests

• cargo test -p paimon --lib: 2,806 passed, 2 ignored.
• first_row_scan_test, dynamic_bucket_scan_test, incremental_batch_scan_test, scan_planning_parity_test, audit_log_table_test: 76 passed.
• cargo clippy -p paimon --lib --test first_row_scan_test --test dynamic_bucket_scan_test -- -D warnings and cargo fmt --all -- --check passed.
• Built the Python wheel from this branch and ran the Python native-planning, split serialization and global-index suites: 196 passed, 276 native plans exercised.
• Regression assertions cover bucket growth, restored cross-partition indexes, deduplicate/first-row migration behavior, partition-filtered incremental events, compacted first-row reads, overlapping files, residual filters, sequence ties, retracts, null partitions, decimals and nanosecond timestamps.

API and Format

No public API or format version changes. New cross-partition retracts carry Java-compatible DELETE kinds and old-partition values. Existing files are not rewritten.

Documentation

Implementation comments describe the corrected planning and read contracts. This does not claim support for reading first-row PK-clustering files sorted by non-key columns.

[leaves12138]

Reviewed c48317e against Java's DeleteExistingProcessor, FirstRowMergeFunction, and scan/merge contracts. No blocking findings.

The migration retract now preserves the old partition's physical values and uses DELETE. First-row all-files reads keep overlapping versions together, retain the earliest sequence, validate retracts, honor ignore-delete, and evaluate value predicates after merging. Ordinary materialized reads and incremental physical-event reads retain their separate semantics.

Local validation:

• cargo test -p paimon --lib: 2,806 passed, 2 ignored.
• first_row_scan_test, dynamic_bucket_scan_test, incremental_batch_scan_test, scan_planning_parity_test, and audit_log_table_test: 76 passed.
• Additional Rust/Java interoperability probes passed: Rust reads Java-written first-row files across four snapshots with value filters, projection and limit hints; Java reads Rust-written cross-partition events and verifies the old null/decimal/nanosecond partition values, DELETE kind, partition-filtered events, and final snapshot state.
• cargo fmt --all -- --check and targeted cargo clippy with -D warnings passed.

The full GitHub CI matrix is still running. This review does not extend support to PK-clustering files sorted by non-key columns.