feat(pk-index): maintain BTree indexes during compaction

[wangyong9999]

Purpose

Closes #291

This completes the Java-compatible source-backed primary-key BTree index lifecycle for fixed-bucket primary-key tables. #194 added the read path; this PR adds the missing maintenance path:

• validate the Java-equivalent table/index prerequisites and BTree options;
• restore committed source files and primary-key source-backed payload metadata into each bucket writer, without mixing Java data-evolution (DEIX) payloads;
• read physical source rows without deletion-vector filtering, sort (value, group row id) with the existing bounded/spill-capable buffers, and build one payload per indexed field and positive data level;
• isolate synchronous build failures to one field and level, retain successful payloads, leave only failed levels uncovered for normal scan fallback, and rebuild them on a later maintenance attempt;
• reconcile missing, stale, duplicate, replaced, removed-definition, and empty-level payloads during compaction, and commit matching index ADD/DELETE entries in the same snapshot as the data changes;
• retain files referenced by current-branch live tags during snapshot expiration, reject expiration while another branch exists, and preserve the retry anchor when an external payload deletion fails; and
• document the synchronous C++ maintenance model and its current boundaries.

The implementation reuses the source metadata, BTree payload format, index manifests, commit messages, sort buffers, path factories, and reader/writer abstractions already in the repository. It adds no storage protocol, index family, background scheduler, retry policy, or manual action.

Java's asynchronous scheduling, retry/fairness policy, and manual rebuild actions are not ported. Realtime and postpone-bucket writers do not automatically build source-backed payloads. Removed-definition cleanup is emitted when a bucket is subsequently opened by write or compaction.

Tests

• Added coverage for option parsing, failed-increment state preservation, schema validation, deterministic payload construction, field/level build-failure fallback and repair, writer restore and compaction reconciliation, removed definitions, data-evolution payload separation, current-branch tag retention, branch-safe expiration, external-delete retry, orphan cleanup, and Parquet/ORC lifecycle behavior. The ORC case forces dictionary encoding and lazy decoding. Regression coverage also verifies user columns named _PK_INDEX_ROW_ID with in-memory and spill sorting, and preservation of tagged data files together with their extra files.
• GCC 8.3 Release paimon-core-test and paimon-primary-key-sorted-index-inte-test targets built with the project's -Wall -Werror configuration (local x86_64 build, reusing the patched bundled dependencies).
• paimon-core-test: 2,230/2,230 passed.
• paimon-primary-key-sorted-index-inte-test: 12/12 passed.
• pre-commit run --all-files: all hooks passed.
• Apache RAT 0.16.1: 0 unknown licenses after excluding the linked-worktree .git pointer and local build directory.
• git diff --check: passed.

API and Format

• No new supported table option or query/build API. Options::PK_CLUSTERING_OVERRIDE centralizes the existing Java-compatible key, which remains rejected by the unsupported C++ commit path.
• No new storage format or protocol. Payloads continue to use the existing BTree/global-index files, GlobalIndexMeta, primary-key source metadata v1, index manifests, and commit-message format.
• The internal restore contract explicitly requests primary-key source-backed payload scanning and excludes Java data-evolution source metadata.
• OrphanFilesCleaner supports primary-key tables and retains reachable index manifests, index payloads, and data-file extra files. It still does not enumerate global-index.external-path; snapshot expiration deletes retired external payloads by exact committed path and retains metadata for retry on failure.

Documentation

Updated docs/source/user_guide/primary_key_global_index.rst with automatic build and compaction maintenance, build-failure fallback, cleanup ownership, tag/branch expiration boundaries, spill requirements, and the synchronous/realtime/postpone limitations.

Generative AI tooling

Generated-by: Codex (GPT-5)

[wangyong9999]

cc @lxy-9602 @SteNicholas could you please take a look when convenient? Thanks~

[wangyong9999]

Merged the latest main and resolved the conflicts. cc @lxy-9602 @SteNicholas, could you please take a look when convenient? Thanks!

[wangyong9999]

Rebased onto the latest main and resolved the conflicts. cc @lszskye, could you please take a look when you have a chance? Thanks!

[lszskye]

+1

[lxy-9602]

+1