perf(vindex): optimize IVF index build reads

[jerry-024]

Purpose

Reduce IVF index-build wall time and raw-vector temporary-disk I/O without materially changing training-sample quality. When Parquet offset indexes make sparse sampling worthwhile, the builder reads deterministic short training ranges and streams the full scan into the index writer; otherwise it keeps the existing tempfile path.

Changes

• Plan deterministic stratified training ranges with elementary blocks of at most 128 rows and a bounded range count.
• Probe eligible vector files concurrently, require meaningful estimated I/O savings, and fall back for unsupported layouts or probe/autotuning failures.
• Read selected Parquet row groups concurrently under the shared byte budget while preserving row order and validating row IDs. This applies to predicate-free RowSelection reads, including deletion vectors and row-range splits, not only vindex.
• Pipeline full reads and add_vectors with a bounded capacity of 2, eliminating the raw-vector tempfile for eligible IVF builds.
• Preserve the full-scan/tempfile fallback and serial index upload.
• Keep diagnostics limited to key phase timing and sparse/fallback selection.

10M direct-OSS build benchmark

Environment: Intel Xeon 6982P VM (8 cores / 16 threads, 64 GiB), Rust 1.94.0, paimon-vindex-core 0.5.0, Rayon 16, row-group parallelism 8, serial upload, direct OSS VPC endpoint, Paimon local cache disabled, and local NVMe/ext4 spill disk. Main and PR runs used the same data and configuration, ran serially in alternating order, and had three trials per configuration. OS page-cache state was not controlled.

Dataset/configuration: 10M x 768 vectors in 10 Parquet files and one index shard; IVF-PQ, cosine, nlist=4096, pq.m=192, and 262,144 training rows.

Read budget	main 30d20c3 trials / median	PR d350c46 trials / median	Median change
256 MiB	358.626 / 350.421 / 348.228 s / 350.421 s	209.205 / 212.959 / 207.902 s / 209.205 s	-40.3%
768 MiB	346.440 / 349.081 / 355.451 s / 349.081 s	152.782 / 153.429 / 150.221 s / 152.782 s	-56.2%

At 256 MiB, median peak RSS changed from 3,639 MiB to 4,166 MiB. At 768 MiB, it changed from 4,272 MiB to 5,121 MiB. The PR removed 28.61 GiB of raw-vector temporary-file output; index size remained approximately 1,862.81 MiB.

The sparse training pass increased instrumented source reads from 26.53 GiB / 1,110 calls to 29.21 GiB / 3,907 calls, including the probe. The end-to-end gain therefore comes from removing the tempfile round trip and overlapping full reads with add_vectors, not from fewer source requests. The counters are at the FileRead wrapper and exclude transport-level retries and metadata operations outside that wrapper.

Recall verification

The 10M dataset used the same 100 queries and ground truth with nprobe=128:

Revision	Recall@10	Recall@100
main 30d20c3	0.9340	0.9203
PR d350c46	0.9340	0.9203

The ordered-data regression used 1M x 128 vectors, IVF-PQ with nlist=1024 and nprobe=32, and three deterministic PR sampling seeds:

Layout	main Recall@10 / Recall@100	PR median Recall@10 / Recall@100
Cluster-sorted	0.6514 / 0.52952	0.6518 / 0.53052
Shuffled	0.6501 / 0.52840	0.6508 / 0.52980

No recall regression was observed in these runs.

Fallback and reader validation

• On 100k-row full-page and near-full layouts, both revisions selected sparse=false in all three trials. The PR added only 512 KiB and one instrumented read call, so it did not perform a second full source scan. Median build times were 3.845 s to 3.739 s and 6.717 s to 6.922 s, respectively.
• On a disk-backed reader fixture with approximately 308 MiB projected row groups and 2 ms synthetic read latency, the PR kept one sparse row group in flight under a 256 MiB budget and reached two under 768 MiB. At 768 MiB, median sparse-read time changed from 0.131 s to 0.077 s. Tracked source-buffer peaks were 19.25 MiB and 38.51 MiB, and returned to zero after completion. This fixture validates concurrency and buffer lifetime, not OSS throughput.

Tests

• cargo test -p paimon --lib (2,743 passed, 0 failed)
• GitHub build, unit, integration, MSRV, Linux, macOS, and Windows checks
• Three-trial, same-budget direct-OSS comparison at 256 MiB and 768 MiB
• Recall@10 and Recall@100 on the 10M dataset and on cluster-sorted/shuffled 1M datasets with three PR seeds
• Sparse/fallback selection and reader concurrency/buffer-lifetime validation

API and format

No public API or on-disk format changes.

Documentation

No user-facing documentation changes are required.

[JingsongLi]

Found two reproducible regressions in the sparse training path: sampling can omit an entire data distribution, and the offset-index capability check can enable a second effectively full source read.

[JingsongLi]

[P2] Preserve representative training samples across the shard

These fixed gaps permanently exclude contiguous regions; the seed only shifts the remainder allocation by approximately one row. For a 100,000-row shard with the default sample ratio of 1 and 65,536 retained vectors, the last sampled row is 99,468, so a newly appended distribution in the final 500 rows is never trained.

I reproduced this using the actual planner and vindex 0.4 trainer: 90,000 vectors [0], then 9,500 [1], then 500 [100], with IVF-SQ, L2, and nlist=nprobe=1. Querying [100] returns 10/10 results from the final cluster with the previous full-stream reservoir sampling, but 0/10 with these ranges. The SQ bounds are trained only on the older distributions, so the new vectors are clamped to the old upper bound even though every vector is subsequently added and every list is searched. Please retain the existing reservoir path until representative sampling is preserved, and add a recall regression test for data clustered by append order.

[JingsongLi]

Verified at 28d6670: both original tail-distribution cases (100,000 rows with a final 500-row cluster, and 1,000,000 rows with a final 10,000-row cluster) now return 10/10 correct-cluster results. The specific fixed-gap regression described above is addressed. A separate small-sample case still fails and is documented on the current planner line here: #800 (comment)

[JingsongLi]

[P2] Check actual page savings before enabling the extra source pass

An offset index being present does not mean these ranges avoid reading pages. With the default 100,000-row shard, 65,536 training rows, 128-dimensional vectors, and default Zstd/page settings, the 64 sample ranges leave gaps too small to skip pages or survive the reader's existing 1 MiB range coalescing. A tracking FileRead measured exactly 47,788,791 data bytes for both the sample and the full read, excluding metadata; the offset-index check still returned true. This fixture fits the default writer-buffer and file-size limits.

The subsequent full scan therefore doubles source data reads relative to the previous single source scan, including remote reads when the files are on OSS. This measures bytes, not overall wall time versus the saved local spill. Please estimate selected page ranges after coalescing and fall back, or adapt the sample plan, when the sample would read essentially the whole projection.

[JingsongLi]

Rechecked at 28d6670: this remains reproducible with the current 512 short, jittered ranges. For the same default 100,000-row shard with 128-dimensional vectors, the sample and full scan still each fetch exactly 47,788,791 data bytes, excluding metadata. The latest concurrency and selected-page budget changes do not prevent the extra effectively full source pass. The sparse-path gate at writer.rs:173 still only checks offset-index availability; please include actual page savings after range coalescing, or fall back when the sample reads essentially the entire projection.

[JingsongLi]

Rechecked at 28d6670. The previous 100,000-row and 1,000,000-row tail-distribution cases now pass. Two additional reproducible issues remain in small-sample planning and sparse-read admission; the source-read amplification also remains, with updated measurements in its existing inline thread. Validation: 24 index-build tests and 53 Parquet tests passed, plus four isolated verification cases.

[JingsongLi]

[P2] Keep small training samples distributed across the shard

When training_rows <= 128, this calculation produces just one contiguous range. A realistic incremental shard with 1,000 new rows and train.sample-ratio=0.1 therefore trains on 100 adjacent rows, whereas the previous implementation sampled every tenth row across the whole shard.

Using the actual planner and vindex 0.4 trainer, I reproduced this with 450 vectors [0], 450 [1], and 100 [100], IVF-SQ, L2, and nlist=nprobe=1. For snapshot 1, bucket 0, and an empty partition, the new range is [385,484], which excludes the entire final distribution. Querying [100] returns 10/10 results from the final cluster with the baseline, but 0/10 with this plan: SQ clamps those vectors to the older upper bound and returns rows 450–459. Please retain multiple strata for small samples, or fall back to the original sampling path, and cover this incremental-shard case in a recall regression test.

[JingsongLi]

[P2] Include retained coalesced buffers in sparse read admission

scan_ranges sums the selected pages before ArrowFileReader merges byte ranges separated by at most 1 MiB. Its returned Bytes slices retain the larger coalesced allocations, but the new concurrent row-group admission uses only the smaller selected-page estimate. This leaves live source buffers out of the memory estimate used to increase parallelism.

I verified this with four valid Parquet row groups of 16,384 rows × 128 floats, default Zstd/page settings, and one selected row every 4,096 rows. With a 20,971,520-byte budget, all four groups were admitted and simultaneously retained 25,390,984 bytes of owned read buffers. Allocation ownership and release were tracked with Bytes::from_owner and Drop; these counts exclude decoded Arrow arrays, and all tracked buffers were released on completion. Please account for the coalesced allocations when calculating admission costs, or retain conservative full-column admission when they cannot be estimated.