[zjw1111]

Purpose

Closes #445.

Add Java-compatible V2 file index containers with 64-bit payload offsets and lengths and support for streaming payload writes. Keep V1 as the default write format and automatically select the reader from the stored version.

Split the format implementations into V1/V2 writers, add FileIndexWriter::WriteTo() with a SerializedBytes() fallback, and implement direct streaming for Lumina. The data-file index writer spills to an external file when the container exceeds the inline threshold. Plugins need a streaming WriteTo() implementation to avoid buffering the complete payload.

Tests

• Add format tests for Java-generated V2 golden bytes, empty containers, streaming containers larger than 2 GiB with readable indexes beyond INT32_MAX, V1 payload buffering until Finish(), and rejection of callbacks that write bytes but report an absent index. Retain V1 golden-byte compatibility coverage.
• Add FileIndexWriter tests for the default SerializedBytes() fallback, short writes, and null or zero-length serialized payloads.
• Add OffsetOutputStream tests for relative positions, writes, flush forwarding, and keeping the underlying stream open on close.
• Add spillable output tests for the inline threshold boundary, preserving buffered bytes when spilling, and abort cleanup according to file ownership.
• Add and extend data-file index writer tests for streaming-only plugins, empty indexes, V1/V2 embedded and external round trips, dictionary-encoded columns, multiple index types on one column, failure cleanup, and including the V2 footer and trailer in the storage threshold.
• Extend option tests for the default format version, explicit V1/V2 selection, invalid values, and preserving the version when rewriting field names.
• Extend Lumina tests for streaming into a stream with an existing prefix, relative payload offsets, continued writes after WriteTo(), and empty output through both serialization interfaces.
• Parameterize WriteAndReadInteWithIndexTest across C++ V1, C++ V2, and Java fixtures, alongside ORC/Parquet and prefetch settings. C++ cases write the data and indexes before reading and randomly select embedded or external index storage.

The V2 golden bytes were generated by running the unmodified Java Paimon format sources at commit 5a6739c6c10e96052bac678eb245742ad92bc556; the 86-byte fixture matches the C++ output.

API and Format

• Add file-index.format.version=1|2 (default: 1).
• V2 stores payloads before a footer and trailer, with int64 offsets and lengths. The V1 serialized layout remains compatible.
• Widen FileIndexer::CreateReader() offsets and lengths to int64. Individual plugins retain their own payload limits; bitmap and range bitmap explicitly reject lengths above INT32_MAX.
• Add Result<bool> FileIndexWriter::WriteTo(OutputStream*) const. The stream is borrowed and reports positions relative to the payload start. False means no payload and must write no bytes; true means a non-empty payload was written. The default implementation treats null or zero-length SerializedBytes() results as no index.
• Replace container WriteColumnIndexes() with WriteIndex() and Finish(). CreateWriter() takes an explicit version argument. Downstream API implementations and callers must adapt to these interface changes.

Documentation

Describe version selection and streaming behavior in public API comments, and document each serialized layout in file_index_format_v1.h and file_index_format_v2.h.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

[lxy-9602]

+1