Repository navigation
feat(format): support Vortex file format - #376
Merged
Merged
Conversation
zhouyc-ali
force-pushed
the
feat/vortex-file-format
branch
3 times, most recently
from
September 22, 2026 15:09
36b8af1 to
0d74468
Compare
lxy-9602
reviewed
Sep 23, 2026
SteNicholas
reviewed
Sep 26, 2026
lxy-9602
reviewed
Sep 29, 2026
Pin the vendored Vortex from commit d82de0e to tag 0.75.0 so paimon-cpp interoperates bidirectionally with paimon-java's vortex-jni 0.73.0: 0.75 reads 0.73-written files and, verified end-to-end, 0.73 reads 0.75-written files. Newer pins (>=0.77) break the cpp->java direction with "Encoding not found in registry". Adapt the FFI call sites to the 0.75 signatures (error message accessor, the arrow/dtype converters drop their session argument, VortexFile::data_source) and fix a latent ownership bug uncovered by the downgrade: at 0.75 vx_data_source_dtype returns a BORROWED pointer (arc_wrapper new_ref, no refcount bump), so it must not be released with vx_dtype_free. Freeing it spuriously decremented the data source's DType Arc, a use-after-free that surfaced later as a segfault in the scan path. The pointer is now treated as borrowed; it stays valid for the schema conversion, which the data source outlives.
…Array NormalizeViewArray only recursed through STRING_VIEW/BINARY_VIEW/STRUCT/LIST, so a FIXED_SIZE_LIST (or LARGE_LIST) with view-typed elements fell into the default branch and was returned unchanged. NormalizeViewType already normalizes those element types, so the read schema declared e.g. fixed_size_list<utf8> while the data stayed fixed_size_list<utf8view> (Vortex hardcodes Utf8 -> Utf8View on export). The ArrowSchema mismatch made callers reinterpret the view buffers as standard strings, silently corrupting data on read. Add LARGE_LIST and FIXED_SIZE_LIST branches symmetric to the LIST branch, including the unchanged-values fast path, reusing the source null bitmap, null count, offset, length and (for LARGE_LIST) value offsets. Covered by WriteThenReadListOfString, WriteThenReadFixedSizeListOfString and WriteThenReadListOfStruct, whose data includes nulls, empty strings and strings long enough to exceed the 12-byte inline view threshold.
Address four review findings plus a related write-path abort: - Scan in storage order. The scan used Vortex's default ordered=false, which may emit chunks out of order via buffer_unordered; NextBatch assigns physical row positions by batch order, so unordered chunks would misalign deletion vectors and primary-key merge. Set ScanOptions.ordered(true), matching paimon-java's VortexRecordsReader. - Recurse into nested fields when projecting to the read schema. ProjectToReadSchema only selected top-level columns, so reading a pruned nested field (e.g. r.b from r:ROW<a,b>) kept the full child under a pruned parent type, mismatching the exported Arrow schema and silently corrupting the output. Project struct/list/fixed-size-list children recursively. - Fix a use-after-free on the stream error path. get_last_error returns a pointer owned by the ArrowArrayStream; it was read after ReleaseStream() invalidated it. Copy the message before releasing. - Rebase sliced batches to offset 0 before vx_array_from_arrow. arrow-rs cannot import a sliced (offset > 0) top-level struct (the parent offset is re-applied to already-offset children, tripping an arrow-data slice assertion that aborts). Offset-0 batches pass through untouched, so the hot path is unchanged. - Attribute the adapted sink entry points in callback_io.rs to vortex-ffi's sink.rs (Apache-2.0, Copyright the Vortex contributors) per Apache-2.0 section 4. Adds regression tests: nested-struct projection, row order across batches, and sliced top-level struct write/read.
…x TSAN races The Debug/TSAN matrix jobs failed the check-clang-tidy gate and the TSAN race gate on the Vortex files: - clang-tidy (modernize-use-using / modernize-use-auto): convert the hand-written C-ABI typedefs in vortex_ffi.h to `using` aliases / plain structs, and use `auto` for the cast-initialized locals in vortex_io_callbacks.cpp and vortex_stats_extractor.cpp. - TSAN: Vortex deserializes its footer/dtype and drives segment reads from internal runtime threads inside the un-instrumented Rust staticlib, so TSAN cannot see the happens-before edges and reports false races whose only non-paimon frames are the vortex_array/vortex_error/vortex_file/vortex_layout/vortex_mask Rust modules. Suppress those module-scoped patterns (they match Rust frames only, so races confined to paimon C++ are still reported), following the existing vortex_buffer/vortex_io/CallbackReadAt precedent.
…validation, CI deps) - Serialize host OutputStream access between the Vortex writer task and a caller-thread Flush(): VortexOutputContext gains a stream mutex and a FlushStream() entry, and VortexFormatWriter::Flush() plus the write/flush callbacks all take it, so Write and Flush can no longer interleave on a stream with no concurrent-access contract. The remaining internally buffered data is drained when the sink closes in Finish(). - Thread the caller's Arrow memory pool into VortexFormatWriter (Create now takes it and the builder passes GetArrowPool(pool_)), so sliced-batch rebasing allocates from the Paimon pool instead of Arrow's default pool. - Validate Vortex-incompatible schema types wherever Vortex can be selected (default file format, file.format-per-level, changelog-file.format), mirroring the Lance validation, instead of only the default format. - setup_flatc.sh: install libclang before the flatc early-exit, so an environment that already has a matching flatc but lacks libclang no longer silently skips the bindgen build dependency. - Tests: FlushBetweenBatchesThenRead (flush interleaved with background writes) and SchemaValidationTest.TestVortexDataTypes (per-level/changelog validation).
…Pool fwd decl The Debug build's check-clang-tidy gate analyses every changed translation unit. Any TU that includes paimon/type_fwd.h and also uses arrow::MemoryPool (as the Vortex writer now does for its rebasing pool) trips bugprone-forward-declaration-namespace on the paimon::MemoryPool forward declaration, even though paimon::MemoryPool is a real public type defined in paimon/memory/. Annotate the forward declaration with NOLINT plus a comment explaining that it is intentional.
Report one unknown statistic per schema field, validate the schema at the writer boundary, close the input stream with its last owner, align scanned batches to the read schema, make Flush() a no-op and shrink the write channel.
zhouyc-ali
force-pushed
the
feat/vortex-file-format
branch
from
October 10, 2026 07:38
1c814ff to
64d5ed4
Compare
lxy-9602
reviewed
Oct 13, 2026
SteNicholas
reviewed
Oct 13, 2026
Member
There was a problem hiding this comment.
@zhouyc-ali, thanks for updates. I have left some comments from reviewing 64d5ed4: write-callback shutdown ordering, TIME statistics extraction, and TIMESTAMP_LTZ reads across timezones.
…stamp zones Destroy the format writer before closing the stream in SingleFileWriter::Abort, and reject writes on a writer that was already aborted, so a format writer with a background task quiesces while its stream is still usable. The Vortex writer marks its callbacks aborted, drops the sink and waits for the in-flight one. Report an unknown INT statistic for TIME columns, and retype timezone-bearing timestamps to the read schema's zone before alignment.
cpplint's runtime/int check rejects the C type long long in the PAIMON_LOG_WARN argument, which failed the pre-commit job.
SteNicholas
reviewed
Oct 15, 2026
Member
There was a problem hiding this comment.
@zhouyc-ali, thanks for updates. I left two correctness comments in the Vortex writer. PTAL.
gcc rejects binding a const std::string& loop variable to an initializer_list of const char* under -Werror=range-loop-construct.
arrow-rs rejects sliced arrays at any nesting level with an aborting assertion, and a sliced child hides inside an offset-zero batch, so normalize offsets unconditionally before handing the batch to Vortex. Bind the writer's Arrow pool to the exported batch so buffers rebased from it outlive the writer.
SteNicholas
approved these changes
Oct 15, 2026
SteNicholas
left a comment
SteNicholas
left a comment
Member
There was a problem hiding this comment.
@zhouyc-ali, thanks for updates. LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #319
Adds optional read and write support for the Apache Paimon Vortex file format, integrated through
the upstream Vortex Rust FFI and gated by
PAIMON_ENABLE_VORTEX.conversion. Vortex exports strings and binaries as Arrow view types; the reader normalizes them
back to the standard types declared in the Paimon schema, recursing through structs, lists and
fixed-size lists.
vortex-jni 0.73.0 (each implementation reads the other's files, verified end-to-end). Newer Vortex
(>= 0.77) emits encodings that vortex-jni 0.73.0 cannot read, breaking the C++ -> Java direction.
InputStream/OutputStream(works with anyFileSystem: local, OSS, S3, Jindo), instead ofbuffering whole files in memory on read or staging a local temporary file on write. The stock
Vortex C API cannot route IO through paimon's filesystem, so the bridge is added to
vortex-ffias an injected Rust module (
crates/vortex_callback_io/) plus a one-line patch(
cmake_modules/vortex.diff).to the beginning when
SetReadSchemais called.FileInfo; Vortex exposes no per-column statistics to Paimon(consistent with the Java implementation).
PrefetchFileBatchReader(Vortex performs random-access reads through its own callbacks).
ordered=true, matching paimon-java) so the physical row positions theread path assigns by batch order stay aligned for deletion vectors and primary-key merge; rebase
sliced batches to offset 0 before handing them to Vortex, and copy the Arrow stream error message
before releasing the stream.
-DPAIMON_ENABLE_VORTEX=ONtobuild_paimon.sh, install thestableRust toolchain (Vortex MSRV 1.95), and build
flatc25.12.19 (ci/scripts/setup_flatc.sh), whichVortex needs to compile its FlatBuffers schemas at build time.
Predicate/projection pushdown into Vortex is not implemented; all rows and columns are scanned and
projected to the read schema.
Tests
PAIMON_ENABLE_VORTEX=ONpaimon-vortex-format-test): 12/12 passed, including round-trips forlist<utf8>,fixed_size_list<utf8>andlist<struct>(view normalization), nested-structprojection (
r.bfromr:ROW<a,b>), row order across batches (ordered scan), and slicedtop-level struct write/read
cargo test -p vortex-ffi callback_io: 3/3 passedTestVortexJavaCompatibility)git diff --checkclean; cpplint/cmake-format/rustfmt/pre-commit and Apache RATare enforced by CI
API and Format
Adds the optional Vortex file-format factory and the
PAIMON_ENABLE_VORTEXCMake option. It readsand writes the existing Apache Paimon Vortex file format through the upstream Vortex Rust FFI (pinned
to 0.75.0) and does not change the Vortex binary format or any existing public API under
include/.Building Vortex requires a Rust toolchain (>= 1.95) and
flatc25.12.19; both are set up in CI.Documentation
Updated
docs/source/user_guide/format_table.rstto list Vortex among the formats a format tabledoes not reach, and added
crates/vortex_callback_io/README.mddescribing the injected module andits build wiring.
Generative AI tooling
Generated-by: Qoder