Repository navigation
[Feature] Support Lance file format #313
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Apache Paimon Java supports Lance as a data file format through the paimon-lance module. Paimon C++ currently cannot read or write tables configured with file.format=lance, which prevents Java and C++ engines from interoperating on those tables.
Solution
Add optional Lance file format support to Paimon C++, aligned with the Java implementation and Lance 0.39.0:
- register a Lance file format factory behind a disabled-by-default CMake option;
- support reading and writing Lance files through the existing file format interfaces;
- support column projection, precise row selection, row counts, and file row-id reporting;
- validate the same supported and unsupported logical types as Java;
- add C++ round-trip tests and Java/C++ compatibility fixtures in both directions.
The implementation should follow the existing Mosaic Rust FFI integration and lifecycle patterns.
Anything else?
Java reference module: https://github.com/apache/paimon/tree/master/paimon-lance
Are you willing to submit a PR?
- I am willing to submit a PR!
Activity
wangyong9999 commentedon Sep 10, 2026
Validation is complete on draft PR #315. The Java 0.39 compatibility fixtures, compaction/data-evolution/deletion-vector coverage, and fork CI are green. Single-node HDFS/FUSE point-lookup tests also completed with zero lookup errors. Under each format’s native defaults, Lance improved single-key throughput by 11.4%–25.8% across the tested concurrency range and improved CPU-saturated 4-key throughput by 13.9%–16.4%. A two-way page-layout audit matched the dominant payload pages at both Parquet’s native 4,096 rows (~4.2 MB) and Lance’s native 8,192 rows (~8.4 MB). The stricter 4,096-row comparison still showed +5.8% to +10.1% Lance throughput, with P99 from -19.0% to +1.0%. Full methodology, both sensitivity bounds, results, and limitations: #315 (comment)