Repository navigation
Parquet export of struct, list and map columns (#721) - #750
Merged
Merged
Conversation
Rust narrows varbin offsets to the smallest integer type holding the data
buffer's length: u8 under 256 bytes, u16 under 64 KiB. VarBinOffsetArray
read every type but i32/u32 as 64-bit, so a small Rust-written binary
column decoded garbage offsets and failed ("varbin element bytes [...) out
of range"). Found exporting #721's mnist image struct to Parquet.
The fix sits in VarBinOffsetArray, which every varbin producer (varbin,
zstd, dict) constructs from a file-declared offsets type; a non-integer
type is now rejected there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A null struct pushes null into every field builder. The leaf builder takes it as a placeholder the struct's validity masks, but the list and struct builders rejected it when their own dtype was non-nullable, so a Parquet file with an optional struct over a required list or struct failed to import. They now append an empty list / null fields instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ParquetExporter refused every nested column, so 4 of 10 public Raincloud datasets failed `vortex export x.vortex out.parquet`. Hardwood's columnar writer already takes Arrow-shaped nested input (struct validity, list offsets), so the exporter now declares the schema recursively (one Slot adapter over Hardwood's root/struct/element builders) and writes each leaf from gathered row indices: a list's elements, a list view's ranges and a fixed-size list's slices need no copy. Parquet stores no entries under a null list or null struct, while a Vortex null may still span some, so those rows get zero entries. Struct, List, FixedSizeList and Map map to a group, LIST, LIST and MAP; a FixedSizeList re-imports as a List. An unsupported type now throws VortexException naming the column path (e.g. emb.list.element) instead of UnsupportedOperationException. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #721.
Three commits, kept separate (rebase merge):
vortex.varbinoffsets of every integer width. Rust narrows varbin offsets to the smallest integer holding the data buffer (u8 < 256 B, u16 < 64 KiB);VarBinOffsetArrayread everything but i32/u32 as 64-bit, so a small Rust-written binary column failed to read. Cross-check:VarBinNarrowOffsetsInteropIntegrationTest(fails without the fix).VortexExceptionnaming the column path. Cross-check:NestedParquetExportInteropIntegrationTest(vortex-jni writes the issue's mnist / cohere / coig shapes, Java exports, Parquet rows match).Known limitations (documented in
docs/reference.md): aFixedSizeListre-imports as aList; a present struct with every field null exports as a null struct (the reader folds struct validity into the fields); a struct/list/map whose name contains.is rejected; map columns export but don't re-import (the importer has no MAP support).Full
./mvnw verifygreen locally.🤖 Generated with Claude Code