Skip to content

Parquet export of struct, list and map columns (#721) - #750

Merged
dfa1 merged 3 commits into
mainfrom
fix/721-parquet-nested-export
Nov 16, 2026
Merged

dfa1 merged 3 commits into
mainfrom
fix/721-parquet-nested-export

Conversation

@dfa1

@dfa1 dfa1 commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Closes #721.

Three commits, kept separate (rebase merge):

  1. fix(reader): read vortex.varbin offsets of every integer width. Rust narrows varbin offsets to the smallest integer holding the data buffer (u8 < 256 B, u16 < 64 KiB); VarBinOffsetArray read everything but i32/u32 as 64-bit, so a small Rust-written binary column failed to read. Cross-check: VarBinNarrowOffsetsInteropIntegrationTest (fails without the fix).
  2. fix(parquet): import a null struct holding a required list or struct. The list/struct column builders rejected the null a null parent struct pushes down; it broke the Parquet → Vortex → Parquet round trip.
  3. feat(parquet): export struct, list and map columns. Recursive schema declaration over Hardwood's builders; columnar data written from gathered row indices with struct validity and list offsets. Entries under a null list/struct are dropped (Parquet stores none). Unsupported types now throw VortexException naming the column path. Cross-check: NestedParquetExportInteropIntegrationTest (vortex-jni writes the issue's mnist / cohere / coig shapes, Java exports, Parquet rows match).

Known limitations (documented in docs/reference.md): a FixedSizeList re-imports as a List; a present struct with every field null exports as a null struct (the reader folds struct validity into the fields); a struct/list/map whose name contains . is rejected; map columns export but don't re-import (the importer has no MAP support).

Full ./mvnw verify green locally.

🤖 Generated with Claude Code

dfa1 and others added 3 commits November 16, 2026 07:38
Rust narrows varbin offsets to the smallest integer type holding the data
buffer's length: u8 under 256 bytes, u16 under 64 KiB. VarBinOffsetArray
read every type but i32/u32 as 64-bit, so a small Rust-written binary
column decoded garbage offsets and failed ("varbin element bytes [...) out
of range"). Found exporting #721's mnist image struct to Parquet.

The fix sits in VarBinOffsetArray, which every varbin producer (varbin,
zstd, dict) constructs from a file-declared offsets type; a non-integer
type is now rejected there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A null struct pushes null into every field builder. The leaf builder takes
it as a placeholder the struct's validity masks, but the list and struct
builders rejected it when their own dtype was non-nullable, so a Parquet
file with an optional struct over a required list or struct failed to
import. They now append an empty list / null fields instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ParquetExporter refused every nested column, so 4 of 10 public Raincloud
datasets failed `vortex export x.vortex out.parquet`. Hardwood's columnar
writer already takes Arrow-shaped nested input (struct validity, list
offsets), so the exporter now declares the schema recursively (one Slot
adapter over Hardwood's root/struct/element builders) and writes each leaf
from gathered row indices: a list's elements, a list view's ranges and a
fixed-size list's slices need no copy.

Parquet stores no entries under a null list or null struct, while a Vortex
null may still span some, so those rows get zero entries. Struct, List,
FixedSizeList and Map map to a group, LIST, LIST and MAP; a FixedSizeList
re-imports as a List. An unsupported type now throws VortexException naming
the column path (e.g. emb.list.element) instead of
UnsupportedOperationException.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1
dfa1 merged commit c30cbb9 into main Nov 16, 2026
6 checks passed
@dfa1
dfa1 deleted the fix/721-parquet-nested-export branch November 16, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet export fails on struct and list columns

1 participant