feat(file): v2 writer/reader support columns of unequal length

[wjones127]

What

The v2 file writer advanced every column from a single global row counter, so a single file could only hold columns of equal length. Sparse data overlay files need columns whose item counts differ within one file (each field covers a different set of rows). This is technically allowed by the file format but unused today.

This PR adds that capability:

• FileWriter::write_column(column_index, array) — writes one column, advancing only that field's row counter. Called across multiple calls a field appends; a field never written ends up as a zero-length column; an empty array is a no-op. write_batch is unchanged — it still advances all fields together, so ordinary rectangular files round-trip exactly as before.
• FileReader::column_num_rows(column_index) — exposes a single column's length (sum of its pages' row counts). Always derivable from page metadata; this just surfaces it.
• Reader projection-length validation — the reader combines a projection's columns into rectangular batches, so they must share a length. A projection whose columns differ in length is now rejected up front with a descriptive error (naming each column's length) instead of panicking inside the decoder. RangeFull/RangeFrom resolve to the projected common length rather than self.num_rows (the file's longest column), so a single short column reads back at its own length. Rectangular files are unaffected (the common length equals num_rows).

Per-field lengths are computed from each top-level field's root (first) physical column, which equals its top-level row count for primitive, struct, and list fields in both v2.0 and v2.1.

Tests

lance-file lib suite (81 tests) + doctest pass; cargo fmt and cargo clippy --tests -D warnings clean. New/extended, all rstest over V2_0 + V2_1 where relevant:

• test_write_columns_unequal_lengths: writes a 5-row, a 1-row, and a 0-row column in one file; asserts per-column row counts; reads each back at its own length; random access (sorted indices) within the longer column; empty-array writes are a no-op on both written and unwritten columns.
• test_read_unequal_length_projection: equal-length projections full-scan; mismatched-length projections error before any batch (error names each column's length); single-column RangeFull/RangeFrom/RangeTo resolve to that column's own length; out-of-range bounds error.
• test_read_nested_columns_under_validation: struct and list columns (which map to multiple physical columns) still read under the new validation path.
• test_write_column_validation_errors: missing explicit schema, out-of-bounds field index, and a null written into a non-nullable field are each rejected.
• test_blocking_read_unequal_length: the blocking read path applies the same validation.
• test_write_batch_keeps_equal_lengths: rectangular files still produce equal-length columns (backwards compatible).

Acceptance criteria (OSS-1323)

• v2 writer can write a single file with columns of differing item counts (no shared global row counter)
• v2 reader reads each column back independently at its own length
• Random access by position within a shorter/longer column returns correct values
• Per-column row counts derivable from file metadata (column_num_rows)
• Round-trip tests incl. zero-length and single-row columns alongside longer ones
• Existing equal-length files round-trip unchanged

Closes #7404 (OSS-1323). Root of the Data Overlay Files stack; OSS-1322 builds on this.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.68836% with 32 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-file/src/reader.rs	92.08%	17 Missing and 5 partials ⚠️
rust/lance-file/src/writer.rs	95.61%	5 Missing and 5 partials ⚠️

📢 Thoughts on this report? Let us know!

[westonpace]

Very cool to see this coming together. Some minor nits but nothing blocking. Biggest concern is probably whether we are calling advance_columns in the write spot.

[westonpace]

I think elsewhere we'd just throw an error if the column index was out of bounds. Any particular reason to make this Option instead of Result?

[wjones127]

I'm open to it. I guess option is nice in cases where there's only one reason it wouldn't have a value, but it does mean you don't get an error message. I'll switch to an error with a clear message.

[wjones127]

Done — column_num_rows now returns a Result with an out-of-bounds message instead of Option.

[westonpace]

Used to partition a projection's flat column_indices by top-level field.

Maybe I just haven't read far enough in but I don't understand this sentence.

[wjones127]

Removed count_projected_columns — the length walk is now validate_field_length, which should read more clearly, so the confusing sentence is gone.

[westonpace]

Minor nit: I wonder if we should put these rules in a helper function

[wjones127]

Done — extracted the shape rules into a shared field_column_shape helper, now used by both from_field_ids_helper and the validation walk.

[westonpace]

Do we have any guarantees for nested fields? For example, if I have a struct with two children do both fields need to have the same length?

[wjones127]

That's a good question. I'll look into this. I think I'm inclined to say yes, otherwise it can get especially thorny with lists. But I guess in theory structs could be more flexible. Let me see what is possible and what we need to clear up.

[wjones127]

Yes, and it's now enforced on read. write_column can only target a top-level field, so a struct's children always advance together from a single array and stay equal-length — there's no way to write divergent children today. To guard against a future/foreign writer, I made the reader's length check structure-aware: a struct's children must share its row count, otherwise a clear error (the decoders would otherwise panic in v2.0 / underflow in v2.1). List/map/fixed-size-list items are exempt since their cardinality legitimately differs. Covered by test_validate_struct_child_lengths.

[westonpace]

I think this is probably a reasonable restriction. Things could get very complicated otherwise and the read/write amplification should still be pretty trivial with this requirement. I'm sure I could dream up some counter-example but let's keep it simple until we hit something real.

[westonpace]

I don't even know if we support this in general but I wonder how this handles the empty struct case. It looks like it would return an empty vec (which is probably correct?)

Not important at all, just idle musing

[wjones127]

Maybe worth a simple unit test.

[wjones127]

Added test_validate_length_list_and_empty_struct — an empty struct validates to its single column's length.

[westonpace]

I think it should be an error here and not a break? This means the user supplied a projection with an unequal number of fields / indices which is invalid.

[wjones127]

Done — a projection with the wrong number of column indices now errors instead of silently truncating.

[westonpace]

Super minor nit but seems you could do something like...

for (field_idx, array) in items) {
  let new_total = self.field_rows_written[*field_idx] + array.len() as u64;
  self.field_rows_written[*field_idx] = new_total;
  self.rows_written = self.rows_written.max(new_total);
}

...and save yourself from looping through all the fields on each call.

[wjones127]

Done — advance_columns folds each written field's new total into rows_written directly instead of rescanning.

[westonpace]

Same comment as before.

[wjones127]

Done — write_batch now grows rows_written by num_rows (every field advances uniformly).

[westonpace]

This method is called in the write_column case but not in the write_batch case (it has its own bookkeeping of self.field_rows_written and self.rows_written. It might be nice to call that out.

Alternatively, does this really need to be a dedicated method? It's only called once and similar logic is inlined in write_batch instead of a dedicated method.

[westonpace]

This says the returned tasks must be written before we advance the columns. But in write_columns we have...

        let encoding_tasks = encoding_tasks
            .into_iter()
            .flatten()
            .collect::<FuturesOrdered<_>>();

        self.advance_columns(&columns);
        self.write_pages(encoding_tasks).await?;

So it looks like we aren't doing the writing until after advance_columns?