fix: honor nested projections in the mem-wal LSM scanner

[hamersaw]

Problem

A projection naming a struct leaf is widened back to the whole struct. SELECT meta.a on a mem-wal table returns meta: Struct<a, b> instead of meta: Struct<a>.

Every projection surface in the mem-wal scanner resolved names through flat, top-level lookups — field_with_name / column_with_name against the Arrow schema — so a dotted path either missed entirely or matched only its parent. validate_projection_names rejected meta.a outright; canonical_output_schema silently dropped it.

Fix

Schema. canonical_output_schema and validate_projection_names resolve through lance_core::datatypes::Schema, which already narrows a struct to the selected leaf and merges sibling leaves of one parent into a single field (meta.a + meta.c → meta: Struct<a, c>, via do_project's candidate_field.merge()). canonical_output_schema returns Result as a consequence.

Data. The eight dataset arms (base table + SSTable across the scan, point-lookup, vector and FTS planners) now project by schema rather than by expression. That distinction is load-bearing — ProjectionPlan::from_schema documents that a partial nested projection "cannot be done easily using expressions", and expressions are what Scanner::project builds. Scanner::project_with_schema is added for it.

The six memtable execs share a new take_projected_columns, which takes whole stored columns and trims them to the projected shape with RecordBatchExt::project_by_schema — already recursive through structs, lists and maps, and null-buffer preserving. Columns whose type already matches pass through untouched, so a projection with no nested paths costs nothing.

Both arms therefore emit the same narrowed schema before UnionExec, and SSTable reads fetch only the selected sub-columns rather than whole structs.

Two inconsistencies fixed along the way

• MemTableScanner::output_schema silently dropped a column that compute_projection_indices rejected outright — the two disagreed on the same input. Both now resolve through one projected_data_fields and error alike.
• validate_projection_names used Schema::project, which errors on a missing top-level column but returns an empty struct for a missing child — so meta.nope passed validation. It now uses Schema::resolve, which checks every path segment.

Tests

• projection.rs: four new cases — leaf narrowing, sibling merging at the parent's position, whole-struct passthrough, and nested validation accepting meta.a while rejecting meta.nope.
• builder.rs: projecting_a_struct_leaf_narrows_the_memtable_output — end-to-end through MemTableScanner, asserting the sibling does not survive.
• Full mem_wal suite: 692 passed, 0 failed.

Lint status

cargo fmt --all clean. cargo clippy -p lance --lib --tests clean on every touched file. I have not run the full cargo clippy --all --tests --benches -- -D warnings that AGENTS.md asks for — the machine this was developed on is disk-bound and the full bench build was not viable. Flagging it explicitly rather than implying it passed; CI will cover it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YR3wEnDq7cD2a9v2DdTGsZ