feat(merge): legacy promotion path + body-col schema evolution

[g-talbot]

Stacked on #6410. Two adversarial-review follow-ups grouped because they share the streaming engine's input-routing and union-schema seams.

(b) Legacy-prefix promotion

A new operation type pairs a rg_partition_prefix_len = 0 split with prefix_len > 0 peers in one merge, so legacy splits can be folded into prefix-aligned buckets instead of aging out via retention. Adds:

• ParquetMergeOperation::promote_legacy(splits, target_prefix_len): relaxes MP-3 to allow mixed rg_partition_prefix_len as long as every input is <= target. Sort_fields + window equality unchanged.
• ParquetMergeOperation::target_prefix_len_override: Option<u32> field records the promotion target; None is the default regular-merge form.
• merge_parquet_split_metadata(..., mixed_prefix_ok): skips the input-side prefix-len equality check in promotion mode. The output prefix_len still comes from the writer's KV stamp via MergeOutputFile.output_rg_partition_prefix_len (CS-1 holds by construction post-F1).
• merge::execute_merge_operation(op, sources, ...): new thin executor that opens each input as either LegacyInputAdapter (when split.rg_partition_prefix_len < target) or StreamingParquetReader (otherwise), then feeds them to the streaming engine. Becomes the seam PR-7 will wire from above.

F6 + F13: Body-col schema evolution

The merger now supports MC-4 across heterogeneous body-col schemas.

• F6: normalize_type collapses Binary/LargeBinary (and dict variants) to Binary, analogous to the existing string-flavour collapse. Two inputs whose body col differs only by byte-array flavour merge cleanly; before this they hit a "type conflict" at alignment time.
• F13: streaming_writer.rs::write_list_via_serialized_column_writer (renamed from ..._non_nullable_...) now handles nullable outer List<T> / LargeList<T>. MC-4 forces the union to be nullable when a List col is present in only some inputs; before this the writer rejected the merged output. Uses Dremel max_def_level = 2 (0 = outer null, 1 = empty list, 2 = element present) for nullable outer; non-nullable path unchanged.

Test plan

• cargo nextest run -p quickwit-parquet-engine --all-features — 493 unit tests pass locally.
• cargo clippy -p quickwit-parquet-engine --tests --all-features with -Dwarnings.
• cargo doc --no-deps -p quickwit-parquet-engine warning-free.
• cargo fmt --all -- --check (nightly via PATH override).
• New tests covering the new paths:
  • Planner: test_promote_legacy_pairs_legacy_with_aligned_peer, test_promote_legacy_rejects_higher_prefix_input, test_promote_legacy_still_enforces_sort_fields, test_promote_legacy_all_at_target_is_valid.
  • Aggregator: test_mixed_prefix_ok_skips_input_equality_check.
  • Executor end-to-end: test_promote_legacy_executor_end_to_end (legacy single-RG + aligned multi-RG → 3-RG aligned output passing assert_unique_rg_prefix_keys + CS-1).
  • Executor negative: test_executor_mismatched_sources_count_bails.
  • Schema-evo: test_mc2_mixed_schemas_round_trip covers Utf8↔Dict, LargeBinary↔Binary, List<Float64> in A only, A-only Int32, B-only Int64, plus a common Float64 control col.

Out of scope / follow-up

• Cross-file SS-3 ordering: the F12 missing-col constant uses [0x00, 0x00] which sorts before non-null values. Fine for single-file SS-3 (constant across all RGs); needs a 0xff-prefixed sentinel if cross-file mixed-schema SS-3 becomes a production scenario.
• Wiring execute_merge_operation into the compaction executor (PR-7): the seam is now in place, the actual planner integration that emits promote_legacy operations is a separate change.

🤖 Generated with Claude Code