feat(format): store large row id sequences in a hidden data file column

[wjones127]

Draft prototype of §5.3 of the Stable Row IDs design, "Row-ID sequences as a hidden column".

A fragment's row id sequence is run-encoded, so an appended fragment costs about 20 bytes of manifest and never needs to leave it. A fragment assembled from many places has no runs to exploit and falls back toward a byte per row per id. Inline, that cost is rewritten into every manifest version, so the manifest grows with the table and every commit rewrites all of it.

This adds a third arm to the row_id_sequence oneof:

DataFile column_row_ids = 12;

a hidden uint64 column at the reserved field id -3, one row id per physical row in offset order. It is located by the same fields/column_indices pair as a user column and read back with the ordinary data file reader, so it inherits the file format's encodings and page layout rather than introducing a new file type. The pre-existing external_row_ids arm — an opaque byte range, never written by any release — is left untouched for compatibility.

Scope

Compaction is the only write path. assign_row_ids runs in lance-table with no object store, so it cannot spill, and an appended fragment's sequence is a Range that is nowhere near the threshold. CompactionOptions::inline_row_ids_max_bytes sets the budget, defaulting to the 200 KiB inline limit the format has always documented.

The spilled file is reached through Fragment::referenced_lance_files, which is what cleanup, transaction validation, file listing, and shallow-clone base_id rewriting all already walk — so those paths pick it up without changes of their own. cleanup_keeps_a_live_spilled_file covers the failure mode with teeth: if the file were not reachable from there, an ordinary cleanup would delete a live file and leave the fragment claiming row ids it can no longer read.

Gating

FLAG_UNSTABLE_SPILLED_ROW_IDS is bit 9 (value 512), deliberately above FLAG_UNKNOWN. Every released build therefore already refuses such a dataset with no change of its own, and this build understands the bit only in debug builds or with LANCE_ENABLE_UNSTABLE_SPILLED_ROW_IDS=1.

Benchmarks

LANCE_ENABLE_UNSTABLE_SPILLED_ROW_IDS=1 cargo bench --bench rowid_spill — 8 fragments × 1,000,000 rows, in two workloads that bracket what the run encoding can do with a sequence. Within each workload the two arms differ only in inline_row_ids_max_bytes: the inline arm never spills (today's behaviour), the spilled arm takes the 200 KiB default.

deleted — 30% of rows deleted, then compacted. The deletions become holes but the surviving ids are still ascending, so each sequence encodes as a range plus a bitmap: 0.18 B/row.

metric                               inline        spilled      ratio
manifest size                        5.73M          3.82M       1.50x
  of which row ids                   0.95M          0.00M        infx
  of which row versions              1.91M          1.91M       1.00x
compaction transaction file          2.86M          1.91M       1.50x
cold dataset open                    0.82ms         0.78ms      1.05x
append commit (mean)                 5.26ms         4.06ms      1.30x
load one sequence (cold)             0.07ms         5.11ms      0.01x
row id index build (cold)            0.39ms        26.47ms      0.01x
take by row id (index built)         0.78ms         0.59ms      1.32x
compaction                         204.45ms       234.45ms      0.87x
data files on disk                  74.36M         89.12M       0.83x

shuffled — every row rewritten in random order, then compacted. No run structure survives, so the encoding degrades to U64Segment::Array, a bitpacked array of absolute values: 4.00 B/row, 22x the deleted case. This is the worst case for inline sequences, and the shape a reclustering pass would leave behind.

metric                               inline        spilled      ratio
manifest size                       30.52M          0.00M   16285.82x
  of which row ids                  30.52M          0.00M        infx
  of which row versions              0.00M          0.00M       1.00x
compaction transaction file         61.04M         30.52M       2.00x
cold dataset open                   17.17ms         0.16ms    105.11x
append commit (mean)                59.52ms         2.94ms     20.23x
load one sequence (cold)             1.61ms         2.85ms      0.56x
row id index build (cold)          180.57ms       197.59ms      0.91x
take by row id (index built)         1.18ms         0.88ms      1.34x
compaction                         299.54ms       268.08ms      1.12x
data files on disk                 135.93M        158.01M       0.86x

The shuffle is not a synthetic manifest edit: the benchmark reads every row with its stable row id, permutes them, writes the data back out, and commits an Operation::Rewrite carrying the permuted sequences on the new fragments — which is how a reclustering pass would have to preserve row ids. Both arms then compact that state. A take by row id is checked against the value the row should hold, so a spilled sequence that resolved to the wrong row would fail the benchmark rather than just look fast.

Local NVMe. The byte rows are deterministic and reproduced exactly across runs. The latency rows move between runs, so a difference under about 2x should be read as noise; every conclusion below either rests on a byte count or on a gap far wider than that.

What the two cases have in common

Inline sequences are written two or three times per commit, not once. A commit puts the fragment list in the manifest and writes the whole transaction to its own file under _transactions/. When that transaction serializes under MAX_INLINE_TRANSACTION_BYTES (20 MiB) it is also copied into the manifest. That is visible in both tables: in deleted the manifest delta is 1.91M for 0.95M of row ids, exactly 2x, because the 2.86M transaction fits under the cap; in shuffled the 61.04M transaction does not fit, so the manifest holds the sequences once and the transaction file holds them again.

The read cost, once the row id index exists, is not a per-query tax. take by row id (index built) is the control row and never favours either arm consistently (0.78 vs 0.59 ms, 1.18 vs 0.88 ms). What spilling adds is a one-time index build per dataset open.

deleted: a modest manifest win, paid for on the read side

The manifest shrinks by a third (5.73M → 3.82M) and the transaction file by the same 0.95M. But the bitmap encoding is extremely good here — 0.18 B/row against 2.76 B/row for the general-purpose column encoding — so spilling costs ~15 MiB on disk to save ~1 MiB of manifest, and the dataset-wide row id index build goes from 0.39 ms to 26.47 ms, roughly 70x. Compaction is ~13% slower. Cold open and commit latency are inside the noise band at this size, and a 5.73 MiB manifest decodes in well under a millisecond on a local filesystem either way.

On this workload the design is a bad trade on raw bytes, and pays off only because the manifest is rewritten on every commit while the data file is written once.

shuffled: the case the design is actually for

Every cost above either inverts or disappears once the inline encoding has nothing to exploit:

• The manifest stops carrying row ids at all: 30.52M → effectively zero, and the transaction file halves.
• Cold open drops from 17.17 ms to 0.16 ms, a 105x gap — two orders of magnitude, far outside the noise band, and the same effect the deleted case was too small to show.
• A small append commits 20x faster, 59.52 ms → 2.94 ms, because the writer no longer rewrites 30 MiB of row ids to add ten rows.
• The index-build objection evaporates: 180.57 ms against 197.59 ms, a 9% difference. Decoding 30 MiB of protobuf arrays costs about what reading the same ids out of a column does, so the 70x penalty in deleted is an artifact of the bitmap encoding being nearly free, not a property of spilling.
• Compaction is 12% faster spilled, because writing 30 MiB into the manifest and transaction file costs more than writing the column.
• Bytes on disk favour the column: 2.89 B/row spilled against 4.00 B/row inline. The trade reverses — the same file-format encoding that lost by 15x on sorted-with-holes ids wins on shuffled ones.

Two caveats on the shuffled table. Its row-version rows are zero in both arms because the benchmark's rewrite leaves version metadata unset and the compaction has none to carry forward, so that manifest is row-ids-only; with version tracking active the spilled arm's manifest would not be empty. And the 30.52M transaction file in the spilled arm is the old fragments' inline sequences, which the setup rewrite created — the new fragments contribute only file references.

Known gaps

• The row version sequences are untouched. §5.3 also covers _row_created_at_version (-4) and _row_last_updated_at_version (-5). In the deleted case those are 1.91 MiB, larger than the 0.95 MiB of row ids this change moves, so on that workload spilling row ids alone removes only part of the manifest bloat.
• inline_row_ids_max_bytes is on CompactionOptions. It belongs in table config at GA; a per-call compaction option is the smallest thing that let the two arms be measured.
• Row version tracking degrades on a spilled fragment. Three synchronous paths in lance-table need the sequence and cannot do IO to get it: refresh_row_latest_update_meta_for_partial_frag_rewrite_cols had a todo!() there and now returns Error::NotSupported rather than panicking; resolve_update_version_metadata logs a warning and uses default versions; refresh_row_latest_update_meta_for_full_frag_rewrite_cols treats the fragment as empty and writes no metadata at all. Only the last is silent, and all three want the read to become async.
• The whole sequence is materialized on read. The column is written and read a page at a time, but building a RowIdSequence needs every id, so the read path allocates one u64 per row.
• A dedicated file per spilled sequence. Co-locating _rowid in the fragment's main data file would remove one file open per fragment from the read path.
• A delta-aware encoding for the _rowid column is not attempted. It would narrow the 2.76 B/row the column costs on sorted-with-holes sequences, which is what makes the deleted case a loss on disk.
• Adding a field to CompactionOptions pushed a future in dataset_schema_evolution.rs past the workspace's large_futures deny, so two call sites there are now Box::pin-ed. Included because it is a direct consequence, not a drive-by.

Refs #8931