fix(cascade): per-kind prune staleness + rebuild safety

[gloryfromca]

Follow-up to #379. An adversarial review of that PR's own review-response fixes found five real defects, all in code #379 introduced. Each was verified against the code before fixing; the negative results are noted at the end.

P1

Prune-staleness health signal was masked across kinds. _prune_staleness reported time since the newest successful prune across all kinds, so on a multi-kind deployment (every real one) a single kind whose cleanup died was hidden by the ~5 others pruning on schedule — /health stayed green while that table's index dir grew unbounded, the exact incident the signal exists to catch. Now reports the worst kind and names it in the reason. The failure streak could not cover this either (an intervening light beat resets it), so the docstrings now state that split of duties explicitly.

Heavy-beat commit conflicts fired spurious fallback rebuilds. The benign-conflict carve-out excluded the heavy beat, justified by "runs under the write lock, so it can't hit this benignly" — but that lock is in-process only, so a second process (a long cascade backfill, a cascade sync) preempts prune's Rewrite commit. Counting those as real failures let sustained cross-process churn reach the threshold and trigger a fallback rebuild, which drops every index before recreating it; a rebuild that also lost the race was swallowed as a warning, leaving the table with no FTS index (every /search on that kind 500s) until the next 12h sweep. Commit conflicts are now benign on both beats; a prune that genuinely stops succeeding is detected by prune-staleness, which doesn't destroy indexes to "fix" a lost race.

cascade rebuild had no live-daemon guard while --help and docstrings advertised it as safe — rebuild --yes against a running server dropped the tables under the daemon's cached handles, corrupting the rebuild and leaving a permanent-failure backlog. It now refuses when the OME jobstore lock is held, reusing backfill's detection and its exit code 3.

cascade rebuild died on the damage it repairs. It ran the pre-drop migration pass (ensure_business_indexes) against the broken table, so on the corruption classes it exists to fix (missing column, un-alterable type) the recovery path raised before the drop — now skipped via _runtime(ensure=False).

The nullable-vector migration error still told users to wipe the index directory — which #379's own runbook documents as the wrong recovery (queue stays done, index comes back empty). Now points at everos cascade rebuild.

P2

• rebuild resets the queue before dropping, so every crash window converges on "queue pending → re-index" instead of empty tables with a fully-done queue (a silently empty deployment); KeyboardInterrupt exits 130 with a resume hint.
• backfill's post-write prune passed a zero retention window from a separate process, able to delete files under a daemon /search still holding that version — now passes the daemon's window.
• Dropped the schema-drift error's "restart first" step: the startup migrations only alter nullability, never a name or type, so a name/type drift never self-heals.
• Runbook: documents the /health cascade block (thresholds, what flips healthy, why failed_permanent doesn't), and its quoted schema-drift error now matches the code. Pin comment no longer references a harness that isn't in this repo.

Tests

#379's three safety mechanisms were unpinned — a one-line revert of any passed the suite. Added: per-kind staleness (worst kind + named), heavy-beat benign conflict, benign-filter negative case (an error whose message merely contains retryable must still count), prune recurrence across light beats, the prune timeout releasing the write lock, timeout-below-cadence, the rebuild server guard, and a tier3 assertion that the /health cascade block is actually wired (its unit tests hand-build lifespan_data, so a rename would ship silently). Froze the last fabricated-monotonic() test.

The prune-recurrence test is mutation-verified: hoisting the attempt-clock advance out of the heavy branch (which would make prune run once per process lifetime) turns it red.

Verification

make lint clean (ruff, import-linter contracts 3/3, datetime, openapi --check), 1891 unit + 182 integration passing, zero skips introduced.

Not changed (deliberate)

Decoupling the alert thresholds / retention from the instance cadence and exposing them through CascadeConfig is a real gap (the docstrings say "tune via the constructor" but the orchestrator passes none of them) — it's a config-plumbing change, kept out of a fix PR. Same for making the /health failure-block counters nullable instead of zeroed (schema change) and rebuild's missing --root/--verbose.

🤖 Generated with Claude Code

---

Rollout / deployment notes

TL;DR — no migration, no config change, no API change. Pull, uv sync, restart.

What changes at runtime

Change	Operator-visible effect
Prune-staleness alert is now per kind (worst kind, named in reasons)	/health can now report cascade.healthy: false in a case that previously stayed green — a single kind whose version cleanup died. That is the fix working, not a new fault. The reason string names the table.
Commit conflicts are benign on both maintenance beats	cascade_lancedb_optimize_failed warnings drop under cross-process churn (they were mostly lost races), and spurious cascade_lancedb_optimize_fallback_rebuild events stop. If you alert on either event name, expect the rate to fall.
cascade rebuild refuses to run while a server is up	A scripted everos cascade rebuild --yes that used to (silently corrupt and) proceed now exits 3 with an explanation. Any automation calling it must stop the server first.
cascade rebuild resets the queue before dropping	No operator action; makes a mid-command crash recoverable (re-run, or just start the server) instead of leaving an empty index.
backfill passes the daemon's retention window to prune	Slightly less reclaimed at the instant backfill finishes; the next daemon prune (≤5 min) picks it up.
Migration / schema-drift errors now point at cascade rebuild	Recovery text no longer tells you to delete the index dir (which yields an empty index).

Upgrade steps

git pull && uv sync
# restart the server (in-memory scheduler state only; nothing persisted changes)

• No data migration. The only new state is an in-memory field on the worker's per-kind scheduler (last_prune_attempt_at); it is rebuilt on every start. Nothing is written to SQLite, LanceDB, or markdown.
• No config change. All thresholds keep their shipped defaults (prune cadence 300s, retention 60s, prune timeout 60s, alert at 3 missed cadences).
• No API/schema change. docs/openapi.json is unchanged by this PR (dump_openapi.py --check passes) — the cascade block's fields are the same; only the text inside reasons changed.
• Rollback is a plain revert: no persisted state to undo.

What to watch after deploying

1. cascade.healthy on GET /health — if it flips false with a version cleanup stalled for kind '<x>' reason, that kind's cleanup genuinely stopped (previously invisible). Check disk on that table.
2. Rate of cascade_lancedb_optimize_failed — should go down, not up.
3. Any automation invoking cascade rebuild — must now run with the server stopped (exit code 3 otherwise).

Validation on real hardware

Beyond the unit/integration suites, the pre-fix code in this line was soaked for ~110h+ (runs 1–6) including a 2h run with cross-process maintenance concurrency and malformed-input injection applied simultaneously: integrity CLEAN (no schema drift, md↔index counts consistent), disk bounded and fully reclaimed (4.36 GB peak → 205 MB floor, then flat), 0 crashes / 0 OOM. A 1h high-rate run (run7: 2.5x write rate, concurrent CLI maintenance for the
full hour, doubled fuzz) against this PR's code came back CLEAN on integrity
(no schema drift, md↔index counts consistent), 0 crashes / 0 OOM / 0 restarts,
/add and write error rates 0%.

It also found a real defect, which the last commit fixes. One table stopped
reclaiming versions permanently — 150→200 versions retained, disk 11x live size
— while the other two sat at 1 version each, and nothing was logged, because
nothing failed: the maintenance task simply never returned. The pre-fix
staleness signal (newest prune across kinds) would have hidden this completely;
the per-kind signal in this PR reported it and named the table, which is exactly
what it was changed for.

Root cause was structural: the maintenance scheduler allows one task per table,
so it skips a kind whose task is still in flight; the prune deadline sat inside
the lock covering only the cleanup call; and the other six critical sections on
that lock had no deadline at all. One operation stuck anywhere outside that
narrow window wedged the table forever. LanceRepoBase._locked(budget, op) now
bounds acquisition and body for all seven, so no path can wait for or hold
the lock indefinitely; expiry raises the retryable VectorStoreBusyError.

Verified both ways: mutation test (moving the deadline back inside the lock makes
a waiter block until the enclosing window expires — 1001ms vs 51ms) and on real
hardware (restarting into the fix collapsed the stuck table from 200 versions /
490MB to 4 versions / 192MB, /health back to healthy with staleness cycling
normally, and zero deadline-exceeded warnings — no false kills).

---

Follow-up findings from the soak runs on this branch

The 1h high-rate run (run7) and the 2h re-run (run8) turned up three more real
defects, each fixed in a later commit here.

A table could stop reclaiming versions permanently. run7: one table held 150
versions and 11x its live size on disk while the other two sat at 1 version
each — and nothing was logged, because nothing failed; the maintenance task
simply never returned. The scheduler allows one task per table (a LanceDB table
takes one writer), so it kept skipping that kind. The prune deadline sat inside
the lock covering only the cleanup call, and the other six critical sections had
no deadline at all. Fixed by bounding all seven through _locked(budget, op)
with the deadline covering acquisition. Mutation-verified (moving the deadline
back inside the lock makes a waiter block until the enclosing window expires,
1001ms vs 51ms) and confirmed on hardware: run8 ended with all three tables at
versions=1, FTS indices complete with zero unindexed tail, and lock-deadline
warnings at 2 per 9416 writes, both auto-retried.

Worth noting the per-kind staleness fix in this PR is what surfaced it: the
previous signal took the newest prune across kinds, so the two healthy tables
hid the dead one and /health stayed green.

Write-lock budgets were guessed. 120s for a row write was arbitrary, and the
budget doubles as the detection latency for a wedged table. Measured the four
operations across table and batch sizes (10k–100k rows, 50–500 rows per call):
2–25ms, worst 63ms, flat in both dimensions. Writes are now 15s, rebuild 300s.

A mismatched query vector cost 13–14s and an unhandled 500. Every slow search
in run8 was a failing search — search:vector p50 251ms / p99 1.6s, but its 8
requests over 10s were exactly its 8 failures. LanceDB only notices a width
mismatch after building the query and reports it as an opaque ValueError.
Now validated at the single point every query vector passes through.

Also capped traceback rendering: structlog's default (show_locals=True, 100
frames) rendered one exception into 6423 log lines — 85MB across 11 exceptions
in one run — at ~290ms of synchronous CPU each. Locals off and 15 frames brings
the same traceback to 103 lines / 10ms.

Run 8 verdict (2h, 15 writes/s, concurrent cascade sync every second for
the full run, doubled fuzz, 56.5k entries): integrity CLEAN, disk bounded and
reclaimed (3489MB peak → 644MB floor), 0 crashes / 0 OOM / 0 restarts. The 25
permanently-failed files match the 23 malformed markdown files fuzz injected —
that is the intended path for bad input, not a defect.

One thing I could not close: LanceError(IO): Spill has sent an error appears in
runs 6–8 and scales with the fuzz rate. It comes from the cascade write path
(merge_insert) alongside cascade_worker_unrecoverable, and the counts line up
with the injected malformed files, so it is bad input being rejected — but the
message is undiagnosable, and seven hypotheses had to be tested and discarded
before that was clear (schema drift, NaN/inf/zero query vectors, poisoned stored
vectors, spill-directory space, the index-rebuild window, and dual FTS indices —
each disproved by a reproduction attempt). Translating LanceDB write errors into
something that names the field and type would be worth doing separately.

CHANGELOG [Unreleased] was empty (#379 merged without entries), so this branch
now records both PRs' user-visible changes.