chore(release): v1.2.2 - #386
Merged
Merged
Conversation
run9 reproduced the stall the previous commits were supposed to close, on a different table: episode went 13 minutes without a prune — versions climbing 63→66 while foresight and atomic_fact both collapsed to 1 — with **zero** failure, timeout, or conflict logs. Its last successful prune was logged at 11:41:59 and the staleness clock matched to the second. The deadline covered the critical section but not the await ahead of it: `table = await self._table()` sat outside `_locked`, so a hang while resolving the table handle never returned. The scheduler runs one maintenance task per kind and skips a kind whose task is still in flight, so that kind stops being maintained permanently, silently, because nothing failed. Move the handle resolution inside the deadline for all seven locked operations, and give the lock-free compaction beat its own `_deadline` (it takes no lock so it cannot block writers, but it can still park a kind by never returning). Belt and braces in the scheduler: both beats now run under `_MAINTENANCE_TASK_TIMEOUT_SECONDS`, a last-resort bound on the whole call, so any await I have not thought of costs one cadence rather than forever. Regression test: a repo whose `_table_lookup` never resolves must make prune, optimize and add all raise `VectorStoreBusyError` and leave the lock free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump to 1.2.2 and cut the changelog. This release is the storage-layer reliability work: LanceDB maintenance split into lock-free compaction and write-locked reclamation (fixing unbounded index growth), every write-lock critical section bounded by a deadline that covers acquisition, a per-kind prune-staleness signal on GET /health, `everos cascade rebuild` for a drifted or corrupt index, startup detection of column type drift, and a query-vector width check that fails fast instead of 13s deep inside LanceDB. Carries the table-handle deadline fix (previously #385) rather than shipping 1.2.2 with a known stall: the deadline covered the critical section but not the await ahead of it, so a hang while resolving a table handle parked that kind's maintenance permanently and silently. Found by a 1h high-rate soak run after #384 merged. No migration, no config change, no API change: `docs/openapi.json` differs only in the version string. The only operator-visible requirement is that `everos cascade rebuild` now refuses to run while a server holds the OME lock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kendrick-Song
approved these changes
Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps to 1.2.2 and cuts the changelog. Also carries the fix that was #385 rather than shipping this release with a known stall — that PR is closed in favour of this one.
What's in 1.2.2
The storage-layer reliability work from #379 and #384, plus one fix found after #384 merged.
The headline fix. Index maintenance bundled compaction and version cleanup into one commit, and under sustained writes that commit kept being preempted — a soak run measured 16 successes against 547 conflicts over 21 hours while the index directory grew to the disk guardrail. Compaction is now lock-free and reclamation runs under the per-table write lock, so it completes on every beat.
Everything else, in one line each:
GET /healthgrew acascadereadiness block. Alert oncascade.healthy— it flips false only on operational faults, names the affected table inreasons, and deliberately ignores the data-quality backlog (failed_permanent) so the signal does not sit red until a human edits markdown.everos cascade rebuildrecovers a drifted or corrupt index from markdown. It refuses to run while a server holds the OME lock (exit 3) — the one operator-visible behaviour change in this release; scripted callers must stop the server first.cascade rebuildinstead of "delete the index directory" (which leaves the queue marked done and the index empty).dimfails immediately withCONFIGURATION_ERRORinstead of 13–14s and an unhandled 500.cascade_lancedb_optimize_failedalert volume drops and spurious fallback index rebuilds stop.The #385 fix, and why it belongs here
After #384 merged, a 1h high-rate run reproduced the stall it was supposed to close, on a different table:
episodewent 13 minutes without a reclaim — retained versions climbing 63 → 71 — while the other two collapsed to 1, with zero failure logs.The deadline covered the critical section but not the await ahead of it (
table = await self._table()). A hang there never returns, and the scheduler skips a kind whose task is still in flight, so that table stopped being maintained permanently and silently. Handle resolution moved inside the deadline for all seven locked operations, the lock-free compaction beat got its own deadline, and the scheduler adds a last-resort 180s bound on the whole call.Worth noting the per-kind staleness alert shipping in this same release is what surfaced it — it went
healthy=falsenamingepisodeonce staleness crossed 900s. The previous signal took the newest reclaim across tables, so two healthy tables hid the dead one.Release mechanics
pyproject.toml,uv.lock,docs/openapi.json(version string only — no schema change),CHANGELOG.mdcut as[1.2.2] - 2026-08-04.everos.__version__verified at 1.2.2.Verification
make lintclean (ruff, import-linter 3/3, datetime discipline, openapi--check), 1896 unit + 182 integration passing.Nine soak / concurrency / fault-injection runs, ~120 hours total. The last three at 2.5x write rate with a second process running
cascade syncevery second and doubled fuzz:Runs 7–9 each found a real defect, all fixed in this release. Details per run are in Confluence under the LanceDB folder.
🤖 Generated with Claude Code