Skip to content

chore(release): v1.2.2 - #386

Merged
gloryfromca merged 2 commits into
mainfrom
chore/release-1.2.2
Aug 4, 2026
Merged

chore(release): v1.2.2#386
gloryfromca merged 2 commits into
mainfrom
chore/release-1.2.2

Conversation

@gloryfromca

Copy link
Copy Markdown
Collaborator

Bumps to 1.2.2 and cuts the changelog. Also carries the fix that was #385 rather than shipping this release with a known stall — that PR is closed in favour of this one.

What's in 1.2.2

The storage-layer reliability work from #379 and #384, plus one fix found after #384 merged.

The headline fix. Index maintenance bundled compaction and version cleanup into one commit, and under sustained writes that commit kept being preempted — a soak run measured 16 successes against 547 conflicts over 21 hours while the index directory grew to the disk guardrail. Compaction is now lock-free and reclamation runs under the per-table write lock, so it completes on every beat.

Everything else, in one line each:

  • Every write-lock critical section is bounded by a deadline that covers acquisition as well as the body, so no path can wait for the lock or hold it indefinitely. Budgets come from measurements (row writes are 2–25ms → 15s; rebuild → 300s; prune → 60s), because the budget doubles as how long a wedged table stays invisible.
  • GET /health grew a cascade readiness block. Alert on cascade.healthy — it flips false only on operational faults, names the affected table in reasons, and deliberately ignores the data-quality backlog (failed_permanent) so the signal does not sit red until a human edits markdown.
  • everos cascade rebuild recovers a drifted or corrupt index from markdown. It refuses to run while a server holds the OME lock (exit 3) — the one operator-visible behaviour change in this release; scripted callers must stop the server first.
  • Startup now detects column type drift, not just missing/extra columns, and points at cascade rebuild instead of "delete the index directory" (which leaves the queue marked done and the index empty).
  • A query vector whose width disagrees with the provider's declared dim fails immediately with CONFIGURATION_ERROR instead of 13–14s and an unhandled 500.
  • Benign commit conflicts no longer count as failures, so cascade_lancedb_optimize_failed alert volume drops and spurious fallback index rebuilds stop.
  • Exception logging no longer renders frame locals: one exception went from 6423 log lines to 103, and request payloads stop leaking into logs.

The #385 fix, and why it belongs here

After #384 merged, a 1h high-rate run reproduced the stall it was supposed to close, on a different table: episode went 13 minutes without a reclaim — retained versions climbing 63 → 71 — while the other two collapsed to 1, with zero failure logs.

The deadline covered the critical section but not the await ahead of it (table = await self._table()). A hang there never returns, and the scheduler skips a kind whose task is still in flight, so that table stopped being maintained permanently and silently. Handle resolution moved inside the deadline for all seven locked operations, the lock-free compaction beat got its own deadline, and the scheduler adds a last-resort 180s bound on the whole call.

Worth noting the per-kind staleness alert shipping in this same release is what surfaced it — it went healthy=false naming episode once staleness crossed 900s. The previous signal took the newest reclaim across tables, so two healthy tables hid the dead one.

Release mechanics

pyproject.toml, uv.lock, docs/openapi.json (version string only — no schema change), CHANGELOG.md cut as [1.2.2] - 2026-08-04. everos.__version__ verified at 1.2.2.

Verification

make lint clean (ruff, import-linter 3/3, datetime discipline, openapi --check), 1896 unit + 182 integration passing.

Nine soak / concurrency / fault-injection runs, ~120 hours total. The last three at 2.5x write rate with a second process running cascade sync every second and doubled fuzz:

  • disk bounded and fully reclaimed (3489MB peak → 644MB floor, then flat);
  • cross-process integrity CLEAN — no schema drift, markdown↔index counts consistent;
  • 0 crashes / 0 OOM / 0 restarts;
  • write-lock deadline warnings at 2 per 9416 writes, both auto-retried, zero permanent failures from them.

Runs 7–9 each found a real defect, all fixed in this release. Details per run are in Confluence under the LanceDB folder.

🤖 Generated with Claude Code

zhanghui and others added 2 commits August 4, 2026 20:33
run9 reproduced the stall the previous commits were supposed to close, on a
different table: episode went 13 minutes without a prune — versions climbing
63→66 while foresight and atomic_fact both collapsed to 1 — with **zero**
failure, timeout, or conflict logs. Its last successful prune was logged at
11:41:59 and the staleness clock matched to the second.

The deadline covered the critical section but not the await ahead of it:
`table = await self._table()` sat outside `_locked`, so a hang while resolving
the table handle never returned. The scheduler runs one maintenance task per
kind and skips a kind whose task is still in flight, so that kind stops being
maintained permanently, silently, because nothing failed.

Move the handle resolution inside the deadline for all seven locked operations,
and give the lock-free compaction beat its own `_deadline` (it takes no lock so
it cannot block writers, but it can still park a kind by never returning).

Belt and braces in the scheduler: both beats now run under
`_MAINTENANCE_TASK_TIMEOUT_SECONDS`, a last-resort bound on the whole call, so
any await I have not thought of costs one cadence rather than forever.

Regression test: a repo whose `_table_lookup` never resolves must make prune,
optimize and add all raise `VectorStoreBusyError` and leave the lock free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump to 1.2.2 and cut the changelog. This release is the storage-layer
reliability work: LanceDB maintenance split into lock-free compaction and
write-locked reclamation (fixing unbounded index growth), every write-lock
critical section bounded by a deadline that covers acquisition, a per-kind
prune-staleness signal on GET /health, `everos cascade rebuild` for a drifted
or corrupt index, startup detection of column type drift, and a query-vector
width check that fails fast instead of 13s deep inside LanceDB.

Carries the table-handle deadline fix (previously #385) rather than shipping
1.2.2 with a known stall: the deadline covered the critical section but not the
await ahead of it, so a hang while resolving a table handle parked that kind's
maintenance permanently and silently. Found by a 1h high-rate soak run after
#384 merged.

No migration, no config change, no API change: `docs/openapi.json` differs only
in the version string. The only operator-visible requirement is that
`everos cascade rebuild` now refuses to run while a server holds the OME lock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gloryfromca
gloryfromca merged commit 84554eb into main Aug 4, 2026
9 checks passed
@gloryfromca
gloryfromca deleted the chore/release-1.2.2 branch August 4, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants