Skip to content

fix(rvf-sdk): atomic id-map sidecar write + fail-loud on corrupt load (#635)#697

Open
ohdearquant wants to merge 1 commit into
ruvnet:mainfrom
ohdearquant:fix/635-idmap-durability
Open

fix(rvf-sdk): atomic id-map sidecar write + fail-loud on corrupt load (#635)#697
ohdearquant wants to merge 1 commit into
ruvnet:mainfrom
ohdearquant:fix/635-idmap-durability

Conversation

@ohdearquant

Copy link
Copy Markdown
Contributor

PR authored by an AI agent on behalf of @ohdearquant.

Fixes #635. Sibling of #634 (same durability defect class, different package).

The @ruvector/rvf NodeBackend persists its string↔i64-label map as a
.idmap.json sidecar. That sidecar is load-bearing: delete(ids) resolves
string ids through idToLabel and silently filters out anything unresolvable, so a
missing/torn/corrupt sidecar returns { deleted: 0 } with no error. Two swallows
made that failure mode both easy to hit and invisible.

Defect 1 — saveMappings() swallowed write failures and wrote non-atomically

  • A failed write was silently discarded → every vector ingested since the last good
    save became undeletable-by-id, with zero signal.
  • A torn write (crash / ENOSPC mid-writeFileSync) left invalid JSON on disk, which
    then triggered Defect 2 on the next open.

Fix: write to a temp file then rename() over the sidecar (atomic publish — a
partial write can never appear at the real path), and raise SidecarWriteFailed
instead of swallowing. delete()/ingest correctness depends on the map, so
persistence is not best-effort.

Defect 2 — loadMappings() degraded a corrupt sidecar to empty maps

Empty maps are worse than a hard failure:

  • nextLabel resets to 1, so the next ingestBatch assigns labels that collide
    with existing vectors — silent corruption, not just loss.
  • delete() silently no-ops for every historical id.
  • The next saveMappings() overwrites the (recoverable) corrupt sidecar, destroying
    the evidence.

Fix: quarantine the corrupt sidecar (rename it aside so it is not clobbered) and
throw SidecarCorrupt, so the caller learns string-id operations are unsafe. A
missing sidecar (fresh store) is still treated as legitimate and loads empty.

Compatibility

  • Two new SDK-level error codes: SidecarWriteFailed = 0xff06,
    SidecarCorrupt = 0xff07. No existing code changed (0xff04/0xff05 left reserved
    to avoid clashing with an in-flight ergonomics PR).
  • Happy path is unchanged: normal ingests still write the sidecar and reopen restores
    state. Behavior only changes on the previously-silent failure paths.
  • dist/ regenerated via tsc (files publishes dist/ only).

Verification

scripts/smoke-idmap-durability.mjs (npm run smoke:idmap) against the built dist/:

  ok - #635 ingest persists sidecar atomically (nextLabel=3, no .tmp)
  ok - #635 reopen restores maps; no label collision on next ingest
  ok - #635 corrupt sidecar -> SidecarCorrupt + quarantined (no silent empty-map reset)

All 3 checks passed.

tsc --noEmit (strict) is clean.

…ruvnet#635)

The .idmap.json sidecar is load-bearing: delete() resolves string ids through it
and silently drops unresolvable ones, so a lost/torn/corrupt sidecar causes silent
data loss or corruption with no signal. Two swallows made that easy to hit:

saveMappings() wrote the sidecar non-atomically and swallowed all errors. It now
writes to a temp file and renames over the sidecar (so a crash/ENOSPC mid-write
cannot leave partial JSON), and surfaces a failure as SidecarWriteFailed instead of
discarding it — delete()/ingest correctness depends on the map, so it is not
best-effort.

loadMappings() degraded a corrupt sidecar to empty maps, which reset nextLabel to 1
and made subsequent ingests assign labels colliding with existing vectors (silent
corruption), then overwrote the recoverable file on the next save. It now quarantines
the corrupt sidecar (renames it aside) and throws SidecarCorrupt so the caller learns
string-id operations are unsafe, rather than silently corrupting data.

Two new SDK error codes (0xff06/0xff07); 0xff04/0xff05 left reserved. dist/ regenerated
via tsc. scripts/smoke-idmap-durability.mjs verifies atomic write, state restoration on
reopen, and fail-loud+quarantine on corrupt load (3 checks) against the built output.

Co-Authored-By: Leo <noreply@khive.ai>
@ruvnet

ruvnet commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Reviewed this (checked out the diff, traced the actual code, and diffed against current main's backend.ts/errors.ts) — one rebase blocker and one real bug found, independent of each other:

1. Needs rebase. git apply --check --3way against current main fails on src/errors.ts (and the generated dist/errors.*, dist/backend.js.map) — PR #708 (merged today) already inserted MetadataNotSupported = 0xff04 into the same enum block your diff expects to find empty. Good news: no numeric collision — SidecarWriteFailed = 0xff06 / SidecarCorrupt = 0xff07 are both still free — it's purely a textual conflict from the enum/message-map insertion point shifting. src/backend.ts itself applies cleanly (confirms #708 didn't touch saveMappings/loadMappings).

2. Real bug in the atomic-write logic, unrelated to the conflict. saveMappings()'s temp file is ${mp}.tmp — no PID or timestamp suffix. Two concurrent NodeBackend instances writing the same store path race: both writeFileSync the same temp path, then both renameSync(tmp, mp) — the second rename either silently publishes whichever process wrote last under the other's identity, or throws ENOENT because the first rename already consumed the temp file. There's no lock guarding this write (the core's LockHeld/LockStale codes don't cover this JS-side sidecar). Suggest something like ${mp}.${process.pid}.${Date.now()}.tmp.

The quarantine-on-corrupt-read path looks correct as-is — verified the SidecarCorrupt throw is unconditional even if the quarantine rename itself fails (no silent proceed-with-corrupt-file case).

Happy to help rebase + fix the temp-file naming if useful — didn't want to push to your branch without asking first.

@ohdearquant

Copy link
Copy Markdown
Contributor Author

@ruvnet yes please help with rebase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants