fix(rvf-sdk): atomic id-map sidecar write + fail-loud on corrupt load (#635)#697
fix(rvf-sdk): atomic id-map sidecar write + fail-loud on corrupt load (#635)#697ohdearquant wants to merge 1 commit into
Conversation
…ruvnet#635) The .idmap.json sidecar is load-bearing: delete() resolves string ids through it and silently drops unresolvable ones, so a lost/torn/corrupt sidecar causes silent data loss or corruption with no signal. Two swallows made that easy to hit: saveMappings() wrote the sidecar non-atomically and swallowed all errors. It now writes to a temp file and renames over the sidecar (so a crash/ENOSPC mid-write cannot leave partial JSON), and surfaces a failure as SidecarWriteFailed instead of discarding it — delete()/ingest correctness depends on the map, so it is not best-effort. loadMappings() degraded a corrupt sidecar to empty maps, which reset nextLabel to 1 and made subsequent ingests assign labels colliding with existing vectors (silent corruption), then overwrote the recoverable file on the next save. It now quarantines the corrupt sidecar (renames it aside) and throws SidecarCorrupt so the caller learns string-id operations are unsafe, rather than silently corrupting data. Two new SDK error codes (0xff06/0xff07); 0xff04/0xff05 left reserved. dist/ regenerated via tsc. scripts/smoke-idmap-durability.mjs verifies atomic write, state restoration on reopen, and fail-loud+quarantine on corrupt load (3 checks) against the built output. Co-Authored-By: Leo <noreply@khive.ai>
|
Reviewed this (checked out the diff, traced the actual code, and diffed against current main's 1. Needs rebase. 2. Real bug in the atomic-write logic, unrelated to the conflict. The quarantine-on-corrupt-read path looks correct as-is — verified the Happy to help rebase + fix the temp-file naming if useful — didn't want to push to your branch without asking first. |
|
@ruvnet yes please help with rebase |
Fixes #635. Sibling of #634 (same durability defect class, different package).
The
@ruvector/rvfNodeBackendpersists its string↔i64-label map as a.idmap.jsonsidecar. That sidecar is load-bearing:delete(ids)resolvesstring ids through
idToLabeland silently filters out anything unresolvable, so amissing/torn/corrupt sidecar returns
{ deleted: 0 }with no error. Two swallowsmade that failure mode both easy to hit and invisible.
Defect 1 —
saveMappings()swallowed write failures and wrote non-atomicallysave became undeletable-by-id, with zero signal.
writeFileSync) left invalid JSON on disk, whichthen triggered Defect 2 on the next open.
Fix: write to a temp file then
rename()over the sidecar (atomic publish — apartial write can never appear at the real path), and raise
SidecarWriteFailedinstead of swallowing.
delete()/ingest correctness depends on the map, sopersistence is not best-effort.
Defect 2 —
loadMappings()degraded a corrupt sidecar to empty mapsEmpty maps are worse than a hard failure:
nextLabelresets to1, so the nextingestBatchassigns labels that collidewith existing vectors — silent corruption, not just loss.
delete()silently no-ops for every historical id.saveMappings()overwrites the (recoverable) corrupt sidecar, destroyingthe evidence.
Fix: quarantine the corrupt sidecar (rename it aside so it is not clobbered) and
throw
SidecarCorrupt, so the caller learns string-id operations are unsafe. Amissing sidecar (fresh store) is still treated as legitimate and loads empty.
Compatibility
SidecarWriteFailed = 0xff06,SidecarCorrupt = 0xff07. No existing code changed (0xff04/0xff05left reservedto avoid clashing with an in-flight ergonomics PR).
state. Behavior only changes on the previously-silent failure paths.
dist/regenerated viatsc(filespublishesdist/only).Verification
scripts/smoke-idmap-durability.mjs(npm run smoke:idmap) against the builtdist/:tsc --noEmit(strict) is clean.