feat: cancel an upload and collect orphaned staged blocks - #950
Conversation
WalkthroughThe engine adds pre-publication upload cancellation, cancellation-aware draining, live staged-block tracking, dead-letter preservation, orphan collection, network retirement, and WASM/TypeScript command support. ChangesUpload cancellation lifecycle
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant EngineFacade
participant UploadCancels
participant Drain
participant Staging
participant Network
Client->>EngineFacade: cancelUpload(op_id)
EngineFacade->>UploadCancels: request cancellation
Drain->>UploadCancels: check at upload boundary
Drain->>Staging: release staged version blocks
Drain->>Network: retire confirmed uploaded blocks
EngineFacade-->>Client: UploadCancelled or cancellation error
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
3a96aef to
c6b5593
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
crates/engine/tests/write_plane.rs (1)
3107-3112: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAssert both producers retired, not the exact batch count.
retirechunks its targets atRETIRE_BATCH_MAX, so the number of HTTP calls depends on the version size. A larger fixture would break this assertion without any behavior change. Assertbatches.len() >= 2instead, or assert that the drain's batch contains a CID the facade's batch does not.♻️ Proposed change
- assert_eq!( - batches.len(), - 2, + assert!( + batches.len() >= 2, "a block confirming inside the facade's window is only covered by the drain's batch" );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/engine/tests/write_plane.rs` around lines 3107 - 3112, Update the assertion in the retire_batches test to verify at least two batches are produced rather than requiring exactly two, accommodating RETIRE_BATCH_MAX chunking while still confirming both producers retired.packages/client/src/facade.ts (1)
111-117: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument the
notAnUploadrejection.
cancelUpload()accepts any bigint operation ID, and the engine rejects metadata operations withNotAnUpload. The WASM boundary exposes this asnotAnUpload. Add it to the public method contract so callers know both refusal outcomes.Suggested documentation update
- * with `tooLateToCancel` once the version's record is publishing. + * with `notAnUpload` for non-upload operations or `tooLateToCancel` + * once the version's record is publishing.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/client/src/facade.ts` around lines 111 - 117, Update the JSDoc for cancelUpload() to document that it rejects with notAnUpload when the supplied operation ID refers to a metadata operation, while preserving the existing tooLateToCancel rejection description.crates/fuse/src/error.rs (1)
87-88: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd regression coverage for both new internal mappings.
TooLateToCancelandNotAnUploadnow use the host-sideVfsError::Internalarm, but the conversion test does not exercise either variant. Add both variants to the test input so a future change cannot classify them as trust or storage verdicts.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/fuse/src/error.rs` around lines 87 - 88, Update the error conversion test input in error.rs to include both EngineError::TooLateToCancel and EngineError::NotAnUpload, ensuring each is covered by the VfsError::Internal mapping assertion and cannot regress to a trust or storage classification.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/engine/src/facade.rs`:
- Around line 2107-2121: Update the cascade handling in the operation method
containing the shown discard logic so a failing cascade step emits
Event::SnapshotUpdated before returning the error. Preserve the existing event
emission on the successful exit, ensuring both the error and success paths
notify the host after the primary operation is removed.
In `@crates/engine/src/sync/drain.rs`:
- Around line 510-516: The cancellation path can retire uploaded blocks before
durable queue removal commits, allowing withdraw to restore an unsafe claim. In
crates/engine/src/sync/drain.rs:510-516, gate retire_cancelled in the
Halt::Cancelled handling on durable removal, or mark the in-flight record
terminal so the claim cannot be restored after retirement. In
crates/engine/src/sync/cancel.rs:51-56, update withdraw to reject claims whose
blocks were retired and have the facade translate that rejection into
TooLateToCancel.
In `@crates/engine/tests/write_plane.rs`:
- Around line 3155-3160: In both upload-cancellation tests, including
a_cancelled_versions_upload_mark_never_counts_towards_the_next_one and
a_leaf_a_lost_release_stranded_on_a_cancel_is_reclaimed_by_the_next_sweep,
assert uploads(&alice) > 0 immediately after the four poll_once calls and before
issuing Command::CancelUpload. Preserve the existing cancellation and
stranded-leaf assertions.
---
Nitpick comments:
In `@crates/engine/tests/write_plane.rs`:
- Around line 3107-3112: Update the assertion in the retire_batches test to
verify at least two batches are produced rather than requiring exactly two,
accommodating RETIRE_BATCH_MAX chunking while still confirming both producers
retired.
In `@crates/fuse/src/error.rs`:
- Around line 87-88: Update the error conversion test input in error.rs to
include both EngineError::TooLateToCancel and EngineError::NotAnUpload, ensuring
each is covered by the VfsError::Internal mapping assertion and cannot regress
to a trust or storage classification.
In `@packages/client/src/facade.ts`:
- Around line 111-117: Update the JSDoc for cancelUpload() to document that it
rejects with notAnUpload when the supplied operation ID refers to a metadata
operation, while preserving the existing tooLateToCancel rejection description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d6a4fa61-847c-48aa-95b6-9056e9c69439
📒 Files selected for processing (15)
crates/engine/src/content/write.rscrates/engine/src/facade.rscrates/engine/src/sync/cancel.rscrates/engine/src/sync/drain.rscrates/engine/src/sync/mod.rscrates/engine/src/sync/staging.rscrates/engine/tests/write_plane.rscrates/fuse/src/error.rscrates/wasm/src/host.rscrates/wasm/src/lib.rspackages/client/src/facade.tspackages/client/src/worker/commandCodec.test.tspackages/client/src/worker/commandCodec.tspackages/client/src/worker/engineWasm.tspackages/client/src/worker/protocol.ts
💤 Files with no reviewable changes (1)
- crates/engine/src/content/write.rs
CodeRabbit body-only nitpicksThe three nitpicks in the review body (no threads attached). All verified against the branch; two applied in 22edb2a, one declined.
Extended beyond the finding to
The premise is that Loosening it also costs the assertion its meaning. What it pins is that two distinct producers retire — the facade against what it could see when the cancel landed, and the drain against the complete confirmed set — which is exactly what its message claims. Under CodeRabbit rated this one 🔵 Trivial / 💤 Low value itself. Its stronger alternative — assert the drain's batch carries a CID the facade's does not — is a real improvement over both forms, but the set-equality assertion immediately below already covers the union, so the marginal gain does not carry its own churn. Generated by Claude Code |
22edb2a to
7ab1e29
Compare
Adds `Command::CancelUpload { op_id }`, content-only and guaranteed until
publish entry. The drain yields at every block boundary so a cancel can
actually land mid-transfer, and an interlock decides the race in one
borrow: either the facade claims the op and the drain abandons its
upload, or the drain claims it for publish and the cancel is refused with
`TooLateToCancel`. A cancelled create takes every later queued op on its
node; a cancelled version takes nothing.
Gives orphan GC its first production caller — cold start and after each
drain pass — with open write handles excluded by CID, and reconciles the
four-way staged-byte lifetime in one place: preserved on a terminally
unrebasable dead letter, released on cancel, on a proven-unopenable blob,
and on an unexpandable staged root. Preservation now survives the cold
start that drops the op record, via a durable preserved-roots record that
GC treats as a second reference source.
Closes #853
Part of #655
Security review: - the orphan-GC live-set guard counted only handles opened, so an already-open handle staging its tail and root mid-sweep could have them collected; the counter now moves on every recorded key - the cancel retired the version's whole manifest, which in the window between an acked record PUT and the op's dequeue would unpin content a live record still names; it now retires only the blocks this session's drain confirmed, the publish claim is sticky across a failed attempt, and the dequeue runs before anything is unpinned Crypto/privacy review: - the cold-start undecodable path deleted the blocks named by a clear, unauthenticated header, so a co-tenant of the origin-shared store could plant a record bearing our public owner tag and destroy a queued version; removing the op record is enough, and orphan GC then reclaims on reference evidence - a preserved dead letter kept ciphertext no key could open, because the abandonment deleted the record carrying the version's content key; the preserved set now holds the whole op record Simplify: collapse staged_manifest into version_leaf_cids, fold the duplicated yield-then-check into cancel_checkpoint, collapse the publish claim and the confirmed-block list into one in-flight record, enumerate staged keys before reading the queue so an idle store short-circuits, and narrow the visibility widenings back to what is called.
…ed mark Rebasing onto #941 put the cancel interlock and the mark-before-release ordering in the same match arm. Both are kept: a confirmed block is recorded for the retire set, then the mark advances only when it is allowed to, then the leaf is released. The window that reorder widens is not the one it looks like. A cancel cannot strand a marked-but-unreleased leaf — the facade releases every leaf the root manifest lists, mark or no mark — and the mark it leaves behind names a root nothing can upload again. What is reachable is the upload the drain was already awaiting when the cancel landed: it confirms after the facade snapshotted its retire batch, so it stayed charged with nothing left to reach it. The drain now retires the complete confirmed set on its cancelled arm, which is idempotent against the facade's. Tests: the cancel retire is asserted as a set over both batches; a cancelled version's stale mark is shown not to count towards the next one; and a leaf a lost release strands on the cancel path is shown to be reclaimed by the sweep, since nothing re-runs that release.
The facade publishes the cancel claim before its durable removal commits, so a pass that stops on that claim cannot assume the op is gone. It retired the confirmed set unconditionally, and a removal that then failed put the op back in play with its leading leaves unpinned — the version publishes naming blocks nobody holds. The retire now follows the facade's own rule and runs only behind a removal of its own. Also emit SnapshotUpdated when a cancel's cascade step fails, guard the mid-transfer precondition in the two cancellation tests that lacked it, cover the TooLateToCancel and NotAnUpload host mappings, and document the notAnUpload rejection on the cancelUpload contract.
The focus window's folder refresh runs before the drain each pass and merges a folder's published children into the base, so it is worth holding the line that it cannot carry a cancelled upload back into the folder the user is looking at. The fixture authors the folder on one device and focuses it on another, because the refresh needs the folder's own ipnsName and only a resolved parent ref supplies it. A second writer's child is planted between the two passes so the assertion fails if the refresh never ran at all.
7ab1e29 to
bed26ae
Compare
|
Rebased onto While tracing the two retirement paths that now coexist in Not fixed here: the fix is publish-state durability, not a rebase concern. #966 is ordered to land after this PR. |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/engine/src/facade.rs (1)
2259-2270: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winRoute refused cancel retires into the drain retry set.
discard_uploaddiscards theretireresult, but cancelled content CIDs are already encoded as string CIDs. Addretire(api, &uploaded).await.is_err()handling here, or call the existingretire_cancelledpass so refused batches are not dropped with their pin rows charged.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/engine/src/facade.rs` around lines 2259 - 2270, Update the cancel-discard flow around discard_upload to handle failed retire calls instead of ignoring the result. When retire(api, &uploaded).await fails, route the already encoded uploaded CIDs into the existing drain retry mechanism, or reuse retire_cancelled, so refused batches remain eligible for retry.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/engine/src/facade.rs`:
- Around line 2259-2270: Update the cancel-discard flow around discard_upload to
handle failed retire calls instead of ignoring the result. When retire(api,
&uploaded).await fails, route the already encoded uploaded CIDs into the
existing drain retry mechanism, or reuse retire_cancelled, so refused batches
remain eligible for retry.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 8cdaa3c2-9bb6-41c5-82a5-64bb85221b0f
📒 Files selected for processing (15)
crates/engine/src/content/write.rscrates/engine/src/facade.rscrates/engine/src/sync/cancel.rscrates/engine/src/sync/drain.rscrates/engine/src/sync/mod.rscrates/engine/src/sync/staging.rscrates/engine/tests/write_plane.rscrates/fuse/src/error.rscrates/wasm/src/host.rscrates/wasm/src/lib.rspackages/client/src/facade.tspackages/client/src/worker/commandCodec.test.tspackages/client/src/worker/commandCodec.tspackages/client/src/worker/engineWasm.tspackages/client/src/worker/protocol.ts
💤 Files with no reviewable changes (1)
- crates/engine/src/content/write.rs
🚧 Files skipped from review as they are similar to previous changes (13)
- crates/engine/src/sync/mod.rs
- packages/client/src/worker/protocol.ts
- packages/client/src/worker/commandCodec.test.ts
- crates/wasm/src/host.rs
- packages/client/src/worker/commandCodec.ts
- crates/fuse/src/error.rs
- packages/client/src/facade.ts
- packages/client/src/worker/engineWasm.ts
- crates/engine/src/sync/cancel.rs
- crates/wasm/src/lib.rs
- crates/engine/tests/write_plane.rs
- crates/engine/src/sync/drain.rs
- crates/engine/src/sync/staging.rs
|
Review of
|
Problem
The staged-block lifecycle had two holes on the #813/#825 write-plane line.
A user could not abandon an upload.
blueprint/web-client.mdrequires "cancel is a facade command killing the op before publish", but noCommandvariant existed and the op queue had no cancel path — the only remedy for a queued upload was a compensating delete, which still pushes the whole file through the network and never returns the staging budget.And
orphan_staging_keyshad no production caller, so blocks nothing references — the residue of a crash between staging a version and journaling its op — accumulated against the staging budget forever. #853 recorded the trap that arms the moment GC gets one: the cold-start dead-letter path removes the op record, which is exactly what turns the preserved staged bytes of a dead letter into collectible orphans.Change
Cancel.
Command::CancelUpload { op_id }, content-only — for a metadata op a compensating mutation is already equivalent. Guaranteed until publish entry and refused after withTooLateToCancel, so a cancel can never mutate published state. The drain yields at every block boundary, without which the guarantee collapses to "cancel only works before the op starts". A small interlock decides the race in one non-awaiting borrow each way: either the facade claims the op and the drain abandons its upload, or the drain claims it for publish and the cancel is refused. The claim is sticky across a failed publish attempt — a PUT that did not confirm may still be live at the name. Staged bytes are released, under the rule that splits release from preservation: preserved when the engine gave up on the op, released when the user did. A cancelled create takes every later queued op on its node; a cancelledUpdateContenttakes nothing. The host seesOpPhase::UploadCancelledon the existing progress event.Only the blocks this session's drain confirmed on the network are retired. A block an earlier session sent cannot be told apart from one a version that has since published still names, and unpinning that is loss where leaving the row charged is only a leak (#916).
Orphan GC. First production caller, at cold start and after each drain pass. Roots expand into their leaf sets; open write handles are excluded by CID, recorded before their bytes are staged. A sweep spans many awaits, so it reads a generation counter — bumped by every open and every recorded key — and abandons the pass if the live set moved under it. An unexpandable root freezes the whole pass, self-clearing via the drain's permanent classification.
The four-way staged-byte lifetime is now reconciled in one place,
crates/engine/src/sync/staging.rs: preserved on a terminally unrebasable dead letter, released on cancel, on a proven-unopenable blob (#818), and on an unexpandable staged root. Preservation survives the cold start that drops the op record via a durable preserved set GC treats as a second reference source. That set holds the whole op record, not just the root CID — the record is the only carrier of the version's content key, so preserving the blocks without it would keep ciphertext no key ever opens, which is the condition that releases a version rather than the one that preserves it.Client surface.
Command.cancelUpload(opId)across the wasm boundary andpackages/client, so #873 has something to call.Tests
crates/engine/tests/write_plane.rs:a_cancel_mid_upload_releases_every_block_and_returns_the_staging_budget— the acceptance case. Fails without the block-boundary yield: the whole version publishes inside one poll and the cancel returnsTooLateToCancel.a_cancel_after_the_version_published_is_refused,a_cancel_of_a_metadata_op_is_refuseda_cancelled_create_cascades_onto_its_node_and_a_cancelled_version_does_nota_cancel_that_cannot_dequeue_retires_nothing_and_leaves_the_op_publishable— drop thewithdrawcall and the op is wedged: claimed, still queued, halted forever. Retire before the dequeue instead and the resumed version publishes with its leading leaves unpinned.an_undecodable_record_never_authorizes_deleting_the_blocks_its_header_names— the clear header is unauthenticated and carries a public owner tag, so a co-tenant of the origin-shared store can plant one; the test asserts the forgery is dead-lettered and the version it names survives and still publishes.orphan_residue_is_collected_and_a_live_handles_blocks_are_not— drop the live-set exclusion and the handle's commit publishes a version whose blocks were swept.a_dead_lettered_ops_blocks_survive_a_cold_start_and_a_gc_pass— droppreserve_dead_letterand the first GC pass eats them.Unit:
sync/cancel.rspins the interlock both ways, the sticky claim, and the session-scoped retire set;sync/staging.rspins the preserved-record round trip, the prune, the whole-set release, that a preserved entry still reopens to its intent and sealed key, and the fail-closed freeze on a wrong tag, a length prefix past the end, and a zero-length entry.Verification
All exit 0:
cargo fmt --all --checkcargo clippy --workspace --all-targets -- -D warningscargo check --workspace --all-targetscargo clippy -p cipherbox-wasm --target wasm32-unknown-unknown --all-targets -- -D warningscargo test --workspace— no failing suitespnpm -r --if-present run typecheck,pnpm -r --if-present run test,pnpm exec eslint .pnpm --filter @cipherbox/client test:browser— 19 passedReview gates
/security-reviewraised two findings, both folded in:/crypto-privacy-review— run because the diff touches trust-boundary and fail-closed reads — raised two more, both folded in:It confirmed the preserved-record codec is total, panic-free and fail-closed in the safe direction, the encode side release-active per rule 8; that a cancel can only ever address this session's own ops; that no key material is reachable from the shared
LiveBlockscell (rule 7); and thatyield_nowreads no clock or RNG./simplifyfindings folded in:staged_manifestcollapsed intoversion_leaf_cids, the duplicated yield-then-check folded intocancel_checkpoint, the publish claim and the confirmed-block list collapsed into one in-flight record, the GC scan now enumerates staged keys first and short-circuits on an idle store, and the new visibility widenings and re-exports narrowed back to what is actually called.Known and deferred: a co-tenant of the origin-shared store can freeze a GC pass with a planted undecodable entry, or delete/pad the preserved-set key. All are budget and availability, never confidentiality or loss, and they sit in the shared-store threat model
blueprint/engine.mddefers to queue-integrity work — noted here rather than fixed in this slice.Rebased onto #941
#941 landed the #924 fix first, so this branch is rebased onto it. My earlier note said the conflict surface was "only the loop head" — that was wrong, and worth correcting: #941 inserts its conditional
mark_uploadedin exactly the spot where this branch records a confirmed block, so it was a real body conflict inupload_blocks.Resolved by keeping both of #941's properties intact — mark before release, and the mark written only when it advances — with the confirmed-block record placed immediately after
upload_block, which is the moment it describes. All three of #941's tests pass unmodified; nothing in its semantics needed changing.The cancel-vs-mark window. Neither of the two shapes it could take is reachable:
upload_mark(root_cid), and the cancelled root can never be uploaded again: its op is gone, and a re-upload of the same file seals under a fresh random per-version key and so addresses differently.a_cancelled_versions_upload_mark_never_counts_towards_the_next_onepins that.A third window was reachable, and #941's reorder is not what opened it: the upload the drain is already awaiting when a cancel lands confirms after the facade has snapshotted its retire batch, leaving one block charged with nothing to reach it — #916's failure mode, bounded to one block per cancel. The drain now retires the complete confirmed set on its
Halt::Cancelledarm;retireis idempotent server-side, so the overlap with the facade's batch is a no-op. The cancel test asserts the union of both batches equals exactly the set of blocks that reached the network.Orphan GC against the new ordering. No change needed. GC classifies by reference, never by the mark: a leaf in a queued op's root manifest is referenced whether or not the mark covers it, and a leaf whose op is gone is residue either way.
UPLOAD_MARK_KEYstays in the always-referenced bookkeeping set.a_leaf_a_lost_release_stranded_on_a_cancel_is_reclaimed_by_the_next_sweepcovers the residue shape #941 made reachable, on the one path where nothing re-runs the release.Rebased onto #945 and #943
Rebased again onto
mainafter #945 (focus-window folder resolve) and #943 (SIWE challenge below the facade). One conflict, incrates/engine/src/facade.rs, in two places — both genuine body overlaps, not adjacent insertions:Engine::commandmatch, where feat: resolve the focus window's folders below the scope root #945 replaced theSetFocusarm that previously fell through toUnimplementedand this branch addsCancelUpload. Both arms kept.dequeue_op, where feat: move the SIWE challenge below the facade #943 addedsiwe_challengeand this branch addedcancel_upload/discard_upload/collect_staging_orphans. Both kept.#945's deliberate ordering is preserved: the resolve tick still refreshes the focus window before the drain, and this branch's orphan sweep still runs after it. No test of #945's or #943's was modified.
Does the focus refresh interact with the cancel? No, and the reason is structural rather than incidental:
cancel_uploadreads and writes only the durable op queue, the in-memory cancel interlock, the staging store and the registry. It never reads or writesself.snapshot, which is the only thingFolderRefreshmutates. So there is no half-updated base for a cancel to observe — andFolderRefreshholds noRefCellborrow across an await, so an interleaving cannot even contend.The reverse direction is prevented by what the refresh merges:
project_foldertakes a folder's published children, and a cancel is refused once its record is publishing, so a cancelled node is one the parent's record cannot name.a_focus_refresh_never_renders_back_an_upload_the_user_cancelledholds that line end to end.Two things that fixture had to get right, and that a future reader will trip on: the refresh needs the folder's own
ipnsName, which only a resolved parent ref supplies — so a device that authored the folder itself never descends into it, and the test focuses on a second device. And a first attempt asserted nothing, because the pass the cancel interrupts has already run its focus refresh; the planted second-writer child therefore lands between two passes, and the assertion fails if the refresh never ran.Closes #869
Closes #853
Part of #655
Summary by CodeRabbit
New Features
Bug Fixes
Tests