Repository navigation
fix(restore): keep a restore's source, refuse orphan targets, take over stale rollback marks - #218
Open
Andrei Kvapil (kvaps) wants to merge 4 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
A definition restored or cloned from a snapshot keeps its restore marker, and while the marker is there every new replica is brought up from that snapshot and placed on its nodes. Nothing protected the snapshot: `s d` deleted it straight away and the auto-snapshot prune deleted by age, so the definition's next replica came up blank and never reached UpToDate. A snapshot delete, on the REST door and in the CLI, now first ends the dependency: a definition whose restore is finished, by the rule an operator's own delete of the marker passes, has the marker taken off, and its later replicas sync from their peers. One whose restore is still running keeps it, and the delete answers a retryable 409 for as long as that lasts. The prune skips such a snapshot for the pass. The clone door already refused rewriting the marker through override_props. Fixes #197 Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Andrei Kvapil (kvaps)
force-pushed
the
fix/restore-source-and-rollback
branch
from
October 24, 2026 20:01
df657bf to
9f6576e
Compare
A restore or clone into a name with no definition but with a diskless replica or a witness under it created the definition over that replica. The stamp on its node then failed, and the rollback of the definition cascaded over every replica under the name, deleting one the request had not made. The materialisation now refuses such a name with 409 before it writes anything, and names the replicas. A replica holding data is still taken as placed, as before, and one already being deleted is no obstacle. Fixes #203 Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
A clone or restore rollback marks the definition in-progress while it runs, and every retry over that mark answers 409. An apiserver replica OOM-killed or lost between writing the mark and taking it off left it for good, and linstor-csi retried CreateVolume against the 409 until an operator cleared the prop by hand. The mark now carries the time it was written. A retry, on the REST doors and in the CLI, takes over a mark older than two minutes, well past the rollback's own budget, and resumes the leftover. The mark only ever stands over a definition left whole, and a rollback still alive past its budget finds the mark gone at its next step and stands down without deleting. Fixes #213 Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Andrei Kvapil (kvaps)
force-pushed
the
fix/restore-source-and-rollback
branch
2 times, most recently
from
October 24, 2026 21:56
6a00c9a to
52d0763
Compare
…fixes - The orphan refusal moves to the store and runs on the CLI restore door too, which created the definition over the orphan the same way. It refuses diskless replicas and leaves the controller's tie-breaker witness to the stamp, which promotes it. - A CLI rollback now runs on a one-minute budget of its own. The stale takeover rests on every rollback being over long before two minutes, and the CLI's had no deadline at all. - A retry that finds the in-progress mark already taken off by another retry proceeds instead of answering that a rollback is running. - Tests for the store-backed prune, which production runs, and for the CLI takeover. - The snapshot-delete refusal sits above handleSnapshotDelete's doc comment rather than under it, and the unused RestoreSourceInUse is gone. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Andrei Kvapil (kvaps)
force-pushed
the
fix/restore-source-and-rollback
branch
from
October 24, 2026 22:14
52d0763 to
2bbc42c
Compare
Andrei Kvapil (kvaps)
marked this pull request as ready for review
October 24, 2026 22:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three fixes on the restore and rollback paths, one commit each.
Restore source (#197)
A restored or cloned definition keeps its restore marker, and while it is there every new replica is brought up from that snapshot and placed on its nodes. Nothing protected the snapshot:
s ddeleted it straight away and the auto-snapshot prune deleted by age, so the definition's next replica came up blank and never reached UpToDate. A snapshot delete, on the REST door and in the CLI, now ends the dependency first: a definition whose restore is finished, by the same rule an operator's own delete of the marker passes, has the marker taken off, and its later replicas sync from their peers. One whose restore is still running keeps it, and the delete answers a retryable 409 for as long as that lasts. The prune skips such a snapshot for the pass. The clone door already refused rewriting the marker throughoverride_props.Orphan replicas under the target name (#203)
A restore or clone into a name with no definition but with a diskless replica or a witness under it created the definition over that replica, failed the stamp on its node, and the rollback cascade then deleted a replica the request had not made. Both restore doors, REST and CLI, now refuse such a name with 409 before writing anything and name the replicas. A replica holding data is still taken as placed, as #190 does, and the controller's tie-breaker witness is still promoted.
Stale rollback mark (#213)
An apiserver replica killed between writing the in-progress rollback mark and taking it off left the mark for good, and linstor-csi retried against the 409 until somebody cleared the prop. The mark now carries the time it was written, and a retry on either door takes over a mark older than two minutes, well past the rollback's 21-second budget, and resumes the leftover. The mark only ever stands over a definition left whole, and a rollback still alive past its budget finds the mark gone at its next step and stands down without deleting. The CLI rollback, which had no deadline, now runs on a one-minute budget of its own, so the two-minute rule holds on both doors. A mark written before this carries no time and is never taken over.
Stacked on #190. #200 touches the same materialisation; the conflict between the two is a two-line merge.
Fixes #197
Fixes #203
Fixes #213
Testing
go test ./...: green apart from the knownpkg/restshutdown flake.go test -tags=integration ./tests/integration/...: green.