Skip to content

fix(restore): keep a restore's source, refuse orphan targets, take over stale rollback marks - #218

Open
Andrei Kvapil (kvaps) wants to merge 4 commits into
fix/csi-clone-and-restore-idempotencyfrom
fix/restore-source-and-rollback
Open

Andrei Kvapil (kvaps) wants to merge 4 commits into
fix/csi-clone-and-restore-idempotencyfrom
fix/restore-source-and-rollback

Conversation

@kvaps

@kvaps Andrei Kvapil (kvaps) commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Three fixes on the restore and rollback paths, one commit each.

Restore source (#197)

A restored or cloned definition keeps its restore marker, and while it is there every new replica is brought up from that snapshot and placed on its nodes. Nothing protected the snapshot: s d deleted it straight away and the auto-snapshot prune deleted by age, so the definition's next replica came up blank and never reached UpToDate. A snapshot delete, on the REST door and in the CLI, now ends the dependency first: a definition whose restore is finished, by the same rule an operator's own delete of the marker passes, has the marker taken off, and its later replicas sync from their peers. One whose restore is still running keeps it, and the delete answers a retryable 409 for as long as that lasts. The prune skips such a snapshot for the pass. The clone door already refused rewriting the marker through override_props.

Orphan replicas under the target name (#203)

A restore or clone into a name with no definition but with a diskless replica or a witness under it created the definition over that replica, failed the stamp on its node, and the rollback cascade then deleted a replica the request had not made. Both restore doors, REST and CLI, now refuse such a name with 409 before writing anything and name the replicas. A replica holding data is still taken as placed, as #190 does, and the controller's tie-breaker witness is still promoted.

Stale rollback mark (#213)

An apiserver replica killed between writing the in-progress rollback mark and taking it off left the mark for good, and linstor-csi retried against the 409 until somebody cleared the prop. The mark now carries the time it was written, and a retry on either door takes over a mark older than two minutes, well past the rollback's 21-second budget, and resumes the leftover. The mark only ever stands over a definition left whole, and a rollback still alive past its budget finds the mark gone at its next step and stands down without deleting. The CLI rollback, which had no deadline, now runs on a one-minute budget of its own, so the two-minute rule holds on both doors. A mark written before this carries no time and is never taken over.

Stacked on #190. #200 touches the same materialisation; the conflict between the two is a two-line merge.

Fixes #197
Fixes #203
Fixes #213

Testing

  • Every new guard was reverted in turn and its named test fails: the release in the REST and CLI deletes, the marker removal, the prune skip with and without a store, the orphan refusal on both doors, the stale takeover in the REST gate and on the CLI, and the staleness check.
  • go test ./...: green apart from the known pkg/rest shutdown flake.
  • go test -tags=integration ./tests/integration/...: green.
  • golangci-lint on the changed lines: clean.
  • Not run on a stand.

@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ac069f12-17ea-43cf-b88c-c42d54307d41

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

A definition restored or cloned from a snapshot keeps its restore
marker, and while the marker is there every new replica is brought up
from that snapshot and placed on its nodes. Nothing protected the
snapshot: `s d` deleted it straight away and the auto-snapshot prune
deleted by age, so the definition's next replica came up blank and
never reached UpToDate.

A snapshot delete, on the REST door and in the CLI, now first ends the
dependency: a definition whose restore is finished, by the rule an
operator's own delete of the marker passes, has the marker taken off,
and its later replicas sync from their peers. One whose restore is
still running keeps it, and the delete answers a retryable 409 for as
long as that lasts. The prune skips such a snapshot for the pass.

The clone door already refused rewriting the marker through
override_props.

Fixes #197

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
@kvaps
Andrei Kvapil (kvaps) force-pushed the fix/restore-source-and-rollback branch from df657bf to 9f6576e Compare October 24, 2026 20:01
A restore or clone into a name with no definition but with a diskless
replica or a witness under it created the definition over that
replica. The stamp on its node then failed, and the rollback of the
definition cascaded over every replica under the name, deleting one
the request had not made.

The materialisation now refuses such a name with 409 before it writes
anything, and names the replicas. A replica holding data is still
taken as placed, as before, and one already being deleted is no
obstacle.

Fixes #203

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
A clone or restore rollback marks the definition in-progress while it
runs, and every retry over that mark answers 409. An apiserver replica
OOM-killed or lost between writing the mark and taking it off left it
for good, and linstor-csi retried CreateVolume against the 409 until
an operator cleared the prop by hand.

The mark now carries the time it was written. A retry, on the REST
doors and in the CLI, takes over a mark older than two minutes, well
past the rollback's own budget, and resumes the leftover. The mark
only ever stands over a definition left whole, and a rollback still
alive past its budget finds the mark gone at its next step and stands
down without deleting.

Fixes #213

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
@kvaps
Andrei Kvapil (kvaps) force-pushed the fix/restore-source-and-rollback branch 2 times, most recently from 6a00c9a to 52d0763 Compare October 24, 2026 21:56
…fixes

- The orphan refusal moves to the store and runs on the CLI restore
  door too, which created the definition over the orphan the same way.
  It refuses diskless replicas and leaves the controller's tie-breaker
  witness to the stamp, which promotes it.
- A CLI rollback now runs on a one-minute budget of its own. The stale
  takeover rests on every rollback being over long before two minutes,
  and the CLI's had no deadline at all.
- A retry that finds the in-progress mark already taken off by another
  retry proceeds instead of answering that a rollback is running.
- Tests for the store-backed prune, which production runs, and for the
  CLI takeover.
- The snapshot-delete refusal sits above handleSnapshotDelete's doc
  comment rather than under it, and the unused RestoreSourceInUse is
  gone.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant