Repository navigation
fix(satellite): repair a crashed replica's activity log instead of re-initialising it - #220
Open
Andrei Kvapil (kvaps) wants to merge 3 commits into
Open
Andrei Kvapil (kvaps) wants to merge 3 commits into
Andrei Kvapil (kvaps) wants to merge 3 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
…-initialising it A replica that goes down while Primary and writing leaves its activity log open, and drbdmeta answers the metadata probe with "please apply-al first". The satellite read that as "no metadata" and ran create-md --force, which destroys the replica's generation identifiers and dirty bitmap; if it held the only up-to-date copy, that copy came back as something to sync into. It treated a crash the same as a foreign superblock, because it had no per-volume record to tell them apart. The record is the set of GIs this replica's volume was seen to carry. The observer writes every GI it reads into a per-resource history file in the satellite's state directory, which outlives the status field (that one is dropped whenever the volume leaves a data-bearing state) and a restart; the GI in the status counts too. A dirty log on a volume with no record is initialised as before: a clone, a receive, a volume added to a live resource. One with a record is repaired with apply-al, and some UUID then in its metadata, current, a peer slot's bitmap UUID or history, has to be one it was seen with, compared without the Primary bit; otherwise the superblock is not this replica's and it gets metadata of its own. A Primary that lost a peer keeps its old current UUID only in that peer's bitmap slot, which get-gi never shows, so the check reads dump-md. A failed apply-al or metadata read is an error rather than a re-initialisation. Fixes #184 Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Andrei Kvapil (kvaps)
force-pushed
the
fix/unclean-al-apply
branch
from
October 24, 2026 21:53
5e5f2cd to
58b83d5
Compare
… record whole drbdmeta prints a v09 volume's history UUIDs as a block over several lines, and v08's as one uuid block. The parser read neither, so a GI that a resync had moved into the history was never found and the replica was initialised after all. It now reads both blocks and only bare UUID statements, so the bitmap and the device UUID stay out. External metadata takes the same check as internal: dump-md finds the meta disk from the .res. Deleting a resource now drops its cached last GIs together with the file, under the lock the record is written under, so a replica created again records its first GI. The record is fsynced before it replaces the old one, since it exists for a node that went down hard. The observer is built by a constructor a test can drive, which pins the wiring into the reconciler. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
…ap fixture Remove the temporary file when the rename fails, state the window the delete lock leaves open, and give the dump fixtures the bitmap header drbdmeta prints for v09. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A replica that goes down while Primary and writing leaves its activity log open, and drbdmeta answers the metadata probe with "please apply-al first". The satellite read that as "no metadata" and ran
create-md --force, which destroys the replica's generation identifiers and dirty bitmap. If it held the only up-to-date copy, that copy came back as something to sync into. It treated a crash the same as a foreign superblock because it had no per-volume record to tell them apart.The record is the set of GIs this replica's volume was seen to carry. The observer writes every GI it reads into a per-resource history file in the satellite's state directory. That file outlives the status field, which is dropped whenever the volume leaves a data-bearing disk state, and survives a restart; the GI in the status counts too. It is fsynced before it replaces the old one, and deleting the resource drops it.
drbdadm apply-al. Then some UUID in its metadata, the current one, any peer slot's bitmap UUID or a history UUID, has to be one it was seen with, compared without the Primary bit. A Primary that lost a peer keeps its old current UUID only in that peer's bitmap slot, whichget-ginever shows, and after a resync the GI sits in the history block. So the check readsdump-md, in both the v09 and the v08 layout. If none matches, the superblock is not this replica's and it gets metadata of its own after all.apply-alor metadata read is an error rather than a re-initialisation.dump-mdfinds the meta disk from the.res.Both metadata paths (
ensureMetadataandensurePerVolumeMetadata) take the same repair. A volume with no record falls back to the old behaviour, so this only ever narrows when metadata is destroyed. A failed repair surfaces in the satellite log and as a requeue; there is no Event for it yet. The check holds the wholedump-mdoutput, bitmap included, in memory once per repair.Stacked on #190.
Fixes #184
Testing
create-md; metadata that does not name it is initialised, internal or external; a failingapply-alor metadata read is an error with nocreate-md; the history survives a restart, keeps the newest 32 GIs per volume and goes with the resource; the observer records GIs only for data-bearing volumes, and the one the manager runs feeds the reconciler. The parser fixtures follow the layoutprint_dump_giin drbd-utils prints. Reverting the block parsing, the Primary-bit comparison, the history lookup, the fail-closed read, the external-metadata check, the cap, the delete cleanup, the repeat cache or the observer wiring reddens the matching test.go test ./...: green apart from the knownpkg/restshutdown flake.