test(longhaul): add data-protection verifier (ScheduledBackup + retention) #419
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| // Copyright (c) Microsoft Corporation. | ||
| // Licensed under the MIT License. | ||
|
|
||
| // Package backup implements the data-protection component of the long haul | ||
| // driver. It provisions a ScheduledBackup against the canary cluster and | ||
| // continuously verifies the properties that only a multi-day run can | ||
| // establish: that backups keep being produced on schedule and completing, | ||
| // and that expired backups are actually garbage-collected so the backup | ||
| // population stays bounded over time (no PVC / VolumeSnapshot accumulation). | ||
| // | ||
| // It deliberately does NOT re-verify the operator's retention *arithmetic* | ||
| // (expiredAt == stoppedAt + retentionDays*24h) — that is a pure function | ||
| // already covered by the operator's unit tests and needs no accumulation. | ||
| // The oracle here is black-box: expired backups disappear. | ||
| // | ||
| // The component runs concurrently with the operation scheduler — per the | ||
| // long-haul design, backup is deliberately NOT isolated so that | ||
| // backup-vs-topology serialization bugs surface here rather than in | ||
| // production. | ||
| package backup | ||
|
|
||
| import ( | ||
| "sync/atomic" | ||
| "time" | ||
| ) | ||
|
|
||
| // completionStallThreshold is the number of consecutive backups that may be | ||
| // scheduled with no intervening completion before the run is failed. A single | ||
| // recovered (completed) backup resets the running gap, so transient chaos- | ||
| // induced failures stay well under this ceiling; only a wholly-broken | ||
| // completion path (every backup failing or hanging) drives the gap this high. | ||
| // At the default 6h schedule this is ~18h of zero successful backups. | ||
| const completionStallThreshold = 3 | ||
|
|
||
| // Metrics tracks aggregate backup-verification counters using atomic | ||
| // operations so the reporter goroutine can snapshot them without locking. | ||
| // | ||
| // Two independent oracles flip the run verdict to FAIL: | ||
| // - RetentionLeaks > 0: a completed backup outlived its retention window | ||
| // (the operator failed to garbage-collect it). | ||
| // - MaxScheduledWithoutCompletion >= completionStallThreshold: backups keep | ||
| // being scheduled but stop completing (a dead completion path the leak | ||
| // oracle alone would miss). | ||
| // | ||
| // The remaining counters are observational and feed the report. | ||
| type Metrics struct { | ||
| // Scheduled counts backups observed to have been scheduled by the | ||
| // ScheduledBackup (advances of status.lastScheduledTime). | ||
| Scheduled atomic.Int64 | ||
|
|
||
| // Completed is the number of child backups observed in the "completed" | ||
| // phase (deduplicated by name across verification cycles). | ||
| Completed atomic.Int64 | ||
|
|
||
| // Failed is the number of child backups observed in a terminal failure | ||
| // phase (deduplicated by name). | ||
| Failed atomic.Int64 | ||
|
|
||
| // Skipped is the number of child backups observed in the "skipped" phase | ||
| // (deduplicated by name). Skipped is an intentional no-op, not a failure, | ||
| // and resets the completion-stall gap like a completion. Observational. | ||
| Skipped atomic.Int64 | ||
|
|
||
| // RetentionLeaks counts completed backups still present past their | ||
| // retention window (stoppedAt + retentionDays*24h). Non-zero => FAIL. | ||
| RetentionLeaks atomic.Int64 | ||
|
|
||
| // MaxScheduledWithoutCompletion is the high-water mark of consecutive | ||
| // backups scheduled with no intervening completion. It is the completion- | ||
| // liveness oracle: a wholly-broken backup path drives it monotonically | ||
| // upward while Completed stays flat. Reaching completionStallThreshold | ||
| // flips the verdict to FAIL. See observeCompletionGap for how it advances. | ||
| MaxScheduledWithoutCompletion atomic.Int64 | ||
|
|
||
| // LastChildCount is the number of child backups observed on the most | ||
| // recent verification cycle (the live backup population — expected to | ||
| // stabilize near retentionWindow/scheduleInterval at steady state). | ||
| LastChildCount atomic.Int64 | ||
|
|
||
| // LastScheduledUnix is the Unix timestamp of the most recently observed | ||
| // status.lastScheduledTime; 0 until the first backup is scheduled. | ||
| LastScheduledUnix atomic.Int64 | ||
| } | ||
|
|
||
| // NewMetrics creates an empty Metrics. | ||
| func NewMetrics() *Metrics { | ||
| return &Metrics{} | ||
| } | ||
|
|
||
| // observeCompletionGap records gap as a new high-water mark for consecutive | ||
| // scheduled-without-completion backups if it exceeds the current maximum. | ||
| func (m *Metrics) observeCompletionGap(gap int64) { | ||
| for { | ||
| cur := m.MaxScheduledWithoutCompletion.Load() | ||
| if gap <= cur { | ||
| return | ||
| } | ||
| if m.MaxScheduledWithoutCompletion.CompareAndSwap(cur, gap) { | ||
| return | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // MetricsSnapshot is a point-in-time copy of Metrics. | ||
| type MetricsSnapshot struct { | ||
| Scheduled int64 | ||
| Completed int64 | ||
| Failed int64 | ||
| Skipped int64 | ||
| RetentionLeaks int64 | ||
| MaxScheduledWithoutCompletion int64 | ||
| LastChildCount int64 | ||
| LastScheduled time.Time | ||
| } | ||
|
|
||
| // Snapshot captures the current metric values atomically. | ||
| func (m *Metrics) Snapshot() MetricsSnapshot { | ||
| var lastScheduled time.Time | ||
| if unix := m.LastScheduledUnix.Load(); unix > 0 { | ||
| lastScheduled = time.Unix(unix, 0) | ||
| } | ||
| return MetricsSnapshot{ | ||
| Scheduled: m.Scheduled.Load(), | ||
| Completed: m.Completed.Load(), | ||
| Failed: m.Failed.Load(), | ||
| Skipped: m.Skipped.Load(), | ||
| RetentionLeaks: m.RetentionLeaks.Load(), | ||
| MaxScheduledWithoutCompletion: m.MaxScheduledWithoutCompletion.Load(), | ||
| LastChildCount: m.LastChildCount.Load(), | ||
| LastScheduled: lastScheduled, | ||
| } | ||
| } | ||
|
|
||
| // HasRetentionLeak returns true if any completed backup has outlived its | ||
| // retention window. A true result flips the overall run verdict to FAIL. | ||
| func (s MetricsSnapshot) HasRetentionLeak() bool { | ||
|
Collaborator
There was a problem hiding this comment. 🟠 Major / Question — a wholly-broken backup subsystem can still report PASS. Only So if every child backup terminally fails — Is the non-fatal
Collaborator
Author
There was a problem hiding this comment. Addressed in 7c4405a. Added a completion-liveness oracle: the verifier now tracks the high-water mark of consecutive backups scheduled with no intervening completion (
Collaborator
Author
There was a problem hiding this comment. Good catch, and agreed on the sustained-failure gate. `HasCompletionStall()` ("no completed backup within N schedule intervals", N=3) is now wired into the run verdict in `cmd/longhaul/main.go` `buildSummary` via `appendReason`, alongside the retention-leak oracle. This keeps tolerance of transient/chaos-induced `Failed` (still observational) but flips the run to FAIL when the backup path is genuinely broken (`Completed=0` while schedules keep advancing). Skipped children reset the gap so standby/failover doesn't false-trip it. |
||
| return s.RetentionLeaks > 0 | ||
| } | ||
|
|
||
| // HasCompletionStall returns true if backups kept being scheduled but stopped | ||
| // completing for completionStallThreshold consecutive schedules. A true result | ||
| // flips the overall run verdict to FAIL, catching a wholly-broken backup path | ||
| // that the retention-leak oracle alone would miss. | ||
| func (s MetricsSnapshot) HasCompletionStall() bool { | ||
| return s.MaxScheduledWithoutCompletion >= completionStallThreshold | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| // Copyright (c) Microsoft Corporation. | ||
| // Licensed under the MIT License. | ||
|
|
||
| package backup | ||
|
|
||
| import ( | ||
| "time" | ||
|
|
||
| . "github.com/onsi/ginkgo/v2" | ||
| . "github.com/onsi/gomega" | ||
| ) | ||
|
|
||
| var _ = Describe("Metrics", func() { | ||
| It("snapshots counters atomically", func() { | ||
| m := NewMetrics() | ||
| m.Scheduled.Add(3) | ||
| m.Completed.Add(2) | ||
| m.Failed.Add(1) | ||
| m.RetentionLeaks.Add(1) | ||
| m.LastChildCount.Store(7) | ||
| now := time.Now() | ||
| m.LastScheduledUnix.Store(now.Unix()) | ||
|
|
||
| snap := m.Snapshot() | ||
| Expect(snap.Scheduled).To(Equal(int64(3))) | ||
| Expect(snap.Completed).To(Equal(int64(2))) | ||
| Expect(snap.Failed).To(Equal(int64(1))) | ||
| Expect(snap.RetentionLeaks).To(Equal(int64(1))) | ||
| Expect(snap.LastChildCount).To(Equal(int64(7))) | ||
| Expect(snap.LastScheduled.Unix()).To(Equal(now.Unix())) | ||
| }) | ||
|
|
||
| It("reports zero LastScheduled when never scheduled", func() { | ||
| snap := NewMetrics().Snapshot() | ||
| Expect(snap.LastScheduled.IsZero()).To(BeTrue()) | ||
| }) | ||
|
|
||
| DescribeTable("HasRetentionLeak", | ||
| func(leaks int64, want bool) { | ||
| m := NewMetrics() | ||
| m.RetentionLeaks.Add(leaks) | ||
| Expect(m.Snapshot().HasRetentionLeak()).To(Equal(want)) | ||
| }, | ||
| Entry("clean", int64(0), false), | ||
| Entry("one leak", int64(1), true), | ||
| Entry("several leaks", int64(3), true), | ||
| ) | ||
|
|
||
| DescribeTable("HasCompletionStall", | ||
| func(gap int64, want bool) { | ||
| m := NewMetrics() | ||
| m.observeCompletionGap(gap) | ||
| Expect(m.Snapshot().HasCompletionStall()).To(Equal(want)) | ||
| }, | ||
| Entry("no schedules", int64(0), false), | ||
| Entry("one in flight", int64(1), false), | ||
| Entry("two in flight", int64(2), false), | ||
| Entry("at threshold", int64(completionStallThreshold), true), | ||
| Entry("past threshold", int64(completionStallThreshold+2), true), | ||
| ) | ||
|
|
||
| It("observeCompletionGap keeps only the high-water mark", func() { | ||
| m := NewMetrics() | ||
| m.observeCompletionGap(2) | ||
| m.observeCompletionGap(5) | ||
| m.observeCompletionGap(3) // lower value must not lower the mark | ||
| Expect(m.Snapshot().MaxScheduledWithoutCompletion).To(Equal(int64(5))) | ||
| }) | ||
| }) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Required] This window is too tight for a reliable end-to-end backup assertion. The driver only runs
checkOnceon the first 5-minute ticker, while the first*/1schedule can occur nearly a minute after bootstrap and snapshot creation/completion can take additional time. If the backup is not completed by that single tick, the final report hasBackups Completed = 0and this otherwise healthy smoke run fails intermittently. Please either make the smoke run materially longer (for example, at least 10–15 minutes) or make the verifier perform an immediate/short-cadence initial check before relying on the 5-minute interval.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed by making the verify cadence configurable rather than draining at shutdown. The smoke run sets `LONGHAUL_BACKUP_VERIFY_INTERVAL: 30s` (vs the 5m default) so the verifier's real periodic loop ticks ~12x within the 6m window and reliably observes a scheduled+completed backup. We chose the short-cadence loop over a one-off immediate check so the gate genuinely proves the periodic verifier works, not just that a shutdown snapshot happened.