Skip to content

feat(dev): add missing snapshots checker#6736

Merged
LesnyRumcajs merged 1 commit intomainfrom
check-missing-snaps-dev-cmd
Mar 13, 2026
Merged

feat(dev): add missing snapshots checker#6736
LesnyRumcajs merged 1 commit intomainfrom
check-missing-snaps-dev-cmd

Conversation

@LesnyRumcajs
Copy link
Member

@LesnyRumcajs LesnyRumcajs commented Mar 13, 2026

Summary of changes

Changes introduced in this pull request:

  • added an utility command that can be used by infra team to validate the correctness of the archive

Reference issue to close (if applicable)

Closes

Other information and links

❯ cargo run --bin forest-dev -- archive-missing --chain mainnet
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.26s
     Running `target/debug/forest-dev archive-missing --chain mainnet`
Checking mainnet epochs 0..=5834000 (current network epoch: 5837000)
Fetching archive listings...
Archive has 193 lite and 1897 diff snapshots.

50 of 2140 required snapshots are MISSING:

  Missing lite snapshots:
    lite at height 5790000
    lite at height 5820000
  Missing diff snapshots:
    diff at height 3987000
    diff at height 3990000
    diff at height 4020000
    diff at height 4050000
    diff at height 4080000
    diff at height 4110000
    diff at height 4140000
    diff at height 4170000
    diff at height 4200000
    diff at height 4230000
    diff at height 4260000
    diff at height 4290000
    diff at height 4320000
    diff at height 4350000
    diff at height 4380000
    diff at height 4410000
    diff at height 4440000
    diff at height 4530000
    diff at height 4560000
    diff at height 4590000
    diff at height 4620000
    diff at height 4650000
    diff at height 4680000
    diff at height 4710000
    diff at height 4740000
    diff at height 4770000
    diff at height 5433000
    diff at height 5436000
    diff at height 5607000
    diff at height 5766000
    diff at height 5769000
    diff at height 5772000
    diff at height 5775000
    diff at height 5778000
    diff at height 5781000
    diff at height 5784000
    diff at height 5787000
    diff at height 5790000
    diff at height 5793000
    diff at height 5796000
    diff at height 5799000
    diff at height 5802000
    diff at height 5805000
    diff at height 5808000
    diff at height 5811000
    diff at height 5814000
    diff at height 5817000
    diff at height 5820000
Error: 50 of 2140 required snapshots are missing
❯ cargo run --bin forest-dev -- archive-missing --chain calibnet
   Compiling forest-filecoin v0.32.4 (/home/rumcajs/prj/forest3)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.35s
     Running `target/debug/forest-dev archive-missing --chain calibnet`
Checking calibnet epochs 0..=3533334 (current network epoch: 3536334)
Fetching archive listings...
Archive has 118 lite and 1175 diff snapshots.

4 of 1296 required snapshots are MISSING:

  Missing diff snapshots:
    diff at height 3132000
    diff at height 3135000
    diff at height 3138000
    diff at height 3435000
Error: 4 of 1296 required snapshots are missing

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Outside contributions

  • I have read and agree to the CONTRIBUTING document.
  • I have read and agree to the AI Policy document. I understand that failure to comply with the guidelines will lead to rejection of the pull request.

Summary by CodeRabbit

New Features

  • Added a new command to verify the availability of archive snapshots over a specified epoch range
  • Supports optional epoch bounds (defaults: from 0 to current epoch minus 3000)
  • Works with mainnet and calibnet networks
  • Reports missing snapshot data and exits with an error if any required snapshots are unavailable

@LesnyRumcajs LesnyRumcajs requested a review from a team as a code owner March 13, 2026 15:41
@LesnyRumcajs LesnyRumcajs requested review from akaladarshi and hanabi1224 and removed request for a team March 13, 2026 15:41
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 13, 2026

Walkthrough

Introduces a new ArchiveMissingCommand to verify archive snapshot availability across epoch ranges. The command computes required lite and diff snapshot epochs, fetches remote archive listings via HTTP, parses JSON responses, and identifies missing snapshots within the specified range.

Changes

Cohort / File(s) Summary
Archive Missing Command Implementation
src/dev/subcommands/archive_missing_cmd.rs
New module implementing archive snapshot verification with height extraction via regex, JSON listing parsing, epoch range computation for lite (every 30,000 epochs) and diff (every 3,000 epochs within lite segments) snapshots, HTTP client-based remote fetching, comparison logic to detect missing snapshots, and comprehensive unit tests covering height extraction, epoch computation, JSON parsing, and end-to-end scenarios.
Command Registration
src/dev/subcommands/mod.rs
Wires new ArchiveMissing variant into Subcommand enum and routes execution to the command's async run() method.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • feat: snapshot calculator #6715: Implements the same core functionality for computing required lite/diff snapshots and extracting archive listing heights to determine snapshot availability.

Suggested reviewers

  • akaladarshi
  • hanabi1224
🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The PR title 'feat(dev): add missing snapshots checker' directly and clearly summarizes the main change: introducing a new command to check for missing archive snapshots. It is concise, specific, and accurately reflects the primary feature added.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch check-missing-snaps-dev-cmd
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch check-missing-snaps-dev-cmd
📝 Coding Plan
  • Generate coding plan for human review comments

Comment @coderabbitai help to get the list of available commands and usage tips.

@LesnyRumcajs LesnyRumcajs enabled auto-merge March 13, 2026 15:45
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/dev/subcommands/archive_missing_cmd.rs (1)

54-59: Consider using expect() instead of unwrap() for consistency with coding guidelines.

While this regex pattern is a compile-time constant that will always succeed, using expect() with a descriptive message is preferred per the project's coding guidelines.

Suggested change
 fn extract_height(url: &Url) -> Option<ChainEpoch> {
-    static RE: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"_height_(\d+)").unwrap());
+    static RE: LazyLock<Regex> =
+        LazyLock::new(|| Regex::new(r"_height_(\d+)").expect("valid regex pattern"));
     let path = url.path();
     let caps = RE.captures(path)?;
     caps[1].parse().ok()
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/dev/subcommands/archive_missing_cmd.rs` around lines 54 - 59, The regex
uses Regex::new(...).unwrap() in extract_height which violates the guideline;
replace the unwrap() with expect(...) supplying a short descriptive message
(e.g., "invalid regex in extract_height") when initializing the static RE so the
Regex::new call still panics on programmer error but with a clear message; keep
the rest of extract_height (path, captures, parse) unchanged and reference the
static RE and function extract_height when applying the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/dev/subcommands/archive_missing_cmd.rs`:
- Around line 54-59: The regex uses Regex::new(...).unwrap() in extract_height
which violates the guideline; replace the unwrap() with expect(...) supplying a
short descriptive message (e.g., "invalid regex in extract_height") when
initializing the static RE so the Regex::new call still panics on programmer
error but with a clear message; keep the rest of extract_height (path, captures,
parse) unchanged and reference the static RE and function extract_height when
applying the change.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 52bf56a5-b4a5-4912-a9e4-1b5433f8c7fb

📥 Commits

Reviewing files that changed from the base of the PR and between bb374bb and 8a39918.

📒 Files selected for processing (2)
  • src/dev/subcommands/archive_missing_cmd.rs
  • src/dev/subcommands/mod.rs

@LesnyRumcajs LesnyRumcajs added this pull request to the merge queue Mar 13, 2026
@codecov
Copy link

codecov bot commented Mar 13, 2026

Codecov Report

❌ Patch coverage is 69.63563% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.50%. Comparing base (bb374bb) to head (8a39918).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/dev/subcommands/archive_missing_cmd.rs 69.91% 74 Missing ⚠️
src/dev/subcommands/mod.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
src/dev/subcommands/mod.rs 24.24% <0.00%> (-0.38%) ⬇️
src/dev/subcommands/archive_missing_cmd.rs 69.91% <69.91%> (ø)

... and 15 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb374bb...8a39918. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Merged via the queue into main with commit 990ff29 Mar 13, 2026
35 of 36 checks passed
@LesnyRumcajs LesnyRumcajs deleted the check-missing-snaps-dev-cmd branch March 13, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants