check: does --max-duration really need --repository-only?

Follow-up from #9925 (this thread: #9925 (comment)).

Right now borg check --max-duration requires --repository-only. One of the old reasons was that the archives check needed the repository phase to run first so it had a complete repository index to work from.

That is not the case in borg2 anymore. ArchiveChecker.check builds its own chunk index with build_chunkindex_from_repo(...) in src/borg/archive.py, so it does not depend on the repository check having run.

The one reason that still stands is that the archives check has no max_duration support, so a time-boxed run can only really cover the repository pack check.

So the open question is whether we keep the requirement (just documented) or drop it and add time-boxing to the archives check too. Filing this so it does not get lost.

In a similar way as it remembers which packs it has checked when and what the outcome was, it could also remember this for the check archives. Needs some benchmarking first, whether checking an archive takes long enough so this is worthwhile doing.

I benchmarked this. For normal-sized archives a single archive's check is cheap, and --verify-data wouldn't benefit from per-archive tracking at all as it's currently written.

Setup: borg 2.0.0b23 (current master), local repo on ext4, aes256-ocb + sha256. Archives phase isolated with --archives-only; each number is the min of 3 runs.

Sensitivity to archive data size (repo ~1.4 GiB, 4 archives):

archive	meta (s)	verify-data (s)
1 MiB	3.71	9.98
256 MiB	3.39	9.81
1 GiB	3.62	9.84
20k files	4.04	10.50

Sensitivity to item count (metadata check, tiny files):

files	meta (s)	marginal (s)	ms / 1k files
1	11.63	—	—
10k	11.89	+0.26	26
50k	12.35	+0.72	14
100k	14.02	+2.39	24
200k	16.25	+4.62	23

The baseline differs between the two tables because the fixed cost scales with total repo item count, not with the selected archive.

Three things stand out:

Most of the archives-check time is a fixed per-invocation cost: building the chunk index from the repo. A per-archive resume doesn't save this, it's paid once per borg check no matter what -a selects.
--verify-data is effectively repo-global. The time stayed ~10s whether the selected archive was 1 MiB or 1 GiB, same as checking the whole repo. Resume tracking can't help that phase unless it's first reworked to verify per-archive.
The one thing that scales per archive is item count in the metadata check, roughly linear at ~23 ms per 1000 files. That extrapolates to ~23s at 1M files and ~2min at 5M files.

So the tracking pays off only for archives with millions of items. Below that, the fixed index-build cost swamps the per-archive part, and simple time-boxing of the archives check (with that fixed cost counted against the budget) would cover it.

~23s at 1M files

At 1000 archives, this would be 23.000s = 6h, so we are getting into "it's worth it" territory there.

If there are more files per archive, it happens even earlier than 1000 archives.

Re-ran this on current master (a586f69). My 08-03 numbers are out of date: since 122fa47 the archives check uses the index that is already loaded instead of rebuilding it from the packs. The fixed per-run cost went from ~11s to ~0.5s, so the per-archive part now dominates.

aes256-ocb, local repo, --archives-only, 3 runs each:

repo	time
1 archive, 1 file	0.55s
1 archive, 100k files	2.3s
10 archives × 100k files	14.3s

On the 10-archive repo a full check takes 14.5s and --repository-only 1.2s, so the archives phase is ~90% of the run. That's ~13.5 ms per 1000 files per archive, so 1000 archives of 1M files would be ~3.75h, the same range as your 6h estimate.

--verify-data is still repo-wide (it walks iter_packs()), so tracking wouldn't help it.

What I'd do:

cache/checked-archives, same layout as cache/checked-packs (archive id -> timestamp, result), pruned to the ids still in archives/.
Without --repair, skip an archive when its record is ok and younger than --max-age. --repair and --verify-data never skip.
With --max-duration, check the least recently checked archives first and stop between archives, never in the middle of one. Then --max-duration no longer needs --repository-only, and --archives-only --max-age can be allowed.

Two things I'd like your opinion on before I start:

A clean record only means the archive's chunks were all there when it was checked. If a pack gets recorded corrupt later, or a repair drops objects, that record can hide a missing chunk until it expires. I'd clear all archive records in those cases. Or is --max-age alone good enough here, like for packs?
One --max-duration budget shared by both phases, or a separate one per phase?

I'd stack this on #10464, since it touches the same argument checks and the tracker code.

Guess clearing checked-archives cache makes sense if corrupted packs are detected/repaired.

One --max-duration budget for both. Guess users will adjust start + max-duration to not run into business work times

check: --max-age and --max-duration cover the archives check, fixes b…

check tests: assert the changed archive ids only without borg.exe, refs

check: set the complete flag in the item loop's else clause, refs bor…

repository: check trackers warn once per run if the store denies the …

repository: split CheckTracker out of PackTracker, refs #10025

Merge pull request borgbackup#10504 from mr-raj12/check-archives-trac…

Navigation Menu

Repository navigation

check: does --max-duration really need --repository-only? #10025

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

check: does --max-duration really need --repository-only? #10025

Description

Activity

ThomasWaldmann commented on Aug 3, 2026

mr-raj12 commented on Aug 3, 2026

ThomasWaldmann commented on Aug 4, 2026

mr-raj12 commented on Sep 30, 2026

ThomasWaldmann commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions