check --repair: validate the repository index rebuild with the key, #9901

[mr-raj12]

Follow-up to #9901. The repository-level index rebuild in Repository.check() was the only rebuild that walked the pack object headers without a validator, because Repository had no key. It relied on the sha256 pack check done before it, which catches accidental damage but is not authentication. As discussed in #9901, the repository code always runs on the client in borg2, so it can use the key.

Changes

• Repository.check() takes a new validate argument and passes it to build_chunkindex_from_repo() for the repair rebuild, together with an on_drop counter.
• An object that fails validation is not indexed. Each pack byte range the rebuild skips counts as an error and is logged. In repair mode this fails a repository-only run and only gives a warning on a full check, the same as a corrupt pack, because the archives phase rebuilds the index again with the validator and reports the chunks that are missing.
• With validate=None the rebuild behaves as before.
• borg check --repair builds the validator with object_validator(RepoObj(key)):
  • A full check uses the key that do_check already reads before the repository check.
  • --repository-only --repair reads the key from the manifest only. It does not read chunk objects to identify the key type, because it finds those through the chunk index, which may be the corrupt part.
  • If the key cannot be read (no manifest, key not found, invalid key), a warning is logged and the rebuild runs without validation. A wrong passphrase still raises.
• The repair mode section of the borg check help is updated.

Behaviour change

borg check --repository-only --repair now asks for the passphrase. A script that sets only BORG_CHECK_I_KNOW_WHAT_I_AM_DOING and no BORG_PASSPHRASE will now stop at the passphrase prompt. The key is read before the pack verification starts, so the user is not prompted after a long run. It is read even when the index turns out to be intact and no rebuild is needed. borg check --repository-only without --repair does not read the key.

Tests

• repository_test.py::test_check_repair_rebuild_validates_objects (repository-only and full): a validator rejects one object; the rebuilt index leaves it out, every other object is still readable, and the return value matches the mode.
• check_cmd_test.py::test_check_repository_only_repair_validates_index_rebuild: an encrypted repository with a byte flipped in one object's metadata slot, the pack stored under the sha256 of its new content so it still passes the pack check, and a corrupt index. check --repository-only --repair exits with 1 and the object is not indexed.
• check_cmd_test.py::test_check_repository_only_repair_without_key: the same repository with the key made unreadable. The command warns, exits with 0, and indexes the changed object.

pytest -n auto -k "not remote and not binary": 2869 passed, 531 skipped.

Fixes #9901.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.99%. Comparing base (bd7282c) to head (5289af9).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10369      +/-   ##
==========================================
- Coverage   88.06%   87.99%   -0.07%     
==========================================
  Files         103      103              
  Lines       18913    18934      +21     
  Branches     2919     2923       +4     
==========================================
+ Hits        16655    16661       +6     
- Misses       1565     1579      +14     
- Partials      693      694       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

Guess it would be better to always work with the key and not fall back to keyless.

If the passphrase is wrong, key is missing, etc. just abort.