`borg2 repo-list|transfer --from-borg1`: Unknown algorithm 'XXH64'

[PhrozenByte]

While testing 2.0.0b22, both borg2 repo-list --from-borg1 and borg2 transfer --from-borg1 emit the following warning with a Borg 1 repo:

Cannot verify integrity of /run/media/daniel/prometheus/Borg/index.1597: Unknown algorithm 'XXH64'

xxh64 was removed in favor of sha256 in #9672, but existing Borg 1 repositories naturally still use the legacy algorithm.

Is this warning expected and safe to ignore when accessing a Borg 1 repository via --from-borg1?

If so, would it make sense to downgrade this message from warning to info/debug (perhaps only when --from-borg1 is used), since users cannot do anything about it?

[PhrozenByte]

Additional question: does using sha256 for this have any noticeable performance impact when the borg2 repo was intentionally created with --id-hash blake3 due to missing SHA hardware acceleration, or are these hashes only computed infrequently anyway?

[ThomasWaldmann]

Oh, crap. Didn't think of legacy integrity configs still using xxh64.

If it just logs a warning, but continues, you can ignore it. It is just trying to check the integrity of the files it loads, but can't do that. That doesn't mean the file is bad.

When users switch from borg 1.x to borg2, we can assume that the files had valid integrity while they were used with borg1, so it is not absolutely required for borg2 to check it again.

Alternatively, I could add back the xxh64 dependency, so the check can work again.

[ThomasWaldmann]

does using sha256 for this have any noticeable performance impact when the borg2 repo was intentionally created with --id-hash blake3 due to missing SHA hardware acceleration?

The code that threw this exception is integrity-checking file data when borg loads these files. It is used for some important index and cache files (esp. borg1 stuff), but NOT for the borg2 packs or borg2 index.

The borg2 packs and also the index use sha256 though - the storage key (name) of these objects is always sha256(content). That makes these objects checkable on the server side (REST server) and using content-addressing also gives nice caching properties.

The decision for sha256-only for this application was made due to widespread hw acceleration support (even some microcontrollers have hw for this now), algorithm stability, algorithm support elsewhere.

Verifying a pack against accidental corruption can be even done without borg, just sha256sum the pack. If the result is the same as the pack name, the pack contents are fine.

[PhrozenByte]

The borg2 packs and also the index use sha256 though - the storage key (name) of these objects is always sha256(content). […]
The decision for sha256-only for this application was made due to widespread hw acceleration support […]
Verifying a pack against accidental corruption can be even done without borg, just sha256sum the pack. […]

Hmm… 🤔 Wouldn't the same also be possible with Blake3?

SHA hardware acceleration is certainly widespread, but not universal (note to myself: since #9168/#9779 has been implemented now, I should backport #9103 to master). Intel, for example, only introduced SHA extensions in 2021, so there is still a fair amount of hardware in active use without SHA hardware acceleration.

Do you have an estimate of how much performance impact this has on systems without SHA hardware acceleration? From my understanding, computing the pack IDs could become a noticeable part of the workload, perhaps loosely comparable to --id-hash, albeit somewhat less significant? I'm currently running borg2 transfer --from-borg1 on an Intel Core i5-8265U (8th gen, i.e. without SHA hardware acceleration), and the transfer is indeed quite slow. I was planning to open a separate issue about this anyway, but I haven't had the time yet to benchmark it properly and isolate the issue.

Regarding interoperability, it's true that GNU coreutils doesn't currently ship a b3sum utility (yet? it does include b2sum for Blake2b). However, there is an official b3sum tool from the Blake3 project itself, and it's available in recent versions of all major Linux distributions.

[ThomasWaldmann]

Yes, any 256bit digest (of a cryptographic hash algorithm) would work in principle.

The performance impact very much depends on other factors. I recently tried backing up to google drive and it was so slow that I am quite sure that hashing performance does not make a noticeable difference for that (and similar cloud storage).

For a fast storage (borgstore posixfs == local filesystem) it would need some benchmarking (borg create and borg check). borg check is still WIP for borg2, the main idea for that is to optimise by clever algorithm.

Move this to a separate ticket?

borg transfer performance: depends a lot on what transfer has to do.

In general, it has more to do when migrating from borg1 because it needs to upgrade/clean the metadata. In some scenarios, it also has to do a full decrypt-decompress-rechunk-rehash-compress-encrypt dance (e.g. when going from blake2b to blake3). That's not fast, but you only have to do it once.

When doing transfers between related borg2 repos in future, it should be faster, because it can transfer the metadata and chunks as they are and only needs to decrypt/re-encrypt them.

[PhrozenByte]

Move this to a separate ticket?

Absolutely. See #9943. Thanks for the explanation so far! 👍