Validate recovery snapshot endorsement chains in memory

[achamayou]

Summary

Implements the recovery-snapshot endorsement-chain design from Discussion #8080, simplified in response to Eddy Ashton’s review: the derived chain is retained only in memory for the current recovering node. No endorsement sidecar is created, read, cached, or reused.

• Applies only to StartType::Recover when a local startup snapshot is selected.
• Before snapshot KV deserialisation and before the node becomes part of the public network, scans configured ledger files directly, without ringbuffer requests or a temporary KV store.
• Extracts bounded previous-service-identity endorsement records from complete public-domain ledger entries after the snapshot.
• Accepts a snapshot directly valid under the operator-provided recover.previous_service_identity; otherwise requires a non-empty complete newest-to-oldest chain from that trusted identity to the snapshot signer.
• Passes the validated chain separately into snapshot receipt verification, retains it only for this recovery attempt, then discards it.
• Leaves snapshot and ledger files unchanged. Invalid, incomplete, injected, out-of-order, epoch-mismatched, or oversized candidate material rejects the snapshot and falls back to full-ledger recovery.
• Keeps snapshot installation outside verification catches, so deserialisation/startup failures propagate rather than replaying against a partially mutated Store.

Trust model

config.recover.previous_service_identity is the sole target trust anchor In.

The ledger suffix is only an untrusted transport for candidate CoseEndorsement records. Snapshot acceptance does not rely on suffix service state, signature transactions, ledger inclusion proofs, or serialized Merkle trees. Instead, the complete candidate list must validate cryptographically:

• every candidate write is after snapshot seqno S;
• the oldest candidate points to an endorsement at or before S;
• each later previous_version points exactly to the preceding candidate write;
• signed COSE epoch headers match the record metadata;
• adjacent epoch ranges are contiguous;
• each COSE signature endorses the preceding service public key;
• the newest signature key equals trusted operator input In; and
• the oldest endorsed range covers S.

The resulting oldest key must then verify the snapshot COSE receipt, including its claims digest, inclusion proof/Merkle root, and detached signature. Tampered or injected suffix records cannot become trusted independently; they break the complete anchored chain and force fallback.

JSON receipt augmentation remains intentionally out of scope. JSON snapshots are accepted only when directly valid under the configured identity.

Tests

• Focused chain tests cover newest-to-oldest ordering, tampering, missing links, wrong In, broken back-pointers, signed epoch metadata mismatch, injected candidates, adjacency, and snapshot coverage.
• Direct-file scanner tests prove endorsement candidates are collected without any following signature transaction, enforce candidate count/item/aggregate and raw-entry bounds, and show malformed candidates fail before snapshot installation.
• Installation-exception coverage proves verification failures may fall back while snapshot deserialisation/install failures propagate without fallback.
• recovery_snapshot_endorsements_test performs two disaster recoveries, recovers from the original stale snapshot through the in-memory two-link chain, checks verification occurs before snapshot deserialisation/replay, hashes the actual node-local snapshot bytes, and verifies a truncated suffix rejects the snapshot and falls back to full replay.
• Built ccf, the logging application, snapshotter_test, and network_identity_subsystem_test; focused unit/e2e tests, clang-tidy, and scripts/ci-checks.sh pass.

Limitations

• Old-style/JSON receipt augmentation is not implemented, matching Discussion Implementation Spec: Augmenting recovery-snapshot receipts with cross-recovery service-identity endorsements #8080.
• The pre-pass extracts only public endorsement candidates; it does not replay or construct application state, or establish ledger commitment for those candidates. Trust comes solely from the complete COSE chain anchored at In.

[Copilot]

Pull request overview

This PR implements recovery snapshot endorsement “sidecars” to allow recovery to start from a snapshot whose COSE receipt was signed by an older service identity, by deriving/verifying a contiguous endorsement chain anchored at the operator-provided recover.previous_service_identity and caching it alongside the snapshot.

Changes:

• Adds sidecar path derivation and lifecycle support (including cleanup when snapshots are deleted).
• Implements synchronous scanning of the public ledger suffix (without building/applying a KV) to collect and validate previous-service-identity COSE endorsements and persist them as a CBOR sidecar.
• Extends Python snapshot/COSE tooling and adds focused C++/Python/e2e tests, plus CI wiring for the new e2e test.

Custom instructions used:

• .github/copilot-instructions.md
• .github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/recovery_snapshot_endorsements.py	New e2e coverage for sidecar derivation, reuse, tamper handling, and fallback.
tests/ci-buckets.txt	Adds the new e2e test to bucket_b.
src/snapshots/filenames.h	Adds helper to derive snapshot endorsement sidecar path.
src/node/test/snapshotter.cpp	Unit tests for sidecar lifecycle and direct ledger-file scanning behavior.
src/node/test/network_identity_subsystem.cpp	Unit tests for newest-to-oldest endorsement validation and fail-closed behavior.
src/node/snapshot_serdes.h	Adds CBOR (de)serialisation for endorsements, sidecar IO, and recovery snapshot verification using endorsement chains.
src/node/rpc/network_identity_subsystem.h	Refactors endorsement signature verification to use shared helper.
src/node/rpc/network_identity_chain_helpers.h	Introduces shared endorsement parsing/validation helpers used by recovery and identity subsystems.
src/node/recovery_snapshot_ledger.h	Adds direct ledger-file scanner for service info + endorsement collection after a snapshot.
src/node/node_state.h	Integrates sidecar verification/derivation into recovery startup flow and adds sidecar cleanup/fallback logic.
src/host/test/files_cleanup_test.cpp	Verifies snapshot cleanup also removes adjacent endorsement sidecars.
src/host/files_cleanup_timer.h	Deletes endorsement sidecars when deleting old snapshots.
python/tests/test_cose_sign.py	Adds Python tests for endorsement sidecar chain verification and strict CBOR decoding.
python/src/ccf/read_ledger.py	Adds CLI options to verify a snapshot receipt against a target cert + optional endorsements sidecar.
python/src/ccf/ledger.py	Adds sidecar path helper and reader; snapshot API caches receipt/digest for verification.
python/src/ccf/cose.py	Adds CBOR encoding/decoding for endorsements and endorsement-chain verification utilities.
CMakeLists.txt	Registers the new e2e test target.

[eddyashton]

I don't think this should be stored in a sidecar file. The endorsement chain can be retained in memory by the current node, and used by that node.

I accept there's significant work in creating this endorsement chain, that we could save if a future recovery attempt is able to use this file, but believe that case will be exceedingly rare. We eagerly create snapshots on successful recovery, so falling into this in the first place is unusual. If we do serial recoveries with no fresher snapshots, its likely that each attempt is actually using the next identity, so this off-by-one endorsement chain doesn't help us (an endorsement chain to Identity N won't help us at N +1, its equivalent to junk because we jump to a crypto check, there's no attempt to augment an existing partial endorsement sidecar with newer certs. And good, that's even more complex!).

Fundamentally, this is a rare-enough path that we should prefer it being as branchless (simple) as possible. The real goal is to read the ledger and recover from the valid snapshot+suffix, but much of the code in this PR is concerned with managing sidecar files and testing that we can correctly ignore corrupt ones - totally unnecessary work.

Discussing this with the robot, it's raised the point that the sidecar file can be reused by other tools, eg read_ledger.py. But I think that's slightly artificial - when the file is there, it looks like good coverage for all of our tooling to use it (including duplicating path construction logic etc). Before the file existed, the tooling took no snapshot signer arg, and simply didn't care about this. I think that's better! There's already a big difference between what the Python tooling does and what the node does (snapshot discovery!), I don't think this is a useful extra bit of verification.

Big rant! TL;DR: Dump the file, store the chain in-memory for the current node, delete half of this PR.