feat: IC-1943 Include refund shares in HTTP outcalls payload

[eichhorl]

Background

Currently, the price for an HTTP outcall is calculated and charged upfront. In the future, we want to implement a "pay-as-you-go" pricing model. This model assigns a fraction (budget) of the total attached cycles to each participating replica (per-replica allowance).

As the request passes through the target server and the transform function, the HTTP adapter of each replica subtracts cycles from its own budget, based on the amount of resources it consumed (download time, downloaded bytes, transform instructions). In the end, the amount of remaining cycles (refund) is passed back to the replica together with the HTTP response.

Proposed Changes

With this PR, we let the replica include (and sign) this refund as part of the gossiped response share.

Before, each replica only signed the response metadata which should be equal across all responses in order to reach consensus. Now, the signed messages may be different even across agreeing replicas, as each of them could sign a different refund share.

Therefore, we change the structure of the aggregated HTTP response proof. Previously, this was just the metadata of the response and a collection of individual signatures over it. Now, it is the metadata and a collection of individual signatures and refund receipts of each replica. In order to verify the proof, the individual messages consisting of both the metadata and the refund share have to be reconstructed, such that the signature can be verified.

For that reason we additionally switch signature verification of aggregated responses to use the new batch verification API for multiple messages, similar to what was done in #10345. As a side effect, this comes with some performance improvements for payload validation:

canister_http_payload_verification/mixed_subnet34
                        time:   [146.72 ms 148.46 ms 150.26 ms]
                        change: [-14.887% -13.555% -12.243%] (p = 0.00 < 0.05)
                        Performance has improved.
canister_http_payload_verification/many_replicated_responses_subnet34
                        time:   [225.27 ms 227.44 ms 229.63 ms]
                        change: [-17.525% -16.435% -15.417%] (p = 0.00 < 0.05)
                        Performance has improved.
canister_http_payload_verification/many_non_replicated_responses_subnet34
                        time:   [11.124 ms 11.263 ms 11.406 ms]
                        change: [-61.100% -60.382% -59.642%] (p = 0.00 < 0.05)
                        Performance has improved.
canister_http_payload_verification/many_divergence_responses_subnet34
                        time:   [116.12 ms 117.52 ms 118.94 ms]
                        change: [-4.0609% -2.4873% -0.9067%] (p = 0.00 < 0.05)
                        Change within noise threshold.
canister_http_payload_verification/many_flexible_responses_subnet34
                        time:   [253.98 ms 256.09 ms 258.22 ms]
                        change: [-3.1565% -2.0740% -0.8815%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Additionally, during payload validation, we verify that none of the refund shares exceed the per-replica allowance. Note that in the current (legacy) pricing model, nothing is refunded yet, so the allowance, and the returned refund is always 0.

[Copilot]

Pull request overview

This PR updates canister-HTTP outcall consensus artifacts so that replicas sign and gossip a per-replica payment receipt (refund share) alongside the shared response metadata, enabling future “pay-as-you-go” pricing where refunds can differ per replica. It also switches signature verification to batched multi-message verification to support non-identical signed messages and improve validation performance.

Changes:

• Introduce CanisterHttpResponseReceiptShare / CanisterHttpResponseProof to carry per-replica payment receipts in signed shares and aggregated proofs.
• Update payload building/validation to reconstruct per-signer shares from proofs, enforce refund allowance bounds, and batch-verify signatures over multiple messages.
• Extend protobuf encoding for canister-HTTP signatures to include payment_receipt and update call sites/tests/benchmarks accordingly.

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rs/types/types/src/exhaustive.rs	Adjust exhaustive-generation traits/IDs for new canister-HTTP signature/proof types.
rs/types/types/src/crypto/sign.rs	Switch signature domain typing from response metadata to receipt share.
rs/types/types/src/crypto/hash/tests.rs	Update hash-stability test to hash signed receipt-share content.
rs/types/types/src/canister_http.rs	Define receipt-share and proof structures; update share/proof type aliases accordingly.
rs/types/types/src/batch/canister_http.rs	Update protobuf conversions for receipt-share signatures and aggregated proofs with payment receipts.
rs/state_machine_tests/src/lib.rs	Update state-machine test signing to sign receipt shares instead of metadata.
rs/protobuf/src/gen/types/types.v1.rs	Regenerate protobuf bindings to include payment_receipt in signature message.
rs/protobuf/def/types/v1/canister_http.proto	Add payment_receipt field to CanisterHttpResponseSignature.
rs/interfaces/src/crypto.rs	Update crypto interface bounds to sign/verify receipt shares.
rs/interfaces/src/canister_http.rs	Add structured validation error for refunds exceeding per-replica allowance.
rs/interfaces/mocks/src/crypto.rs	Update crypto mocks to operate on receipt-share messages.
rs/https_outcalls/consensus/src/pool_manager.rs	Sign receipt shares; enforce refund allowance when validating incoming shares.
rs/https_outcalls/consensus/src/payload_builder/utils.rs	Add refund allowance check and helpers to reconstruct/aggregate receipt-based shares/proofs.
rs/https_outcalls/consensus/src/payload_builder/tests.rs	Expand tests for refund allowance violations across payload sections; adapt to new proof format.
rs/https_outcalls/consensus/src/payload_builder.rs	Build responses with new proof format; validate refunds and batch-verify multi-message signatures.
rs/https_outcalls/consensus/src/gossip.rs	Update bouncer logic to use receipt-share accessors.
rs/https_outcalls/consensus/Cargo.toml	Add cycles and consensus test-utilities deps needed for refund/receipt tests.
rs/https_outcalls/consensus/BUILD.bazel	Add Bazel deps for cycles and consensus test utilities.
rs/https_outcalls/consensus/benches/payload_validation.rs	Update benchmark assembler/signing to use receipt shares and new proof layout.
rs/consensus/utils/src/crypto.rs	Update consensus-crypto trait bounds to sign/aggregate receipt-share signatures.
rs/artifact_pool/src/canister_http_pool.rs	Update artifact pool tests to construct receipt shares with payment receipts.
Cargo.lock	Lockfile updates for newly referenced crates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pierugo-dfinity]

Do you think it would be feasible to implement hashes-in-blocks for CanisterHttp artifacts in the long term?

[eichhorl]

Do you think it would be feasible to implement hashes-in-blocks for CanisterHttp artifacts in the long term?

Unlike ingress and IDKG the CanisterHttpPayload is stored in the payload as plain bytes, which makes this a bit more annoying, but it definitely makes sense.