iocore/net: add RFC 7250 raw public key (RPK) support for TLS hops

[maskit]

Summary

Layered-cache deployments have TLS connections between nodes under the same operator's control, where the CA/hostname-verification machinery that X.509 exists for isn't needed -- a raw public key, pinned per hop, is sufficient. RPK is negotiated alongside X.509 rather than replacing it, so a hop between nodes at different points in a rolling upgrade falls back to a normal certificate exchange rather than failing.

Adds build-time detection of OpenSSL 3.2+'s SSL_CTX_set1_server_cert_type and BoringSSL's SSL_CREDENTIAL_new_raw_public_key, and a library-agnostic SSLRPKUtils helper for loading and pinning trusted keys.

New settings:

• ssl_multicert.yaml: ssl_rpk_enabled, ssl_client_rpk_ca_name
• sni.yaml: client_rpk_enabled, server_rpk_ca

On BoringSSL, accepting an RPK client cert requires switching to SSL_CTX_set_custom_verify, which disables automatic X.509 chain verification for the whole connection, so the X.509 fallback path is reimplemented manually there on both client and server sides.

Test plan

• New unit tests (test_SSLRPKUtils.cc, test_YamlSNIConfig.cc) covering key loading/pinning and sni.yaml parsing
• New autest (tls_rpk_hop.test.py) covering RPK negotiation, X.509 fallback during a rolling upgrade, pin mismatch under ENFORCED/PERMISSIVE, and mTLS pinning via ssl_client_rpk_ca_name
• Verified on both OpenSSL 3.2+ and BoringSSL builds

[Copilot]

Pull request overview

This PR adds RFC 7250 Raw Public Key (RPK) negotiation/pinning support to ATS’s TLS hop logic (layered-cache / ATS-to-ATS use cases), with build-time capability detection for OpenSSL vs. BoringSSL and accompanying unit + AuTest coverage.

Changes:

• Add library-agnostic SSLRPKUtils for loading/pinning trusted raw public keys and integrate it into inbound mTLS and outbound next-hop verification paths.
• Extend ssl_multicert.yaml and sni.yaml parsing/config plumbing to enable RPK offering and key pinning.
• Add unit tests and a new gold test for RPK negotiation, fallback to X.509, and pin mismatch behaviors.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/gold_tests/tls/tls_rpk_hop.test.py	New gold test exercising RPK hop negotiation, fallback, and pin mismatch behaviors.
tests/gold_tests/tls/ssl/server.pubkey.pem	Test fixture: pinned public key for RPK hop.
tests/gold_tests/tls/ssl/server.wrongpubkey.pem	Test fixture: mismatching public key for negative pin tests.
src/traffic_layout/info.cc	Expose TS_USE_RPK in traffic_layout feature output for runtime feature gating.
src/iocore/net/YamlSNIConfig.cc	Parse new client_rpk_enabled / server_rpk_ca SNI YAML keys (with no-RPK build warning).
src/iocore/net/unit_tests/test_YamlSNIConfig.cc	Unit test coverage for new SNI YAML fields.
src/iocore/net/unit_tests/test_SSLRPKUtils.cc	New unit tests for key loading and pin matching logic.
src/iocore/net/unit_tests/sni_conf_test.yaml	Add SNI YAML fixture entry for RPK parsing test.
src/iocore/net/unit_tests/rpk_single.pem	Unit test fixture: single trusted key.
src/iocore/net/unit_tests/rpk_other.pem	Unit test fixture: alternate key.
src/iocore/net/unit_tests/rpk_multi.pem	Unit test fixture: multi-key rotation set.
src/iocore/net/unit_tests/rpk_malformed.pem	Unit test fixture: malformed key file.
src/iocore/net/SSLUtils.cc	Inbound-side RPK client-auth handling; BoringSSL custom-verify path with X.509 fallback rebuild.
src/iocore/net/SSLSNIConfig.cc	Plumb outbound RPK config from parsed SNI items; probe-load pinned key files during config load.
src/iocore/net/SSLRPKUtils.h	New helper API for loading/pinning raw public keys.
src/iocore/net/SSLRPKUtils.cc	New implementation for PEM key loading and DER pin comparisons.
src/iocore/net/SSLNetVConnection.cc	Apply outbound RPK setup during TLS handshake when next hop is configured for RPK.
src/iocore/net/SSLClientUtils.cc	Outbound-side RPK verification + BoringSSL custom-verify callback for RPK/X.509 fallback.
src/iocore/net/P_SSLClientUtils.h	Declare ssl_client_setup_rpk() helper for outbound connections.
src/iocore/net/P_SSLCertLookup.h	Extend multicert settings with rpk_enabled and client_rpk_ca wiring.
src/iocore/net/CMakeLists.txt	Build/link SSLRPKUtils and its unit test.
src/config/ssl_multicert.cc	Parse/emit new ssl_rpk_enabled and ssl_client_rpk_ca_name fields (YAML + legacy).
include/tscore/ink_config.h.cmake.in	Add TS_USE_RPK and feature-detection macros for RPK-related APIs.
include/iocore/net/YamlSNIConfig.h	Add client_rpk_enabled / server_rpk_ca to the parsed config model.
include/iocore/net/SSLSNIConfig.h	Add next-hop properties for outbound RPK offering and pin file path.
include/config/ssl_multicert.h	Add multicert entry fields for RPK enabling and client RPK CA file name.
doc/admin-guide/files/ssl_multicert.yaml.en.rst	Document ssl_rpk_enabled and ssl_client_rpk_ca_name.
doc/admin-guide/files/sni.yaml.en.rst	Document client_rpk_enabled and server_rpk_ca semantics and fallback behavior.
CMakeLists.txt	Detect RPK-capable TLS APIs and define TS_USE_RPK.

Suppressed comments (1)

tests/gold_tests/tls/tls_rpk_hop.test.py:199

• This TestRun also relies on parent_rpk but doesn't start it; if parent_rpk was stopped after the prior run, this run will fail to connect upstream.

tr.Processes.Default.StartBefore(edge_badpin_permissive)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

src/iocore/net/SSLSNIConfig.cc:207

• server_rpk_ca key probing is performed unconditionally. On builds without TS_USE_RPK, this can make sni.yaml loading fail (returning false) if the pinned key file is unreadable, even though the parser explicitly warns that these fields “do not apply” and should be ignored. Gate this probe-load behind #if TS_USE_RPK so non-RPK builds only warn and continue.

  if (!item.server_rpk_ca.empty()) {
    SSLConfig::scoped_config params;
    nps.prop.server_rpk_ca_file = Layout::get()->relative_to(params->clientCACertPath, item.server_rpk_ca.data());
    // Fail the config load now rather than at handshake time if the pinned keys are unreadable.
    SSLRPKUtils::TrustedKeySet probe;

src/iocore/net/SSLClientUtils.cc:507

• ssl_client_setup_rpk() loads and parses the trusted RPK PEM file on every outbound TLS connection handshake. In layered-cache deployments with frequent connects, this introduces repeated disk I/O and PEM parsing overhead that could be avoided by caching the parsed TrustedKeySet by path (e.g., in SNIConfigParams/NextHopProperty as a shared_ptr, and attaching a non-owning pointer per-SSL).

  if (!trusted_key_file.empty()) {
    auto *trusted = new SSLRPKUtils::TrustedKeySet();
    if (!SSLRPKUtils::loadTrustedKeys(trusted_key_file.c_str(), *trusted)) {
      delete trusted;
      return false;
    }

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/iocore/net/SSLUtils.cc:2621

• SSLMultiCertConfigLoader::load_certs allocates a TrustedKeySet and attaches it via SSL_CTX_set_ex_data(), but the return value is not checked. If SSL_CTX_set_ex_data fails, the TrustedKeySet leaks and later verification will behave as if no trusted keys were configured.

    // ssl_client_rpk_ca_ex_free() releases `trusted` when ctx is freed.
    SSL_CTX_set_ex_data(ctx, ssl_client_rpk_ca_index, trusted);

tests/gold_tests/tls/tls_rpk_hop.test.py:179

• In the second TestRun, parent_rpk is not included in StillRunningAfter. AuTest will typically stop processes that are not listed, so later runs that depend on parent_rpk (pin-mismatch cases) may fail because parent_rpk was stopped after this run.

tr.StillRunningAfter = server
tr.StillRunningAfter += parent_x509
tr.StillRunningAfter += edge_fallback

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/iocore/net/SSLRPKUtils.cc:99

• loadTrustedKeys() accepts a PUBLIC KEY PEM even if the DER payload has trailing bytes (d2i_PUBKEY parses a prefix and ignores the rest). In that case the stored trusted key includes the trailing bytes and will never match the peer key DER produced by i2d_PUBKEY(), causing confusing pin mismatches. Treat trailing data as a parse failure (and free the partially-parsed key).

      const unsigned char *p = data;
      // The PEM payload for a PUBLIC KEY block is already a DER SubjectPublicKeyInfo, which is
      // exactly what gets pinned -- but decode it anyway so a corrupt key is rejected at config
      // load rather than silently pinned as opaque bytes.
      pkey = d2i_PUBKEY(nullptr, &p, len);
      if (pkey == nullptr) {
        SSLError("SSLRPKUtils: failed to parse a raw public key from %s", path);

include/iocore/net/SSLSNIConfig.h:56

• The new NextHopProperty member has inconsistent spacing (std::string server_rpk_ca_file;) compared to the surrounding declarations; this is likely to trip clang-format/formatting checks.

  bool        client_rpk_enabled = false;     // offer a RFC 7250 raw public key (derived from the configured client
                                              // cert/key) alongside X.509 when connecting to this next hop
  std::string             server_rpk_ca_file; // full path to the PEM of trusted next-hop raw public keys to pin against

src/iocore/net/P_SSLCertLookup.h:73

• The new members in SSLMultiCertConfigParams have inconsistent alignment/spacing (notably ats_scoped_str client_rpk_ca;) and the trailing comment line is overly long; aligning/wrapping here will match the style used throughout this header and avoid format-check churn.

  bool           rpk_enabled = false;    ///< Offer RFC 7250 raw public keys (using this entry's existing cert/key as the
                                         ///< identity) alongside X.509, negotiated per connection
  ats_scoped_str       client_rpk_ca; ///< Trusted client raw public keys (PEM, may contain more than one) for inbound mTLS pinning
  SSLCertContextOption opt;           ///< SSLCertContext special handling option