Add SLH-DSA support for the TLS 1.3 and DTLS 1.3 handshake

[Frauschi]

Summary

Adds SLH-DSA (SPHINCS+, FIPS 205) as an entity authentication algorithm for the TLS 1.3 and DTLS 1.3 handshake, following draft-reddy-tls-slhdsa. An SLH-DSA leaf key can now be used to sign (and a peer's SLH-DSA leaf to verify) the CertificateVerify message, for both server and mutual authentication.

All twelve parameter sets are wired into the handshake — the SHAKE and SHA2 hash families across 128/192/256 in both the f (fast) and s (small) variants — with per-parameter-set gating so a build only offers, accepts, and maps the variants actually compiled in. Test certificates and configs cover the 128f and 128s sets for both hash families.

Motivation

SLH-DSA is a stateless hash-based signature scheme standardized in FIPS 205. Its appeal is a conservative security foundation (hash functions only), but its signatures are large — from ~7.8 KB (128s) up to ~49 KB (256f) — which stresses several fixed-size buffers in the TLS/cert code that were sized for classical signatures. This PR adds the protocol support and the buffer-sizing work needed to make it viable, including on memory-constrained targets.

What's included

Handshake integration

• Map the SLH-DSA signature schemes to/from the wire in the signature_algorithms extension and CertificateVerify.
• Sign/verify CertificateVerify with an SLH-DSA entity key.
• Load SLH-DSA private keys and certificates and resolve their key/param types (ssl_load.c, ssl.c, ssl_api_pk.c, asn.c, wc_slhdsa.h).
• Advertisement, mapping, and OID handling are gated per parameter set, so partial builds (e.g. --enable-slhdsa=sha2-128s) never advertise or map a variant they cannot perform.

Streamed CertificateVerify send (large signatures)

When the CertificateVerify body exceeds a single record, the signature is generated once into a connection-level buffer and emitted one record at a time, so the output buffer never has to hold the whole signature.

• Resumes correctly across a non-blocking WANT_WRITE without recomputing the randomized signature (the already-sent records are bound into the transcript, so the resumed send must emit the identical bytes — this is a correctness requirement, not an optimization).
• Gated by WOLFSSL_TLS13_STREAM_CERT_VERIFY (TLS 1.3, non-async, PQC signatures). It is algorithm-neutral — it also applies to ML-DSA (or any scheme) whenever the body exceeds the negotiated max_fragment_length.
• DTLS 1.3 and WOLFSSL_ASYNC_CRYPT builds keep the existing in-place fragmented path (see Scope / carve-outs).
• Dual-algorithm (WOLFSSL_DUAL_ALG_CERTS, BOTH) bodies stream as well: the combined two-signature body is sized from per-signature upper bounds and the exact length is recorded after signing (trailing slack is never sent).

Peak send-side memory is ≈ one signature + one record (the record is at most 16 KB, or the negotiated max_fragment_length if smaller), versus the ~2× a naive in-place send holds (the assembled body plus a copy). The reduction is largest exactly where it matters for constrained devices — a small max_fragment_length, where the second term is sub-KB and the peak is essentially the signature itself. It also stops the output buffer from staying inflated to the whole signature for the connection's lifetime.

Buffer sizing

• MAX_X509_SIZE is now parameter-aware for post-quantum certificates, sized from the largest enabled SLH-DSA/ML-DSA signature. The previous flat 8 KB could not hold an SLH-DSA-signed certificate (~8.5 KB), which silently broke the static session-certificate cache paths.
• The CertificateVerify and PreTBS buffers are sized from the actual signature length instead of the worst-case WC_MAX_CERT_VERIFY_SZ (which balloons to ~50 KB with SLH-DSA). WC_MAX_CERT_VERIFY_SZ is retained for API/backward compatibility and documented as no longer used internally.

Tests and certificates

• SLH-DSA entity (client/server) certificates for the SHAKE and SHA2 128f/128s parameter sets, plus generation-script updates.
• TLS 1.3 and DTLS 1.3 entity-cert CertificateVerify test configs covering the fragmented (128f) and single-record (128s) send paths for both hash families, wired into suites.c (server-auth + mutual-auth scenarios).

Enabling

./configure --enable-slhdsa            # all parameter sets (SHAKE)
./configure --enable-slhdsa=sha2       # SHA2 family
./configure --enable-slhdsa=128s,128f  # specific parameter sets

TLS 1.3 is required (enabled by default). The SHA2 family additionally requires SHA-256, SHA-512, and HMAC.

Scope / carve-outs

• DTLS 1.3 is intentionally not on the streaming path. DTLS fragments at the handshake layer (offset/length) via Dtls13HandshakeSend, and its reliability model must retain the full message for retransmission until ACKed — so streaming yields no memory benefit there. A code comment in Dtls13SendFragmented records the follow-up (sign directly into the fragments buffer to avoid the transient second copy; the retransmit copy is a hard floor).
• WOLFSSL_ASYNC_CRYPT builds keep the existing in-place fragmented path. Async introduces a record-AEAD WC_PENDING_E resume point whose cursor-timing requirement conflicts with the non-blocking WANT_WRITE cursor; the in-place path already handles both correctly. Async targets (HW-offload servers) are not memory-constrained, so there is no beneficiary for the added complexity.

Testing

• Full ./tests/unit.test suite: pass (normal handshakes + all SLH-DSA configs + DTLS + crypto API tests).
• Streamed path (128f, server + mutual auth): pass.
• Single-record path (128s) and DTLS 1.3 in-place path: pass.
• Full library build with -Werror: clean; verified the streaming feature-macro
  off path (classical/async builds) also compiles clean.

Known limitations / follow-ups

• Test coverage exercises the 128f/128s fragmentation boundaries; the 192f/256f parameter sets (deepest fragmentation, up to ~49 KB) are not yet covered at the TLS layer. Adding a 192f/256f entity config (guarded on the corresponding WOLFSSL_SLHDSA_PARAM_* macros) would strengthen the streaming/fragmentation coverage.
• The streamed dual-algorithm (WOLFSSL_DUAL_ALG_CERTS, BOTH) path with a fragment-sized signature is not covered by a test config (no dual-alg SLH-DSA test certificates exist); it is correct by construction but unexercised.
• Dual-alg alt-signature verification now uses a tight computed PreTBS bound rather than a fixed size; a regression test for a valid dual-alg certificate with a large alternative signature would be good hygiene.
• DTLS memory reduction (sign into the fragments buffer) is noted in-code as a possible future optimization.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m3

• FLASH: .text -8 B (-0.0%, 122,497 B / 262,144 B, total: 47% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .text +320 B (+0.2%, 181,272 B / 1,048,576 B, total: 17% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +576 B (+0.1%, 771,852 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +768 B (+0.3%, 280,292 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +384 B (+0.1%, 325,504 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +320 B (+0.1%, 236,530 B / 262,144 B, total: 90% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +768 B (+0.3%, 280,868 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +384 B (+0.2%, 236,594 B / 262,144 B, total: 90% used)

linuxkm-standard

• Data: __patchable_function_entries +8 B (+0.0%, 48,256 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• linuxkm-pie
• stm32-sim-stm32h753

[Frauschi]

Jenkins retest this please

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10901

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10901

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src
Findings: 3
2 finding(s) posted as inline comments (see file-level comments below)

Info (1)

SHAKE-variant SLH-DSA cases in EncodeSigAlg lack per-parameter-set guards

Function: EncodeSigAlg
Category: Code Consistency / Defensive Programming

The SHA2-family SLH-DSA cases in EncodeSigAlg are individually gated (e.g. #ifdef WOLFSSL_SLHDSA_PARAM_SHA2_128S) whereas the SHAKE-family cases fall through under the single #ifdef WOLFSSL_HAVE_SLHDSA umbrella. This is an internal inconsistency: if WOLFSSL_HAVE_SLHDSA is defined but a specific SHAKE parameter set (e.g. WOLFSSL_SLHDSA_PARAM_128F) is not, the corresponding case statement is still compiled in even though it can never be reached. AddSuiteHashSigAlgo, which feeds the negotiated suite, is correctly per-variant gated, so the enum value would never appear in args->sigAlgo. Because AddSuiteHashSigAlgo, SlhDsaSigMinorToType, and ProcessBufferTryDecodeSlhDsa all enforce per-parameter-set guards at the point where an algorithm is admitted into the session, a SHAKE variant that is not compiled in cannot appear in a live handshake.

Recommendation: For consistency with the SHA2 branch and to make dead-code elimination explicit, add per-parameter-set preprocessor guards around each SHAKE case in EncodeSigAlg. No functional or security change results; the improvement is defensive consistency.

---

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10901

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10901

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10901

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.