Skip to content

Add CI that builds and runs every example#598

Open
aidangarske wants to merge 70 commits into
wolfSSL:masterfrom
aidangarske:ci-matrix
Open

Add CI that builds and runs every example#598
aidangarske wants to merge 70 commits into
wolfSSL:masterfrom
aidangarske:ci-matrix

Conversation

@aidangarske

@aidangarske aidangarske commented Jul 15, 2026

Copy link
Copy Markdown
Member

CI failures caught and fixed:

Fixtures that expired 2025-09-11 and silently broke examples

1cb1220 CRL expired, silently breaking every CRL example
34a591f certloadverifybuffer.c embedded certs expired while certs/*.pem were regenerated
99cc7ba client-ecc-cert.der missed in that refresh, so it disagreed with its own .pem

Wrong exit codes — examples that "passed" by returning garbage

a1383df three tls/ servers returned a wolfSSL_write byte count from main
cc37913 certverify returned WOLFSSL_SUCCESS (=1) after succeeding
79d1f67 ml_dsa -p printed its table then exit(EXIT_FAILURE)
297705b custom-io client and server returned -1 unconditionally
07ea19f ecc-params reflected only the last of six calls
8aea08f csr_w_ed25519_example and rsa-public-decrypt-app returned byte lengths

Examples that never worked

e24538c client-tls13-resume sent no client cert, so server-tls13 rejected it and it never resumed
666b6a4 aes-xts used identical XTS key halves, which wolfSSL rejects per IEEE 1619
166e6c4 PQ examples used renamed/pre-FIPS-204 APIs and could not build against master

Checks that could not fail by construction

bee99c9 runall.sh bare exit returned the preceding echo's status
88d5659 openssl-verify.sh never exited non-zero, and its signedData verifies never worked

ESP32 — unbuildable by anyone, on any supported IDF

cbe7358 USE_MY_PRIVATE_CONFIG was committed enabled, forcing an include of one developer's private header
c0cd3a2 the DTLS13 examples define no WiFi Kconfig, so CONFIG_ESP_WIFI_SSID was undeclared
0dbb03f client-dtls13.c called XSTRCPY, which wolfSSL does not define
a26f443 a non-static global ctx collided with libnet80211.a under -fno-common
f0f3a18 build against ESP-IDF v5.3; 6.x drops the PERIPH_*_MODULE API esp32_aes.c uses

Everything else

24237eb btle SIGPIPE killed whichever peer wrote close_notify last
53670e3 can-bus client busy-spun on stdin EOF instead of exiting
6338d67 server-dtls-nonblocking never set timeval.tv_usec and never re-armed it, so select polled with 0 and burned the DTLS retransmit budget in ms
259ed1a memory-bio-dtls use-after-free: wolfSSL_set_bio(ssl, NULL, NULL) frees the BIOs it was called to protect, so the client tore down BIOs the server was still reading
ddcb1c4 BIO_s_mem is a byte stream, so DTLS records end mid-read and were dropped as truncated datagrams; the dtls profile now sets WOLFSSL_DTLS_RECORDS_CAN_SPAN_DATAGRAMS
db1c0b2 annotated tags resolved to the tag object, not the commit
6eb6a90 a failed identity gate left no results file, so triage read the job as never-ran
7e33bfc / 5676d1f eight tls servers lacked SO_REUSEADDR and could not rebind after a restart
a29961b double free and NULL derefs in custom-io cleanup
064492f dtls rw-threads certificate paths
4483350 generate_ssl.sh common name length limit
117b793 / e1d9831 x509_acert headers, consistent Makefile prefixes
1dc1e79 crypto-keywrap sat under the manifest's skip header while being runnable

And three bugs in wolfSSL itself

Found by running examples nobody had run

608cf8f ocsp-stapling's responder cert was issued by the root, but server1 by intermediate1 — RFC 6960 only accepts a delegate issued by the same CA, so wolfSSL rejected every response (BAD_OCSP_RESPONDER). Swapped to wolfSSL's purpose-built ocsp-responder-int1 cert.
b99c579 softhsm2.sh ran 8 binaries and returned only the last exit code — 7 could fail and it reported success. It also hardcoded /usr/local/lib/softhsm, so it was broken for any distro package.
8b67f8e psa/Makefile read $(PSA_WOLFSSL_INSTALL_DIR), a name that appears nowhere in its own README — which documents PSA_LIB_PATH. Following the docs linked nothing and died on ~20 undefined psa_* symbols.
87c3211 pk/ecc/ecc_sign.c exported r/s with mp_to_unsigned_bin (minimal bytes) but verify reads them at fixed curveSz offsets. A leading-zero r/s (~1 in 256) shifted the signature → intermittent -229. Caught as a flake: identical v5.9.2-stable tag, one run passed, the next failed.
5f04377 mynewt/client-tls-mn.c cast pointers through int — fine on the 32-bit native BSP of 2018, -Werror fatal on today's native64.
104aeea uefi-library tested RSA with a 1024-bit key; wolfSSL's RSA_MIN_SIZE defaults to 2048. Moved to wolfSSL's own certs/client-key.der. (Did not fix the UEFI -262 — see below.)
9a06ba8 mynewt/jenkins.sh hardcoded a wolfSSL master clone, so a stable leg would have silently tested master.

Real bugs found, recorded, not yet fixed

where what
uefi-library wolfCrypt boots under UEFI and AES/SHA/HMAC/CMAC/KDF/RNG pass, then wc_CheckRsaKey fails -262. Key size is ruled out (2048 gives the same). rsa.c collapses every error into RSA_KEY_PAIR_E, hiding the cause; the driver API doesn't export wc_RsaSSL_Sign/VerifyInline to instrument it. Needs a Linux gnu-efi box.
mynewt wolfSSL's Mynewt port does not compile, on master and stable: settings.h sets XMALLOC_OVERRIDE and defines XFREE as a macro, types.h still declares it extern — the macro expands into the declaration.
pkcs11 pkcs11_test fails -262 on a label-generated EC key (gen_ec_keys_labelwc_ecc_verify_hash); the same sequence without a label passes. Deterministic on both refs. Runs as its own step so the other 7 stay asserted.

Copilot AI review requested due to automatic review settings July 15, 2026 00:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a comprehensive GitHub Actions CI system to build and (where possible) run every example in this repository against wolfSSL master and the latest stable tag(s), using a manifest-driven matrix so failures are isolated to per-example jobs.

Changes:

  • Introduces manifest-driven CI workflows (host/emulated/cross) plus a nightly triage workflow that retries, classifies flakes vs real failures, and can file/update issues.
  • Adds Python harness/scripts to generate matrices, enforce “coverage” (no buildable example directory left unaccounted for), run examples with output assertions, and collect artifacts for triage.
  • Updates many examples/Makefiles/scripts/fixtures so they build/run reliably under CI (exit statuses, paths to certs, linking flags, regenerated CRL/certs, etc.).

Reviewed changes

Copilot reviewed 62 out of 64 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
X9.146/Makefile Makefile header tweak for X9.146 examples
x509_acert/README.md Fix build/run instructions and cert paths
x509_acert/Makefile Make OpenSSL prefix optional; adjust lib paths
uefi-static/Makefile Make libgcc/gnu-efi paths less distro-specific
tls/server-tls13.c Ensure successful exit status after shutdown
tls/server-tls13-certauth-clienthello.c Ensure successful exit status after shutdown
tls/server-tls-posthsauth.c Ensure successful exit status after shutdown
tls/client-tls13-resume.c Load client cert/key and CA separately
tls/client-tls.c Load client cert/key and CA separately
staticmemory/memory-bucket-optimizer/tester/Makefile Add include/lib paths for wolfSSL
staticmemory/memory-bucket-optimizer/optimizer/Makefile Add include/lib paths for wolfSSL
staticmemory/Makefile Correct Makefile title/comment
signature/sigtest/Makefile Make OpenSSL path optional and clearer
SGX_Linux/sgx_t.mk Update SGX C++ runtime library name
scripts/run-all-examples.sh Add local manifest-driven runner script
psa/Makefile Fix include path and guard empty PSA_INCLUDE
pq/stateful_hash_sig/xmss_example.c Remove redundant include
pq/stateful_hash_sig/lms_example.c Remove redundant include
pq/ml_kem/README.md Switch build instructions to make
pq/ml_kem/ml_kem.c Update include/header usage; simplify main signature
pq/ml_kem/Makefile Add Makefile for ML-KEM example
pq/ml_dsa/ml_dsa.c Update to FIPS 204 ctx APIs; fix exit code
pq/ml_dsa/Makefile Fix clean target artifacts list
pkcs7/scripts/runall.sh Fix ordering + ensure failures exit non-zero
pkcs7/scripts/openssl-verify.sh Gate on runall, add failure accounting, improve checks
pkcs7/Makefile Correct Makefile title and zlib comment
pkcs11/Makefile Correct Makefile title/comment
pk/srp/Makefile Fix WOLFSSL_INSTALL_DIR prefix
pk/enc-through-sign-rsa/rsa-public-decrypt-app.c Normalize return code from length-returning API
embedded/Makefile Adjust linux-specific target list handling
ecc/ecc-params.c Handle negative lengths; normalize exit status
ebpf/tls-uprobe-trace/Makefile Add wolfSSL include/lib flags
dtls/server-dtls-rw-threads.c Fix cert paths for CI execution
dtls/client-dtls-rw-threads.c Fix cert paths for CI execution
custom-io-callbacks/file-server/file-server.c Fix server return codes / cleanup flow
custom-io-callbacks/file-server/check.sh Make server/client orchestration bounded and reliable
custom-io-callbacks/file-client/file-client.c Fix client return codes
crypto/keywrap/Makefile Add Makefile for keywrap example
crypto/aes-modes/aes-xts.c Fix XTS key halves to be distinct
certvfy/Makefile Add include/lib flags for wolfSSL
certs/crl/regenerate-crl.sh Add script to regenerate CRL fixture
certs/crl/crl.pem Update CRL fixture to unexpired version
certmanager/README.md Fix example invocation name
certmanager/certverify.c Ensure successful exit status
certmanager/certloadverifybuffer.c Update embedded cert fixtures
certgen/Makefile Add Makefile header/comment
certgen/csr_w_ed25519_example.c Normalize return code from length-returning API
can-bus/Makefile Add include/lib flags for wolfSSL
can-bus/generate_ssl.sh Cap CN length to X.509 limits
btle/common/btle-sim.c Ignore SIGPIPE to avoid FIFO-close crashes
.gitignore Ignore new binaries/pycache; fix typo
.github/workflows/nightly.yml Add scheduled nightly orchestration
.github/workflows/examples.yml Add host-tier matrix workflow + coverage gate
.github/workflows/emulated.yml Add emulated-tier workflow (TPM/SGX/CAN/UEFI)
.github/workflows/cross-build.yml Add cross-build workflow (ESP32/Pico/etc.)
.github/workflows/ci-triage.yml Add triage workflow for nightly results
.github/workflows/_resolve-wolfssl.yml Add reusable wolfSSL ref resolution workflow
.github/scripts/run_example.py Add example harness (build/run/assert/output/artifacts)
.github/scripts/manifest.py Add manifest loader + matrix generator + coverage checker
.github/scripts/issue_store.py Add issue identity/body/state machine + scrubbing
.github/scripts/ci_triage.py Add triage logic (retry, classify, issues, AI advisory)
.github/examples-manifest.yml Add source-of-truth manifest for CI coverage/matrix
.github/actions/setup-wolfssl/action.yml Add composite action to build/cache/install wolfSSL

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread custom-io-callbacks/file-server/file-server.c
Comment thread custom-io-callbacks/file-server/file-server.c Outdated
Comment thread custom-io-callbacks/file-server/file-server.c
Comment thread custom-io-callbacks/file-client/file-client.c
Comment thread .github/examples-manifest.yml Outdated
@aidangarske
aidangarske force-pushed the ci-matrix branch 17 times, most recently from 50ffcaf to 74965b0 Compare July 17, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants