Name the certificate verification failures in ssl OSErrors

[dhalbert]

• Fixes ssl: certificate verification failures all report -9984; surface the mbedtls verify flags #11148 (@dhalbert)

Authored by Claude Code with guidance by @dhalbert. This message edited by @dhalbert for clarity.

Every certificate verification failure used to be reported as the same bare error number, so a hostname mismatch was indistinguishable from an untrusted CA or a bad signature algorithm. mbedtls_ssl_get_verify_result() was never called, so the MBEDTLS_X509_BADCERT_* bitmask that records which check failed was thrown away.

Now do_handshake() reads that bitmask before mbedtls_ssl_free() frees the session it lives in, and mbedtls_raise_error_flags() gets an error message for each set bit using mbedtls_x509_crt_verify_info(). For example:

OSError: (-9984, 'MBEDTLS_ERR_X509_CERT_VERIFY_FAILED; The certificate Common Name (CN) does not match with the expected CN; The certificate is not correctly signed by the trusted CA')

CONFIG_MBEDTLS_ERROR_STRINGS is now turned on for espressif to enable MBEDTLS_ERROR_C so error names are available; the other mbedtls ports already defined MBEDTLS_ERROR_C.

Notes on the implementation

• One bit at a time. Several bits are commonly set at once, because mbedtls or's together the flags of every certificate in the chain. So we call mbedtls_x509_crt_verify_info() once for each flag bit and append each string to the error message so we don't have to worry about sizing a buffer large enough for long output
• All ones means "nothing to report". mbedtls sets the flags to (uint32_t) -1 when a verify callback fails, and mbedtls_ssl_get_verify_result() returns 0xFFFFFFFF when there is no session at all. So ignore that flag word and only look at flags on VERIFY_FAILED.
• Memory. The message is built in a vstr. In the unlikely event vstr raises MemoryError due to heap exhaustion, the message construction is caught by a wrapped nlr_push()/nlr_pop() and falls back to a bare OSError(err).

Limits on which errors will appear

• With the default context, the crt bundle installs an mbedtls verify callback that returns a non-fatal code; mbedtls converts that to MBEDTLS_ERR_X509_FATAL_ERROR and sets the flags to all ones (x509_crt.c, "prevent misuse of the vrfy callback"). So anything reducing to NOT_TRUSTED reports only the error name. Failures encountered outside that callback — CN mismatch, unacceptable hash/PK/key — are named. Passing a CA with load_verify_locations() installs no callback, and there the full flags survive.
• BADCERT_EXPIRED, BADCERT_FUTURE and the two CRL date bits can never be set on any CircuitPython build, because MBEDTLS_HAVE_TIME_DATE is deliberately off (lib/mbedtls_config/mbedtls_config.h), CircuitPython does not know the wall clock time unless it is set.

Testing

Built and run on hardware: Metro ESP32-S3 (esp-idf mbedtls) and Pico 2 W (lib/mbedtls). Both ports build clean.

Default context, against badssl.com:

expired.badssl.com:        OSError (-12288, 'MBEDTLS_ERR_X509_FATAL_ERROR')
wrong.host.badssl.com:     OSError (-9984, 'MBEDTLS_ERR_X509_CERT_VERIFY_FAILED; The certificate Common Name (CN) does not match with the expected CN')
self-signed.badssl.com:    OSError (-12288, 'MBEDTLS_ERR_X509_FATAL_ERROR')
untrusted-root.badssl.com: OSError (-12288, 'MBEDTLS_ERR_X509_FATAL_ERROR')
badssl.com:                OK

With an unrelated CA loaded via load_verify_locations(), which exercises the no-callback path and the multi-flag case:

expired.badssl.com:     OSError (-9984, '...; The certificate is not correctly signed by the trusted CA')
wrong.host.badssl.com:  OSError (-9984, '...; The certificate Common Name (CN) does not match with the expected CN; The certificate is not correctly signed by the trusted CA')
badssl.com:             OSError (-9984, '...; The certificate is not correctly signed by the trusted CA')

[dhalbert]

An observation about using Claude for this: I inadvertently set the level for Opus 5 to "medium" instead of "high", and found I had to do a lot more coaching. It was writing code that would work, but it wasn't that clever, and I had to set it back on track several times. For instance, I had to tell it to use vstr. Once I set it back to high it did quite a bit better, and started noticing thing that I might have missed.