Patch CRL fetch failures

[ChrisThibodeaux]

PR for two major issues with fetching CRLs:

1. Unexpected response MIME type
2. Unreliable response body handling

First Bug:

application/octet-stream response types caused CRL/TSA-CRL fetch failures. Possibly only an issue using Openssl >= 3.0

Error output example:

TSA-CRLfile fetch failure:
TSA's CRL distribution point: http://crl.microsoft.com/pki/crl/products/MicrosoftTimeStampPCA.crl
Connecting to http://crl.microsoft.com/pki/crl/products/MicrosoftTimeStampPCA.crl

HTTP failure: Failed to get data from http://crl.microsoft.com/pki/crl/products/MicrosoftTimeStampPCA.crl
Failed to get CRL from http://crl.microsoft.com/pki/crl/products/MicrosoftTimeStampPCA.crl

Use the "-TSA-CRLfile" option to add one or more Time-Stamp Authority CRLs in PEM format.
00E057FE39750000:error:1700006B:CMS routines:cms_get_enveloped_type:content type not enveloped data:../crypto/cms/cms_env.c:41:
00E057FE39750000:error:1E800076:HTTP routines:OSSL_HTTP_REQ_CTX_nbio:unexpected content type:../crypto/http/http_client.c:676:expected=application/pkix-crl, actual=application/octet-stream
00E057FE39750000:error:1E800067:HTTP routines:OSSL_HTTP_REQ_CTX_exchange:error receiving:../crypto/http/http_client.c:874:

Curl showing response Content-Type: application/octet-stream:

$ curl -I http://crl.microsoft.com/pki/crl/products/MicrosoftTimeStampPCA.crl

HTTP/1.1 200 OK
Content-Length: 519
Content-Type: application/octet-stream
Content-MD5: 6Vr5sDUT1ynSj9iQz/Tr6Q==
Last-Modified: Tue, 30 Mar 2021 15:18:44 GMT
ETag: 0x8D8F38F1BA23B59
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 196a6cae-001e-009b-4951-ecca48000000
x-ms-version: 2009-09-19
x-ms-lease-status: unlocked
x-ms-blob-type: BlockBlob
Date: Sun, 14 Dec 2025 03:33:18 GMT
Connection: keep-alive

Failing cert example:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            33:00:00:01:29:a0:47:69:3a:a2:a4:6c:ed:00:00:00:00:01:29
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Time-Stamp PCA
        Validity
            Not Before: Sep  6 20:40:00 2019 GMT
            Not After : Dec  4 20:40:00 2020 GMT
        Subject: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, OU=Microsoft Operations Puerto Rico, OU=Thales TSS ESN:BBEC-30CA-2DBE, CN=Microsoft Time-Stamp Service
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e7:d2:82:2a:30:8e:be:8c:0b:a4:e4:78:0c:44:
                    89:6f:01:3f:46:8d:f5:e9:cc:90:0d:b4:e2:1d:81:
                    a3:ee:97:f5:9a:1d:84:8a:a5:fd:29:81:9e:d6:89:
                    54:59:98:85:ea:1e:2e:ec:ed:cd:fc:8c:0d:3f:99:
                    26:e8:d4:0f:f3:5c:99:fd:d8:23:36:08:c2:46:cf:
                    d3:1d:c2:a6:2d:1d:4a:3b:0b:9a:aa:95:20:6c:ea:
                    4c:fb:08:eb:a7:7a:1d:a8:70:50:b2:46:26:54:2e:
                    5d:3f:b9:67:aa:ba:d7:04:f6:6c:79:5a:98:78:fc:
                    65:c3:20:d5:27:a3:b8:c2:6a:f2:14:76:a8:9f:0b:
                    6c:4c:05:4e:15:0e:93:20:c7:a7:ee:67:b5:0e:95:
                    d8:45:15:5d:81:3c:54:02:a7:a2:87:62:c1:f8:cb:
                    0f:de:66:d7:2d:0c:63:6a:ad:15:35:0f:fc:99:04:
                    e6:e9:92:09:3c:37:cd:16:0a:be:6b:a3:fe:d2:bd:
                    90:74:f5:73:8d:39:c1:88:61:06:33:37:7d:99:2a:
                    f2:79:9a:ac:b5:2e:c3:08:f5:b9:c4:e8:86:6f:8e:
                    eb:8a:4c:7a:7c:d7:ff:ec:eb:85:15:99:0d:a9:fa:
                    5a:4f:14:d7:3e:be:a2:01:88:fd:0f:10:87:af:79:
                    e3:a7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                46:6E:BE:C2:B6:3F:76:A0:13:03:25:8B:F0:60:33:AD:E2:95:92:C9
            X509v3 Authority Key Identifier: 
                23:34:F8:D9:52:46:70:0A:ED:40:FB:76:FB:B3:2B:B0:C3:35:B3:0F
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://crl.microsoft.com/pki/crl/products/MicrosoftTimeStampPCA.crl
            Authority Information Access: 
                CA Issuers - URI:http://www.microsoft.com/pki/certs/MicrosoftTimeStampPCA.crt
            X509v3 Extended Key Usage: 
                Time Stamping
    Signature Algorithm: sha1WithRSAEncryption
    Signature Value:
        71:d6:5e:0c:da:28:a2:ef:7c:97:21:de:1e:fd:20:d9:37:1d:
        5c:1a:0c:79:63:04:e7:bc:a3:a9:bf:66:38:41:b4:73:21:83:
        24:8d:12:2b:e3:dc:05:25:9c:15:cc:05:b6:c9:a7:27:f0:dc:
        7c:2b:13:15:c8:15:62:33:b3:7a:ca:0f:f5:31:f2:d3:f2:fa:
        f1:9a:b1:db:3c:be:1a:ac:ed:f7:2d:e2:10:72:78:4b:53:cc:
        79:a4:e7:b9:1b:e8:20:a1:e2:02:b7:ef:b4with:bb:44:4e:ac:0f:
        cb:58:a0:a2:20:ae:b7:9d:69:21:c4:9f:64:6a:10:7f:f6:a3:
        e8:6c:35:ea:2f:6a:4a:30:c0:82:fa:cc:67:b4:fe:41:70:ec:
        3f:cf:69:9d:0e:9a:db:3a:be:ca:21:92:dd:c9:5a:db:8a:c6:
        40:84:95:7f:94:45:93:61:a7:22:3e:56:d8:c8:0e:3a:4a:89:
        00:4f:a9:95:b5:cc:cf:fd:da:78:da:75:5e:64:dd:82:03:8d:
        0c:8a:11:46:a2:c0:b5:6f:22:d2:40:a7:f6:d2:50:cd:ed:2a:
        4a:ec:fc:44:2f:8d:9c:ff:b0:e0:f4:fc:fc:07:76:5c:9c:5a:
        fb:80:73:87:5a:a7:a3:cb:d5:52:4c:bc:5d:de:a8:0e:98:62:
        70:a0:15:27

Second Bug:

Unreliable handling of GET responses causing truncated DER data.

Error output example:

CRL distribution point: http://crl.comodoca.com/COMODORSACodeSigningCA.crl
Connecting to http://crl.comodoca.com/COMODORSACodeSigningCA.crl
Failed to decode CRL from http://crl.comodoca.com/COMODORSACodeSigningCA.crl

Use the "-CRLfile" option to add one or more CRLs in PEM format.
0080CC2331790000:error:0680008E:asn1 encoding routines:asn1_d2i_read_bio:not enough data:../crypto/asn1/a_d2i_fp.c:216:
0080CC2331790000:error:0480006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:763:

To test this endpoint returns a valid CRL:

1. Download CRL file directly from http://crl.comodoca.com/COMODORSACodeSigningCA.crl by any direct means
2. Run openssl crl -inform DER -in COMODORSACodeSigningCA.crl -noout -text

Sample output:

Certificate Revocation List (CRL):
        Version 2 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Code Signing CA
        Last Update: Dec 13 16:44:33 2025 GMT
        Next Update: Dec 20 16:44:33 2025 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                29:91:60:FF:8A:4D:FA:EB:F9:A6:6A:B8:CF:F9:E6:4B:BD:49:CE:12
            X509v3 CRL Number:
                4868
            2.5.29.60:
                ..20130509000000Z
Revoked Certificates:
    Serial Number: 48C33203EB0C402E4E98D1D764D88F2E
        Revocation Date: Jan 30 09:19:16 2014 GMT

...

    Serial Number: C819853791BEB5BE0E9C375136A9D852
        Revocation Date: Jan  4 00:00:00 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code: 
                Key Compromise
    Serial Number: 11F290A2749073D8C464CCDF9313B7AD
        Revocation Date: Dec 12 00:00:00 2023 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code: 
                Key Compromise
    Serial Number: 04A03DBCE32C5A34420A419FB740AA1A
        Revocation Date: Feb  2 00:00:00 2016 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code: 
                Privilege Withdrawn
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        64:f6:90:fa:32:f2:c6:46:da:bd:14:3e:a2:26:a8:5c:3d:19:
        0c:52:14:e8:6c:71:1d:1a:b7:67:d4:20:93:e2:65:20:4c:6f:
        c0:54:6c:99:03:d8:dd:75:70:78:46:4e:72:dd:81:ae:bf:18:
        c3:82:f4:cc:e8:ed:66:4e:31:3f:21:e3:47:79:25:1f:1d:44:
        27:7b:52:22:47:2b:03:87:a9:bc:da:92:b0:ef:95:b2:3a:b4:
        28:7d:30:4c:77:fa:78:c4:6e:d4:ed:1d:8d:a2:ee:78:cb:41:
        ce:1c:7e:e1:36:b5:a7:ac:77:ec:07:82:0e:30:2e:02:78:7a:
        d5:f9:2f:fd:5d:3a:57:42:48:d0:a0:52:04:d3:8e:59:45:b1:
        67:93:53:fe:11:0f:d6:30:59:d1:c2:a2:32:52:f3:5d:7f:80:
        dd:2b:44:d7:a4:98:cf:be:a4:ff:b5:6a:07:98:41:bb:e5:29:
        80:e0:6b:69:de:14:f1:35:eb:d0:8c:da:80:c6:94:32:d4:1b:
        8c:b6:f7:72:e6:06:a6:53:12:7d:3f:3e:99:8c:12:a7:08:b8:
        49:6a:5e:27:78:27:52:15:6f:62:82:43:e5:0a:87:ba:84:68:
        fd:1a:24:3d:3b:aa:e1:88:28:cc:c0:ff:be:8f:03:6c:c8:f5:
        b9:07:db:5f

[ChrisThibodeaux]

Happy to add tests or additional code comments if needed. Let me know if there's anything else I should include to make this easier to review.

[olszomal]

I really appreciate you finding these issues and sending a PR with a proposed solution.

[ChrisThibodeaux]

@olszomal Not a problem, love the project and more than happy to help where I can! I've updated the branch by dropping the commit that adds bio_slurp_to_mem().

[mtrojnar]

@ChrisThibodeaux Why do you think it's beneficial to invoke OSSL_HTTP_get with expected_content_type and then again without expected_content_type? Wouldn't it be simpler to remove requesting a specific expected_content_type from the first request instead, so that a retry is no longer needed?

We try to adhere to the KISS principle as much as possible (but not more) to facilitate long-term maintainability of this project.

[ChrisThibodeaux]

@mtrojnar I'd prefer to drop expected_content_type. I left it there to not change the current behavior of something I didn't completely understand the need for. I can adjust this to drop the expected_content_type from the initial request and not make a second one.

[mtrojnar]

I can adjust this to drop the expected_content_type from the initial request and not make a second one.

Please do it. Thank you very much for your contribution to the project.

[ChrisThibodeaux]

@mtrojnar Done.

Just making sure that it does still make sense to enforce expected_content_type when retrieving timestamp replies?

[mtrojnar]

@mtrojnar Done.

Thank you.

Just making sure that it does still make sense to enforce expected_content_type when retrieving timestamp replies?

According to testing performed by @olszomal, it's not required.