Test failing in test_crypto.py

[papoteur-mga]

Describe the bug

I'm trying to build the version 0.27.4 in Mageia, using a chroot without Internet access.

When running the tests, I get errors like Unknown error when calling underlying Crypto library. Is there means to have more DEBUG messages?

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Tests pass

Current Behavior

+ PYTHONPATH=/builddir/build/BUILD/python-awscrt-0.27.4-build/BUILDROOT/usr/lib64/python3.13/site-packages:/builddir/build/BUILD/python-awscrt-0.27.4-build/BUILDROOT/usr/lib/python3.13/site-packages
+ /usr/bin/python3 -m unittest
.ssss.......Exception ignored in: <function TestProvider.test_delegate_provider_exception.<locals>.delegate_get_credentials at 0x7fea65f72d40>
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_auth.py", line 204, in delegate_get_credentials
    raise Exception("purposefully thrown exception")
Exception: purposefully thrown exception
.Exception ignored in: <function TestProvider.test_delegate_provider_exception_from_bad_return_type.<locals>.delegate_get_credentials at 0x7fea65f04220>
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/auth.py", line 442, in get_credentials
    _awscrt.credentials_provider_get_credentials(self._binding, _on_complete)
TypeError: Expected valid 'AwsCredentials' (no '_binding' attribute)
................................................EEE...................ssssssssss........127.0.0.1 - - [07/Jul/2025 18:54:22] "GET /test/test_http_client.py HTTP/1.1" 200 -
.127.0.0.1 - - [07/Jul/2025 18:54:22] "GET /test/test_http_client.py HTTP/1.1" 200 -
.sE127.0.0.1 - - [07/Jul/2025 18:54:22] "PUT /test/test_http_client.py HTTP/1.1" 200 -
.127.0.0.1 - - [07/Jul/2025 18:54:22] "PUT /test/test_http_client.py HTTP/1.1" 200 -
.127.0.0.1 - - [07/Jul/2025 18:54:23] "PUT /test/test_request_refcounts.txt HTTP/1.1" 200 -
.127.0.0.1 - - [07/Jul/2025 18:54:23] "PUT /test/test_request_refcounts.txt HTTP/1.1" 200 -
.127.0.0.1 - - [07/Jul/2025 18:54:23] "GET / HTTP/1.1" 200 -
.127.0.0.1 - - [07/Jul/2025 18:54:23] "GET / HTTP/1.1" 200 -
...127.0.0.1 - - [07/Jul/2025 18:54:24] "GET /test/test_http_client.py HTTP/1.1" 200 -
.127.0.0.1 - - [07/Jul/2025 18:54:24] "GET /test/test_http_client.py HTTP/1.1" 200 -
.ssss.......................s.........F...s.ss...sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss/usr/lib64/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=26959) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
........sssssssssssssssssssssssssssssss...opening handshake failed
Traceback (most recent call last):
  File "/usr/lib64/python3.13/site-packages/websockets/asyncio/server.py", line 353, in conn_handler
    await connection.handshake(
    ...<3 lines>...
    )
  File "/usr/lib64/python3.13/site-packages/websockets/asyncio/server.py", line 204, in handshake
    raise self.protocol.handshake_exc
  File "/usr/lib64/python3.13/site-packages/websockets/server.py", line 134, in accept
    ) = self.process_request(request)
        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib64/python3.13/site-packages/websockets/server.py", line 235, in process_request
    raise InvalidUpgrade(
        "Connection", ", ".join(connection) if connection else None
    )
websockets.exceptions.InvalidUpgrade: missing Connection header
...Exception in WebSocket on_incoming_frame_begin callback
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/websocket.py", line 487, in _on_incoming_frame_begin
    self._on_incoming_frame_begin_cb(cbdata)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_websocket.py", line 502, in bad_incoming_frame_callback
    raise RuntimeError("Purposefully raising exception")
RuntimeError: Purposefully raising exception
Exception in WebSocket on_incoming_frame_payload callback
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/websocket.py", line 501, in _on_incoming_frame_payload
    self._on_incoming_frame_payload_cb(cbdata)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_websocket.py", line 502, in bad_incoming_frame_callback
    raise RuntimeError("Purposefully raising exception")
RuntimeError: Purposefully raising exception
Exception in WebSocket on_incoming_frame_complete callback
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/websocket.py", line 519, in _on_incoming_frame_complete
    self._on_incoming_frame_complete_cb(cbdata)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_websocket.py", line 502, in bad_incoming_frame_callback
    raise RuntimeError("Purposefully raising exception")
RuntimeError: Purposefully raising exception
.Exception in WebSocket on_connection_setup callback
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/websocket.py", line 459, in _on_connection_setup
    self._on_connection_setup_cb(cbdata)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_websocket.py", line 334, in bad_setup_callback
    raise RuntimeError("Purposefully raising exception")
RuntimeError: Purposefully raising exception
.Exception in WebSocket on_connection_shutdown callback
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/websocket.py", line 474, in _on_connection_shutdown
    self._on_connection_shutdown_cb(cbdata)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_websocket.py", line 357, in bad_shutdown_callback
    raise RuntimeError("Purposefully raising exception")
RuntimeError: Purposefully raising exception
..Exception in WebSocket.send_frame on_complete callback
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/websocket.py", line 390, in _on_complete
    on_complete(cbdata)
    ~~~~~~~~~~~^^^^^^^^
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_websocket.py", line 480, in bad_completion_callback
    raise RuntimeError("Purposefully raising exception")
RuntimeError: Purposefully raising exception
..
======================================================================
ERROR: test_rsa_signing_roundtrip (test.test_crypto.TestCredentials.test_rsa_signing_roundtrip) [RSA Signing Roundtrip using algo p] (p=<RSASignatureAlgorithm.PKCS1_5_SHA1: 1>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_crypto.py", line 252, in test_rsa_signing_roundtrip
    signature = rsa.sign(p, digest)
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/crypto.py", line 164, in sign
    return _awscrt.rsa_sign(self._binding, signature_algorithm, digest)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: 7179 (AWS_ERROR_CAL_CRYPTO_OPERATION_FAILED): Unknown error when calling underlying Crypto library.

======================================================================
ERROR: test_rsa_signing_roundtrip_der (test.test_crypto.TestCredentials.test_rsa_signing_roundtrip_der) [RSA Signing Roundtrip using algo p] (p=<RSASignatureAlgorithm.PKCS1_5_SHA1: 1>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_crypto.py", line 292, in test_rsa_signing_roundtrip_der
    signature = rsa.sign(p, digest)
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/crypto.py", line 164, in sign
    return _awscrt.rsa_sign(self._binding, signature_algorithm, digest)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: 7179 (AWS_ERROR_CAL_CRYPTO_OPERATION_FAILED): Unknown error when calling underlying Crypto library.

======================================================================
ERROR: test_rsa_signing_roundtrip_pkcs8 (test.test_crypto.TestCredentials.test_rsa_signing_roundtrip_pkcs8) [RSA Signing Roundtrip using algo p] (p=<RSASignatureAlgorithm.PKCS1_5_SHA1: 1>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_crypto.py", line 273, in test_rsa_signing_roundtrip_pkcs8
    signature = rsa.sign(p, digest)
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/awscrt/crypto.py", line 164, in sign
    return _awscrt.rsa_sign(self._binding, signature_algorithm, digest)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: 7179 (AWS_ERROR_CAL_CRYPTO_OPERATION_FAILED): Unknown error when calling underlying Crypto library.

======================================================================
ERROR: test_h2_manual_write_exception (test.test_http_client.TestClient.test_h2_manual_write_exception)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_http_client.py", line 381, in test_h2_manual_write_exception
    connection = self._new_h2_client_connection(url)
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_http_client.py", line 354, in _new_h2_client_connection
    return connection_future.result(self.timeout)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
awscrt.exceptions.AwsCrtError: AWS_ERROR_HTTP_UNSUPPORTED_PROTOCOL: An unsupported protocol was encountered.

======================================================================
FAIL: test_shutdown_complete_singleton (test.test_io.EventLoopGroupTest.test_shutdown_complete_singleton)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-awscrt-0.27.4-build/awscrt-0.27.4/test/test_io.py", line 41, in test_shutdown_complete_singleton
    self.assertTrue(shutdown_event.wait(TIMEOUT))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: False is not true

----------------------------------------------------------------------
Ran 405 tests in 39.765s

FAILED (failures=1, errors=4, skipped=244)

Reproduction Steps

python -m unittest

Possible Solution

No response

Additional Information/Context

No response

aws-crt-python version used

0.27.4

Python version used

3.13

Operating System and version

Mageia Cauldron (10)

[DmitriyMusatkin]

The actual error comes from aws-c-cal and in general implies that underlying crypto lib threw and error and c-cal had no idea what that error was. c-cal should have logged error details from libcrypto under error/warn.
you should be able to set crt logging to trace in unit tests by uncommenting the line here

aws-crt-python/test/__init__.py

Line 29 in 5cf5e7d

# init_logging(LogLevel.Trace, 'stderr')

[papoteur-mga]

Hello,
I applied the uncommenting above.
First error appears:

[INFO] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [channel-bootstrap] - id=0x5571b176f270: Initializing client bootstrap with event-loop group 0x5571b16f7370
[INFO] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AuthCredentialsProvider] - (id=0x5571b19ad010): TLS context not provided, initializing a new one for credentials provider.
[DEBUG] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AWSProfile] - Creating profile collection from file at "/builddir/.aws/config"
[ERROR] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [common-io] - static: Failed to open file. path:'/builddir/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AWSProfile] - Failed to read file at "/builddir/.aws/config"
[DEBUG] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AWSProfile] - Creating profile collection from file at "/builddir/.aws/credentials"
[ERROR] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [common-io] - static: Failed to open file. path:'/builddir/.aws/credentials' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AWSProfile] - Failed to read file at "/builddir/.aws/credentials"
[ERROR] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AuthCredentialsProvider] - static: Profile credentials parser could not load or parse a credentials or config file.
[DEBUG] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AWSProfile] - Creating profile collection from file at "/builddir/.aws/config"
[ERROR] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [common-io] - static: Failed to open file. path:'/builddir/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
[DEBUG] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AWSProfile] - Failed to read file at "/builddir/.aws/config"
[ERROR] [2025-07-10T14:44:34Z] [00007f019e9b6bc0] [AuthCredentialsProvider] - Failed to build config profile collection from file at (/builddir/.aws/config) : Invalid file path.

The path it looks for is suspicious. The base /builddir is the root of the building tree in the chroot.

[bretambrose]

Those aren't really errors. We'd need to see the whole log file.

[papoteur-mga]

Here is the whole log: https://paste.centos.org/view/27ed83dc
After disabling SHA1 algorithm. the tests pass.

[DmitriyMusatkin]

Mageia Cauldron is a dev version as far as i understand? i tried running latest mageia from docket and could not repro the issue with either openssl or aws-lc.

Which version of openssl are you building against? we typically print it in build logs, but in this case it seems to be missing.
I dont see anything upstream in openssl about not supporting pkcs1.5 with sha1, but looks like all tests are failing somewhere in the code when its trying to configure that.

[DmitriyMusatkin]

small update on this: so fips versions of openssl seems to have dropped support for sha1 when used for rsa signing.
seems like new versions of Mageia ship with fips openssl and you are building aws-crt-python against system openssl, so error is kinda expected.
im looking into how to best deal with this, but we will likely end up disabling those tests on fips mode