Arm/Arm64: Segfault in `System.Security.Cryptography.Tests`

[adamperlin]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1460625
Build error leg or test failing: System.Security.Cryptography.Tests.WorkItemExecution
Pull request: #129140

Error Message

{
  "ErrorMessage": "Segmentation fault         (core dumped) \"$RUNTIME_PATH/dotnet\" exec --runtimeconfig System.Security.Cryptography.Tests.runtimeconfig.json",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1460625
Error message validated: [17 Segmentation fault (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.Tests.runtimeconfig.json]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/12/2026 4:48:31 PM UTC

Report

Build	Repository	Test	Pull Request
1461828	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129304
1461798	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#78746
1461779	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125436
1461360	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129315
1460625	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129140

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
5	5	5

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1460625
Error message validated: [Segmentation fault (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.Tests.runtimeconfig.json]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/12/2026 8:07:06 PM UTC

Report

Build	Repository	Test	Pull Request
1461828	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129304
1461798	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#78746
1461779	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125436
1461360	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129315
1460625	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129140

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
5	5	5

Report

Build	Repository	Test	Pull Request
1462902	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129365
1462897	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129369
1462854	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129348
1462837	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129354
1462782	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129322
1462675	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129362
1462516	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129345
1462635	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129285
1461798	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#78746
1462617	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129354
1462622	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129321
1462578	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129361
1462542	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129360
1462523	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129224
1462502	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129312
1462460	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129134
1462475	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129356
1462433	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129201
1462383	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129352
1462367	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129351
1462318	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#128869
1462287	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#128384
1462293	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129299
1462230	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129349
1462206	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1462037	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129201
1462189	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129059
1462166	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129348
1462156	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#128652
1462162	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129343
1462055	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129322
1461986	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129260
1461947	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129091
1461976	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129261
1461958	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129145
1461828	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129304
1461779	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#125436
1461360	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129315
1460625	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#129140

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
13	39	39

[dotnet-policy-service]

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.

[vcsjones]

I can take a look.

[vcsjones]

Copilot summary (Yes, I read it and made changes as required)

What we know:

• The original Helix dump showed a SIGSEGV during native/OpenSSL-related activity. The kernel core’s primary stack was in/around libcrypto.so.3 cleanup/TLS destructor paths; createdump often reports thesignal-handling/waitpid thread, which can obscure the original crashing context.
• Local repros are intermittent. In repeated runs, some attempts pass cleanly and others crash with exit code 139.
• Crash signatures are not perfectly identical. Some dumps show OpenSSL cleanup or OpenSSL key-generation activity nearby; other repros showed runtime/JIT threads active too. That means we do not yet have a clear root cause proving the exact issue.
• The crash does not look like a normal managed exception or xUnit failure; the process dies before xUnit produces a final failure summary.
• The symptom pattern could fit native memory corruption, incorrect OpenSSL object/provider/libctx lifetime, OpenSSL 3.5.7 behavior, or a bad interaction introduced by recent crypto interop changes. Wehave not proven which.

Correlation with PR #127190 (#127190) is strong:

• Current checkout with PR Prefer using non-deprecated EC OSSL APIs where possible #127190 (Prefer using non-deprecated EC OSSL APIs where possible #127190) included reproduced the crash.
• Reverting squash commit 397e72c for PR Prefer using non-deprecated EC OSSL APIs where possible #127190 (Prefer using non-deprecated EC OSSL APIs where possible #127190), rebuilding native crypto + managed crypto + tests, produced 40/40 clean runs.
• Undoing that revert, rebuilding back to the original PR-included state, reproduced the crash again on attempt 5.

So the current best conclusion is: the crash strongly correlates with PR #127190 (#127190) (“Prefer using non-deprecated EC OSSL APIs where possible”), but the exact root cause is still unknown.

---

I will keep looking for the actual cause but if this becomes very problematic then we can revert #127190

[vcsjones]

Update on the issue (again, read by me for accuracy and correctness)

.NET frees an OpenSSL OSSL_LIB_CTX while other threads may still have OpenSSL thread-local cleanup state associated with that context. When those threads later exit, OpenSSL’s thread destructorcan dereference the freed context and crash.

The failing path is provider-loaded keys:

1. SafeEvpPKeyHandle.OpenKeyFromProvider(...) creates a non-default OSSL_LIB_CTX.
2. It loads the requested provider into that context.
3. It opens an EVP_PKEY from the provider.
4. The EVP_PKEY is used by crypto objects like RSAOpenSsl, ECDsaOpenSsl, etc.
5. When the last managed handle is disposed/finalized, .NET unloads the provider and frees the OSSL_LIB_CTX.
6. But OpenSSL may have registered per-thread cleanup handlers for that libctx on any thread that used it.
7. Later, when one of those threads exits, OpenSSL runs cleanup such as RAND thread-state cleanup and touches the freed libctx.

ASan with an instrumented OpenSSL 3.5.7 build confirmed the use-after-free:

ossl_lib_ctx_get_data
rand_get_global
rand_delete_thread_state
init_thread_stop
init_thread_destructor
__pthread_tsd_run_dtors

This matches OpenSSL’s documented constraint: OSSL_LIB_CTX_free() only performs OPENSSL_thread_stop_ex() for the current thread. Callers must ensure OPENSSL_thread_stop_ex(ctx) is called on every otherthread that used the context before freeing it. .NET cannot reliably know or coordinate all those threads, especially ThreadPool/external/native threads.

The issue reproduces reliably on Alpine/musl and not readily on Ubuntu/glibc, likely because allocator and thread-destructor behavior make the UAF visible on Alpine. Ubuntu may still have the same latentbug but not fault because freed memory remains mapped/unchanged longer or because distro OpenSSL patches alter timing/cleanup behavior.

This is generally the same problem as openssl/openssl#16778.

[vcsjones]

This shouldn't be blocking as of #129367, but will wait for the test failure count to zero out for a 24h period before making any label changes.