[BUG]: Intermittent null internals_pp_manager::get_pp() during concurrent subinterpreter imports on Windows

[rwgk]

Summary

The scheduled master CI intermittently reached a state that internals_pp_manager::get_pp() documents as impossible while importing a pybind11 module concurrently in multiple Python subinterpreters on Windows:

ImportError: get_internals: get_pp() returned nullptr

The failure occurred once during the existing stress test and the complete job passed when rerun. This issue records the expiring CI evidence and the relevant implementation history so the race can be investigated later; it is not yet a proven root cause or proposed fix.

CI evidence (workflow logs will expire)

• Workflow run: https://github.com/pybind/pybind11/actions/runs/34571080671
• Attempt 1: https://github.com/pybind/pybind11/actions/runs/34571080671/attempts/1
• Failed attempt 1 job: https://github.com/pybind/pybind11/actions/runs/34571080671/job/103173120615
• Attempt 2: https://github.com/pybind/pybind11/actions/runs/34571080671/attempts/2
• Successful equivalent job in attempt 2: https://github.com/pybind/pybind11/actions/runs/34571080671/job/103590307395
• Head commit: d87cf0b
• Failed test: tests/test_multiple_interpreters.py::test_import_in_subinterpreter_concurrently
• Failure time: 2026-09-11 06:55 UTC

The failed job otherwise reported 1,295 passed tests, 32 skipped tests, and this single failure. The rerun on 2026-09-12 used the same commit, OS image, Python, compiler, configuration, and test code; it passed the Python tests and the rest of the job. That makes this look scheduling-sensitive rather than environment-version-sensitive.

Environment preserved from the log

• GitHub Actions windows-latest (Windows Server 2025, 10.0.26100)
• Runner image: windows-2025-vs2026, version 20260907.229.1
  • https://github.com/actions/runner-images/releases/tag/win25-vs2026%2F20260907.229
• CPython 3.14.7, regular GIL build, x64
• Debug configuration, C++20
• Visual Studio 18 2026
• MSVC 19.51.36256.0; toolset executable under MSVC 14.51.36231
• CMake 4.4.3
• pytest 9.1.1, pluggy 1.6.0, NumPy 2.5.3
• PYBIND11_INTERNALS_VERSION=12
• PYBIND11_SIMPLE_GIL_MANAGEMENT disabled

Durable reproduction information

The exact test and helper at the failing commit are permanently available here:

• test_import_in_subinterpreter_concurrently
• The test's import-and-check preamble
• check_script_success_in_subprocess
• mod_per_interpreter_gil_with_singleton

The test is already a meaningful stress test, rather than a single opportunistic import:

1. check_script_success_in_subprocess() runs its subprocess eight times by default.
2. Each subprocess creates an InterpreterPoolExecutor(max_workers=16).
3. It submits 32 calls which concurrently import mod_per_interpreter_gil_with_singleton and inspect objects registered in its singleton. The module is explicitly declared with py::multiple_interpreters::per_interpreter_gil().
4. The pytest locals showed the helper loop variable as _ = 3, so three complete subprocess repetitions succeeded and the fourth failed.

The core of each subprocess is:

from concurrent.futures import InterpreterPoolExecutor, as_completed

with InterpreterPoolExecutor(max_workers=16) as executor:
    futures = [executor.submit(test) for _ in range(32)]
    for future in as_completed(futures):
        future.result()

Here test() imports mod_per_interpreter_gil_with_singleton; the import failed in one worker with the diagnostic above. A useful initial reproduction approach is therefore to build the normal test modules on 64-bit Windows with CPython 3.14 and repeatedly run only:

test_multiple_interpreters.py::test_import_in_subinterpreter_concurrently

Increasing the helper's rerun value provides a straightforward way to raise the stress level without changing the concurrency pattern that produced the failure.

Why this is significant

At the failing commit, get_pp() is explicitly documented with "Will never return nullptr":

• internals_pp_manager::get_pp()
• The guard that detected the null return

The null guard was added in #6018 as diagnostic hardening motivated by the unexplained Windows crashes in #5993. It turned this occurrence into a useful ImportError; without the guard, dereferencing the null pointer could instead have crashed. The context in #5993 was different, so this issue should not assume the two problems have the same cause.

This also overlaps directly with #5947, "Fix concurrency consistency for internals_pp_manager under multiple-interpreters." During development of that PR, commit 4ef8b0c disabled the manager's caches and always retrieved the state-dictionary entry. That experiment was reverted in 49952a8 before the final solution was merged. Repeating that experiment is a useful diagnostic comparison, although it is not necessarily the final design.

Initial code analysis and a leading hypothesis

The following is a plausible race to investigate, not a demonstrated root cause.

At the failing commit, ensure_internals() does this in order:

get_internals_pp_manager().unref();
if (PyInterpreterState_Get() != PyInterpreterState_Main()) {
    has_seen_non_main_interpreter() = true;
}
get_internals();

Before has_seen_non_main_interpreter() becomes true, get_pp() and unref() both use the shared, non-atomic internals_singleton_pp_ member. With independent interpreter GILs, two module-initialization threads may execute those paths concurrently. The one-way mode transition is not visibly synchronized with access to that raw shared member. One possible interleaving is:

1. Thread A observes has_seen_non_main_interpreter() == false in get_pp() and prepares to return internals_singleton_pp_.
2. A non-main-interpreter thread B enters ensure_internals(), still observes the flag as false in unref(), and clears internals_singleton_pp_.
3. Thread A returns the now-null shared member.

Publishing has_seen_non_main_interpreter() = true before calling unref() in a non-main interpreter would cause that unref() to clear only the thread-local cache instead of the shared singleton. That is a focused first experiment, but it needs Windows stress testing and an audit of initialization/finalization behavior before being considered a fix.

A second path worth auditing is the thread-local cache update in get_pp():

last_istate_tls() = tstate->interp;
internals_p_tls() = get_or_create_pp_in_state_dict();

If the second operation can throw after the interpreter identity has been cached, a later call in the same interpreter could skip cache initialization and return a null internals_p_tls(). The observed log did not show an earlier underlying exception, so this is only another invariant to check.

Suggested investigation sequence

1. Reproduce with the existing focused test on the recorded Windows/Python environment and a larger subprocess repetition count.
2. Test publishing has_seen_non_main_interpreter() before unref() as a narrowly targeted experiment.
3. Compare against Fix concurrency consistency for internals_pp_manager under multiple-interpreters #5947's always-read-the-state-dictionary implementation to determine whether the singleton/TLS cache transition is essential to the failure.
4. Add temporary tracing or assertions around the flag, current interpreter, last_istate_tls(), internals_p_tls(), and internals_singleton_pp_ if reproduction remains intermittent.
5. Keep the existing test strict. Its musllinux xfail covers a different known failure and should not be generalized to Windows.

Regression status is unknown: this was observed once on current master, and the exact rerun passed.

[rwgk]

@b-pass codex analyzed this by chance; the test happened to flake while I was targeting a systematic bit rot issue. While I had the analysis already, I decided to track it here. If you get a chance to work on a fix, that'd be awesome.