finalize the Python interpreter on shutdown

[jvantuyl]

Okay, so this is way bigger than I expected it to be. Sorry about that. I tried splitting it into a smaller chunks but I couldn't figure out a way to do this more incrementally.

I left out some stuff that supports using this with ExUnit sanely, so let me know if you want to see that.

No worries if you don't accept it. This was born in a fit of rage hyperfocus after spending a week tracking down a really subtle issue that it fixes. I wrote it, so I figured I'd at least send it your way.

Finalization

When a normal Python interpreter exits, it runs a shutdown sequence called "finalization". Pythonx never ran it, so the Python shutdown sequence never happened, which causes subtle problems. Examples include, but are certainly not limited to:

• files created with tempfile are not deleted
• shelve changes not persisted
• logging doesn't emit all logged messages
• readline history is not written
• asyncio thread pools don't drain or log some warnings
• generator context managers don't run their exit hooks
• suspended async with statements don't run their exit hooks
• unclosed files and sockets linger until the BEAM VM exits
• leaked multiprocessing semaphores

That last one was triggered in the wild by tqdm when using pytorch in a Livebook, and led me down the rabbit hole that resulted in this code.

Usage

In normal applications where OTP is running things, you don't have to do anything. A new GenServer (Pythonx.Finalizer) triggers finalization when the application is stopped as part of a normal shutdown (i.e. System.stop/0).

It logs a warning if Py_FinalizeEx returns non-zero. Its shutdown is :infinity: Py_FinalizeEx can take arbitrarily long (a loaded ML framework easily exceeds the 5s default), and giving up early would leave the NIF running on a dirty scheduler while the VM halts underneath it.

In environments without a supervision tree (scripts, Mix tasks), call Pythonx.finalize/0 manually. See its documentation for details.

Options

Two new compile-time config options:

• finalization: enables or disables automatic finalization; manual finalization is always available (boolean; default: true)
• binaries: how strings and bytes are decoded. :fast returns zero-copy references into Python memory, which can crash if accessed after finalization; :safe always copies (default: :fast, the existing behavior)

Implementation

OTP Supervision

Pythonx.Finalizer is the last child in the app supervision tree, so it is stopped first and the Janitor is available throughout finalization. Initialization could live there too, but it comes in several flavors with their own side effects, so I left it alone.

Thread Handling & State

CPython binds the thread that runs Py_InitializeEx to runtime->main_tstate and requires Py_FinalizeEx to run on that same thread. When called from any other thread, CPython 3.13 does not swap the main thread state in, frees the calling thread's state as a "non-main" one, and then dereferences it while flushing std files.

NIFs run on whichever dirty scheduler picks them up, so on any machine with more than one dirty CPU scheduler, finalizing after evals had spread across schedulers segfaulted deterministically.

To address this, Pythonx now owns one native thread (pythonx-main) for exactly those two calls. init starts it and runs Py_InitializeEx there, keeping that thread state aside; finalize runs PyEval_RestoreThread + Py_FinalizeEx on it and joins it, so each re-initialization gets a fresh main thread.

Everything that needs the NIF env stays on the dirty schedulers, which get their own thread states as before. Python sees this thread as threading.main_thread(); evals show up as Dummy-N threads.

The MainThread object is intentionally leaked: if the process exits without finalizing (finalization: false, System.halt/1), destroying a joinable std::thread during static destruction would call std::terminate and abort the VM on its way out.

Reinitializing

Finalize + re-init works for the interpreter and pure-Python code, including the standard library (threading, logging, atexit), with CPython's documented per-cycle leaks. C extensions with process-global state are not guaranteed to survive, because CPython never unloads their shared libraries: numpy 2.1.2 raises a clean RuntimeError on second import, and torch 2.14.0 segfaults inside its module init. This is documented on Pythonx.finalize/0.

Every PyObjectResource is stamped with an init_generation, a steady_clock timestamp set each time the interpreter is initialized. Using or cleaning up a resource from a previous generation fails with a clear error instead of following a dangling pointer. A timestamp rather than a counter makes ordering obvious and junk values recognizable.

The public Pythonx.finalize/0 refuses to run while the application is running, since finalization is the Finalizer's job. Pythonx.__finalize__/0 (undocumented) bypasses the guard for the Finalizer and tests.

Concurrency Handling

Three coordinating structures work together:

• init_mutex (existing, prevents racing initialization)
• PyGILGuard (existing, guards Python data structure access)
• active_threads_mutex (new, guards an active_threads counter)

We cannot take the PyGILGuard during finalization, since the GIL is torn down while we hold it. Instead, active_threads counts NIFs currently running Python code and finalize waits for it to reach zero, mimicking CPython's own wait for threads.

Four operations could race with finalization:

• rogue Python code running somewhere unsupervised
• attempts to run new Python code
• garbage collection of a PyObjectResource triggering a decref
• send_tagged_object during finalization

ensure_initialized now returns an ActiveThreadGuard by value and every NIF holds it in a local for its whole duration. The increment happens under init_mutex, which finalize holds throughout, so new code either sees an initialized interpreter and is counted before finalization can start, or blocks until finalization is done and then raises on an uninitialized interpreter.

The Janitor Process During Finalization

The finalize NIF sends :finalizing to the Janitor before waiting for in-flight threads and :finalized after Py_FinalizeEx, both while still holding init_mutex. Between them, decref messages are ignored: nothing is running, and Py_FinalizeEx reclaims the objects regardless of refcount. Since the BEAM delivers local messages in order, :finalized arrives before any decref from a newly initialized interpreter. I/O is still forwarded the whole time, since finalizers may print.

In-flight decref calls block on the GIL, so janitor_decref also holds an ActiveThreadGuard and finalize waits for them too.

Finalization runs Python code, which may call pythonx.send_tagged_object. Anything it references is about to be reclaimed, so it raises a RuntimeError during finalization instead of sending.

Binaries & Python-allocated Memory

:fast decoding returns resource binaries that reference Python memory directly. After finalization those references are freed memory. The binaries option selects copying (:safe) instead. Only bytes and str are affected; no other type is zero-copy.

PyObjectResource instances are counted, and the Finalizer logs a warning if any survive finalization while binaries is :fast. :fast stays the default because it is the existing behavior.