Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,44 @@
# Changelog

## 4.2.0 (2026-08-29)

### Added

- **`isolated` context mode** - `py_context:new(#{mode => isolated})` runs
CPython in a child OS process per context, with the same `call/eval/exec`,
callback, `erlang.send`/`whereis`, worker-loop and pool API as the embedded
modes. It is the first mode with a hard bound: `py_context:interrupt/1`
stops a blocking C call (a signal in the child) and `SIGKILL` is the
backstop after `kill_after` ms; `py_context:kill/1` kills at once. `rlimits`
(`as`, `cpu`, `nofile`) and a cgroup v2 directory bound the child; a
segfault in a C extension returns `{error, {child_exited, {signal, 11}}}`
and the node survives. The child restarts on crash within a budget
(`restart`, `max_restarts`, `restart_period`); `py_context:child_info/1`
reports its OS pid. Children are reaped by the VM and exit when the BEAM
dies (socket EOF watchdog, `PR_SET_PDEATHSIG` on Linux, `PROC_PDEATHSIG_CTL`
on FreeBSD). `cgroup` is refused outside Linux; rlimits apply everywhere:
`as` is kernel-enforced on Linux and FreeBSD and enforced by an RSS
watchdog in the child on macOS (`{child_exited, {memory_limit, Bytes}}`).
Validated on macOS (arm64) and FreeBSD 14.3 (OTP 28, Python 3.11).
- **`py_context:pass_fd/2`** - hands a file descriptor to an isolated child
over the control socket (`SCM_RIGHTS`), so `erlang.server.serve` works out
of process: Erlang binds once, N killable children accept.
- **Pure-Python ETF codec** (`priv/_erlang_impl/_etf.py`) with the type
mapping of `py_convert.c`; the child needs no C extension. Integers beyond
64 bits round-trip exactly in isolated mode.
- `py:python_executable/0`, `py:kill/1`, `py_nif:os_kill/2`.
- `py_isolated` is a `gen_statem` (states `idle`, `{busy, Id}`, `looping`,
`stopping_loop`, `{restarting, Reason}`): `sys:get_state/1` and
`sys:trace/2` work on isolated contexts, requests arriving during a
restart are served by the new child, and `py_context:kill/1` returns once
the new child is up.
- Timeouts on an isolated context cancel their own request only (queued
requests are dropped, the executing one is interrupted); the kill backstop
is bound to that request, so a busy shared context is never killed because
another caller gave up. Soak-tested: callback storms, interrupt/kill
storms, loop churn, 60 s mixed workload with resource counters checked.
- Guide: `docs/isolated.md`, with what each of the three modes guarantees.

## 4.1.0 (2026-08-15)

### Added
Expand Down
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -603,6 +603,7 @@ When creating Python contexts, you can choose the execution mode:
|------|----------------|-------------|
| `worker` | Any | Dedicated pthread per context, main interpreter namespace (default) |
| `owngil` | 3.14+ | Dedicated pthread + subinterpreter with its own GIL, true parallelism |
| `isolated` | Any | CPython in a child OS process: killable, rlimit-bounded, crash-contained |

```erlang
%% Default: worker mode (recommended)
Expand All @@ -612,8 +613,18 @@ When creating Python contexts, you can choose the execution mode:
%% OWN_GIL mode for true parallelism (Python 3.14+ required)
%% Each context runs in its own pthread with independent GIL
{ok, Ctx} = py_context:new(#{mode => owngil}).

%% Isolated mode: a child process per context. A stuck call is killed, a
%% segfault only takes the child down, rlimits bound memory and CPU.
{ok, Ctx} = py_context:new(#{mode => isolated, kill_after => 1000,
rlimits => #{as => 512 * 1024 * 1024}}).
```

**Isolated mode** is the only mode with a hard bound: `py_context:interrupt/1`
stops a blocking C call, and `SIGKILL` is the backstop. It costs a process per
context (about 16 MB and 40 ms to start) and roughly twice the call latency.
See [Isolated Contexts](docs/isolated.md).

**Worker mode is recommended** because it works with any Python version and automatically benefits from free-threaded Python (3.13t+) when available. Each context owns a dedicated pthread, providing stable thread affinity for libraries with thread-local state (numpy, torch, tensorflow).

**Why OWN_GIL requires Python 3.14+**: Some C extensions (e.g., `_decimal`, `numpy`) have global state bugs in sub-interpreters on Python 3.12/3.13. These are fixed in Python 3.14.
Expand All @@ -629,6 +640,7 @@ py:execution_mode(). %% => worker | owngil
|------|----------------|-------------|
| `worker` (default) | Any | One pthread per context; true parallelism on free-threaded 3.13t+ |
| `owngil` | 3.14+ | Per-interpreter GIL, true parallelism across contexts |
| `isolated` | Any | One OS process per context, parallel and failure-isolated |

## Error Handling

Expand All @@ -651,6 +663,7 @@ py:execution_mode(). %% => worker | owngil
- [Logging and Tracing](docs/logging.md)
- [Asyncio Event Loop](docs/asyncio.md) - Erlang-native asyncio with TCP/UDP support
- [Worker Loops](docs/workers.md) - Long-lived loops in owngil contexts, serving on sockets Erlang owns
- [Isolated Contexts](docs/isolated.md) - Python in a child process: kill, rlimits, crash containment
- [Reactor](docs/reactor.md) - FD-based protocol handling
- [Security](docs/security.md) - Sandbox and blocked operations
- [Changelog](https://github.com/benoitc/erlang-python/releases)
Expand Down
29 changes: 29 additions & 0 deletions c_src/py_nif.c
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,14 @@
* - py_callback.c: Callback system and asyncio support
*/

/* pthread_timedjoin_np (used to bound the owngil worker join on Linux)
* is declared by <pthread.h> only under _GNU_SOURCE. */
#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif

#include <signal.h>
#include <errno.h>
#include "py_nif.h"
#include "py_util.h"
#include "py_event_loop.h"
Expand Down Expand Up @@ -8079,6 +8087,26 @@ static void unload(ErlNifEnv *env, void *priv_data) {
/* Other cleanup handled by finalize */
}

/**
* @brief Send a signal to an OS process (kill(2)).
*
* Used by isolated contexts to SIGKILL their child. The caller holds the
* child's port open until exit_status arrives, so the pid cannot have been
* recycled.
*/
static ERL_NIF_TERM nif_os_kill(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) {
(void)argc;
int pid, sig;
if (!enif_get_int(env, argv[0], &pid) || !enif_get_int(env, argv[1], &sig) || pid <= 0) {
return enif_make_badarg(env);
}
if (kill((pid_t)pid, sig) == 0) {
return ATOM_OK;
}
return enif_make_tuple2(env, ATOM_ERROR,
enif_make_atom(env, errno == ESRCH ? "esrch" : errno == EPERM ? "eperm" : "einval"));
}

static ErlNifFunc nif_funcs[] = {
/* Initialization */
{"init", 0, nif_py_init, 0},
Expand Down Expand Up @@ -8220,6 +8248,7 @@ static ErlNifFunc nif_funcs[] = {
{"create_test_pipe", 0, nif_create_test_pipe, 0},
{"close_test_fd", 1, nif_close_test_fd, 0},
{"dup_fd", 1, nif_dup_fd, 0},
{"os_kill", 2, nif_os_kill, 0},
{"write_test_fd", 2, nif_write_test_fd, 0},
{"read_test_fd", 2, nif_read_test_fd, 0},
/* TCP test helpers */
Expand Down
19 changes: 18 additions & 1 deletion docs/interrupts.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,29 @@ context to deal with that:
ok = py_context:destroy(Ctx).
```

## Interrupting a blocking C call: isolated mode

The limits below apply to the embedded modes. An `isolated` context runs
Python in a child process, where an interrupt is a signal that lands inside
`time.sleep`, a socket read or any other blocking call, and `SIGKILL` is the
backstop if the signal is ignored:

```erlang
{ok, Ctx} = py_context:new(#{mode => isolated, kill_after => 1000}),
{error, timeout} = py_context:eval(Ctx, <<"__import__('time').sleep(60)">>, #{}, 200),
%% Usable at once, no 60 s wait
{ok, 4} = py_context:eval(Ctx, <<"2+2">>, #{}, 5000).
```

`py_context:kill/1` kills at once. See [Isolated Contexts](isolated.md).

## Limits

- CPython delivers an async exception at the next bytecode boundary. Code
blocked inside a C call (`time.sleep`, a numpy kernel, a socket read) is
not interrupted until that call returns. The call still times out on the
Erlang side; the context becomes usable once the C call finishes.
Erlang side; the context becomes usable once the C call finishes. Only
`isolated` mode interrupts such a call.
- An interrupt targets the context, not an individual request. Interrupting a
context that just finished one call and started another stops the new one.
- `py:call/3,4` and `py:eval/1,2` use `infinity` by default. Pass an explicit
Expand Down
Loading
Loading