From c8f77f02587298e4416c17d140f6801d2929dc05 Mon Sep 17 00:00:00 2001 From: Benoit Chesneau Date: Sat, 29 Aug 2026 14:41:17 +0200 Subject: [PATCH] Add architecture, code map and glossary A map of the code: processes and threads, the life of a call in each context mode, the paths Python uses to call Erlang, which code is live and which is legacy, and one meaning per overloaded word (worker, context, pool). The stale file list in the NIF headers now points at these pages. Also completes the 4.2.0 changelog. --- CHANGELOG.md | 8 +- README.md | 1 + c_src/README.md | 71 +++++++++++++ c_src/py_nif.c | 9 +- c_src/py_nif.h | 61 +++-------- docs/architecture.md | 198 ++++++++++++++++++++++++++++++++++++ docs/code-map.md | 98 ++++++++++++++++++ docs/glossary.md | 109 ++++++++++++++++++++ priv/_erlang_impl/README.md | 29 ++++++ rebar.config | 6 ++ 10 files changed, 536 insertions(+), 54 deletions(-) create mode 100644 c_src/README.md create mode 100644 docs/architecture.md create mode 100644 docs/code-map.md create mode 100644 docs/glossary.md create mode 100644 priv/_erlang_impl/README.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f042595..a143464 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,7 +34,8 @@ such a region with ring backpressure, usable as `wsgi.input` in isolated contexts. Handles are plain terms and travel inside any argument or result; `py_shm:read_only/1` and `new(Size, #{writable => false})` hand Python a - read-only mapping. + read-only mapping. `py_buffer:write/3` takes a timeout for the case where + the ring is full and nobody reads (default 30 s). - `py:python_executable/0`, `py:kill/1`, `py_nif:os_kill/2`. - `py_isolated` is a `gen_statem` (states `idle`, `{busy, Id}`, `looping`, `stopping_loop`, `{restarting, Reason}`): `sys:get_state/1` and @@ -48,6 +49,11 @@ storms, loop churn, 60 s mixed workload with resource counters checked. - Guide: `docs/isolated.md`, with what each of the three modes guarantees. +### Fixed + +- `pthread_timedjoin_np` was called without `_GNU_SOURCE`, an implicit + declaration on Linux that newer compilers reject. + ## 4.1.0 (2026-08-15) ### Added diff --git a/README.md b/README.md index aa550d5..6045ac8 100644 --- a/README.md +++ b/README.md @@ -654,6 +654,7 @@ py:execution_mode(). %% => worker | owngil ## Documentation +- [Architecture](docs/architecture.md), [Code map](docs/code-map.md), [Glossary](docs/glossary.md) - how the pieces fit - [Getting Started](docs/getting-started.md) - [Process-Bound Environments](docs/process-bound-envs.md) - Isolated Python state per Erlang process - [AI Integration Guide](docs/ai-integration.md) diff --git a/c_src/README.md b/c_src/README.md new file mode 100644 index 0000000..a8f240e --- /dev/null +++ b/c_src/README.md @@ -0,0 +1,71 @@ +# c_src + +The NIF that embeds CPython in the VM. One translation unit: `py_nif.c` +`#include`s every other `.c` file (see the "Include module implementations" +section near line 265), so build with `rebar3 compile` (CMake through +`do_cmake.sh` / `do_build.sh`), never a single file. Headers declare what +the included files share. + +Read `docs/architecture.md` first for how a call travels; this file says +where things are. + +## Files + +| File | What it owns | Notes | +|---|---|---| +| `py_nif.h` | All shared types: `py_context_t` and its request queue, request types, callback and suspension state, runtime state machine, atoms, globals, declarations | 2.4k lines. The struct comments carry the locking rules; read `py_context_t` before touching threads | +| `py_nif.c` | Runtime init and finalize, resource types, context create/destroy, the request queue, `worker_context_thread_main` and `owngil_context_thread_main`, `owngil_execute_*` (used by both thread kinds), the `nif_context_*` NIFs, process-local envs, `py_ref`, the NIF function table at the end | Sections are banner-separated; `grep -n '^ \* ===\|^/\* ==='` lists them | +| `py_convert.c` | `py_to_term`, `term_to_py`, depth limits, tagged tuples (`{bytes, B}`, `{'$py_shm', ...}`), error tuples `{error, {Type, Msg}}` | The type mapping tables in the comments are the reference for `_etf.py` | +| `py_exec.c` | Executing a call/eval/exec with suspension support; the legacy single executor thread | | +| `py_callback.c` | The `erlang` Python module: `call`, `send`, `whereis`, `schedule*`, `Atom`/`Pid`/`Ref` types, callback delivery paths (suspension, blocking pipe, async pipe), channel and shared-dict methods, callback name registry | `erlang_call_impl` documents the path precedence | +| `py_thread_worker.c` | Python threads calling Erlang through `py_thread_handler` | | +| `py_subinterp_thread.c/.h` | Thread pool of sub-interpreters (owngil contexts, loop pools) | | +| `py_event_loop.c/.h` | `ErlangEventLoop` support: `enif_select` readers and writers, timers, task injection into loops, reactor dispatch, fd registry, Python module `py_event_loop`; plus test-only fd/TCP/UDP NIFs (section "Test Helper Functions") | Largest file | +| `py_channel.c/.h`, `py_buffer.c/.h`, `py_reactor_buffer.c/.h`, `py_shared_dict.c` | Resources with a Python-facing object each | | +| `py_logging.c` | Logging and tracing NIFs | | +| `py_mem_limit.c` | obmalloc arena accounting for owngil memory caps | | +| `py_worker_pool.c/.h` | Legacy pool, no caller in `src/` | Candidate for removal | +| `py_util.c/.h` | Macros, small helpers | | + +## Where the live paths are + +- `py:call/3` in worker or owngil mode: `nif_context_call_async` (`py_nif.c`) + enqueues; `worker_context_thread_main` or `owngil_context_thread_main` + dequeues and calls `owngil_execute_request`; the reply goes out as + `{py_result, Ref, Result}`. +- `erlang.call` from Python: `erlang_call_impl` (`py_callback.c`). +- Interrupt: `nif_context_interrupt` (`py_nif.c`), `interrupt_mutex` rules on + `py_context_t`. +- Type conversion: `py_to_term` / `term_to_py` (`py_convert.c`). +- Isolated mode has no C code of its own: `os_kill` is the only NIF it uses. + +## Rules that are easy to break + +- Only the context's thread touches the context's Python objects. NIFs + called from Erlang processes enqueue requests and return; they do not run + Python for a context that has a thread. +- `Py_BEGIN_ALLOW_THREADS` around every blocking wait; never block on an + Erlang-side resource with the GIL held. +- Never call a Future method while holding `async_futures_mutex` + (`py_callback.c`, comment above the struct explains the pattern). +- `interrupt_mutex` is taken only by threads that do not hold the GIL. +- `queue_mutex` protects the request queue of a context and is taken before + a request's own mutex (`ctx_queue_cancel_all`), never the other way. +- A NIF that can block or run Python is registered with a dirty scheduler + flag in the table at the end of `py_nif.c`. + +## Adding a NIF + +1. Implement `static ERL_NIF_TERM nif_x(ErlNifEnv*, int, const ERL_NIF_TERM[])` + next to related code. +2. Add `{"x", Arity, nif_x, Flags}` to `nif_funcs[]` at the end of `py_nif.c`. +3. Add the stub and its `-spec` and doc to `src/py_nif.erl`. +4. Cover it in a suite; `rebar3 dialyzer` and `rebar3 xref` must stay clean. + +## Legacy code, for orientation + +Not on the path of contexts created today: the `worker_*` NIFs and +"Worker management" section, `async_worker_*` NIFs (return `deprecated`), +the "Legacy mode" inline branches in `nif_context_call/eval/exec`, +`py_worker_pool.c`, and the `cancel_reader/writer` aliases. When in doubt, +follow `nif_context_call_async` and ignore the rest. diff --git a/c_src/py_nif.c b/c_src/py_nif.c index cb8f459..83f83df 100644 --- a/c_src/py_nif.c +++ b/c_src/py_nif.c @@ -29,11 +29,10 @@ * - Resource types for Python objects to ensure proper cleanup * - Dirty NIF flags for GIL-holding operations * - * This file is the main entry point. It includes the following modules: - * - py_nif.h: Shared header with types and declarations - * - py_convert.c: Type conversion (Python <-> Erlang) - * - py_exec.c: Python execution and GIL management - * - py_callback.c: Callback system and asyncio support + * This file is the main entry point and the single translation unit: it + * includes the other .c files (see "Include module implementations"). The + * file map is c_src/README.md; the request lifecycle per context mode is + * docs/architecture.md. */ /* pthread_timedjoin_np (used to bound the owngil worker join on Linux) diff --git a/c_src/py_nif.h b/c_src/py_nif.h index 4f2e37c..45020cb 100644 --- a/c_src/py_nif.h +++ b/c_src/py_nif.h @@ -21,54 +21,19 @@ * * @mainpage Python-Erlang NIF Integration * - * @section intro_sec Introduction - * - * This NIF (Native Implemented Function) library provides seamless integration - * between Erlang/OTP and Python. It embeds a Python interpreter within the - * Erlang VM and provides bidirectional communication capabilities. - * - * @section arch_sec Architecture - * - * The implementation follows a modular design with four main components: - * - * - **py_nif.h** - Shared types, macros, and declarations - * - **py_convert.c** - Bidirectional type conversion (Python ↔ Erlang) - * - **py_exec.c** - Python execution engine and GIL management - * - **py_callback.c** - Erlang callback support and asyncio integration - * - * @section modes_sec Execution Modes - * - * The library supports three execution modes based on Python version: - * - * | Mode | Python Version | Description | - * |------|----------------|-------------| - * | FREE_THREADED | 3.13+ (no-GIL) | Direct execution without GIL | - * | SUBINTERP | 3.12+ | Per-interpreter GIL isolation | - * | MULTI_EXECUTOR | Any | Multiple executor threads with GIL | - * - * @section gil_sec GIL Management - * - * The GIL (Global Interpreter Lock) is managed following PyO3/Granian patterns: - * - * - `Py_BEGIN_ALLOW_THREADS` / `Py_END_ALLOW_THREADS` around blocking ops - * - Executor threads hold the GIL and process queued requests - * - Dirty I/O schedulers are used for Python-calling NIFs - * - * @section callback_sec Callback Mechanism - * - * Python code can call back to Erlang using a suspension/resume pattern: - * - * 1. Python calls `erlang.call('func', args)` - * 2. NIF raises `SuspensionRequired` exception - * 3. Dirty scheduler is released, callback sent to Erlang - * 4. Erlang processes callback, calls `resume_callback/2` - * 5. Python execution resumes with cached result - * - * @section mem_sec Memory Management - * - * - Erlang resources wrap Python objects (prevent GC) - * - Thread-local storage for callback context - * - Proper cleanup in resource destructors + * This NIF embeds CPython in the Erlang VM. The map of the code, the life + * of a call in each context mode, the callback paths and the locking rules + * are documented in docs/architecture.md, c_src/README.md and + * docs/glossary.md; keep those current instead of this comment. + * + * In one paragraph: py_nif.c is the single translation unit and includes + * the other .c files; a context (py_context_t) owns a request queue and a + * pthread that runs Python for it (worker mode: main interpreter, shared + * GIL; owngil mode: a sub-interpreter with its own GIL); Erlang enqueues + * through nif_context_call_async and receives {py_result, Ref, Result}; + * Python calls Erlang through erlang_call_impl (py_callback.c), by + * suspension in worker mode and a blocking pipe in owngil mode. Isolated + * mode runs Python in a child process and uses no C code beyond os_kill. */ #ifndef PY_NIF_H diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..53e6426 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,198 @@ +# Architecture + +This page is the map of erlang_python. It +says which processes and threads exist, how a call travels from `py:call/3` +to Python and back in each context mode, how Python calls Erlang, and which +code paths are live. Read it before opening `c_src/` or `src/py_context.erl`. +The [code map](code-map.md) lists every file; the [glossary](glossary.md) +defines the overloaded words (worker, context, pool). + +## One picture + +``` +Erlang VM child OS process ++--------------------------------------------------+ (isolated mode only) +| erlang_python_sup | +| py_callback registry of Erlang funs Python | +| may call | +| py_shm shared memory regions | +| py_thread_handler spawns a handler process | +| per Python thread that calls | +| Erlang | +| py_logger / py_tracer Python logging, tracing | +| py_context_sup ---- py_context (one per | +| context; mode worker | owngil | isolated) | +| py_context_init starts the default pool | +| py_event_worker_sup / _registry loop drivers | +| py_event_loop, py_event_loop_pool main- | +| interpreter asyncio loops | ++--------------------------------------------------+ + | NIF calls (dirty schedulers) | Unix socket, ETF frames + v v ++------------------------+ +-------------------------+ +| libpython in the VM | | python3 py_isolated_ | +| main interpreter | | child.py | +| worker contexts: one | | reader thread + main | +| pthread each, shared | | thread, own asyncio | +| GIL | | loop | +| owngil contexts: one | +-------------------------+ +| pthread + own sub- | +| interpreter each | ++------------------------+ +``` + +A **context** is the unit of work: an Erlang process (`py_context`) that owns +one Python execution environment and serves calls in order. Pools +(`py_context_router`) route `py:call/3` to a context by scheduler affinity. + +## The three context modes + +| | `worker` | `owngil` | `isolated` | +|---|---|---|---| +| Python runs in | the VM, main interpreter | the VM, a sub-interpreter with its own GIL | a child process | +| Thread | one pthread per context (`worker_context_thread_main`) | one pthread per context (`owngil_context_thread_main`) | the child's main thread | +| Erlang process loop | the receive loop in `py_context` | the receive loop in `py_context` | `py_isolated` (`gen_statem`) | +| Transport | NIF request queue on `py_context_t` | same | Unix socket, frames of the callback pipe format | +| Python -> Erlang | suspension protocol | blocking callback pipe | socket frames | +| Interrupt | `PyThreadState_SetAsyncExc`, next bytecode | same | signal in the child, `SIGKILL` backstop | +| Guide | [context-affinity](context-affinity.md), [pools](pools.md) | [owngil_internals](owngil_internals.md) | [isolated](isolated.md) | + +## Life of a call, per mode + +### worker and owngil (embedded) + +1. `py:call(M, F, A)` picks a context through `py_context_router` and sends + `{call, From, MRef, M, F, Args, Kwargs}` to the `py_context` process + (`src/py.erl`, `src/py_context.erl:call/6`). The caller waits in + `await_reply/3`; a timeout there calls `interrupt/1`. +2. The `py_context` receive loop takes it and calls `handle_call_with_suspension/5`, + which calls the `context_call_async` NIF + (`c_src/py_nif.c`, `nif_context_call_async`). The NIF converts the + arguments (`term_to_py`, `c_src/py_convert.c`) into a request, enqueues it + on the context's queue (`ctx_queue_enqueue`) and returns `{enqueued, Ref}` + at once. The Erlang process is now free to serve callbacks. +3. The context's pthread (`worker_context_thread_main` or + `owngil_context_thread_main`, `c_src/py_nif.c`) dequeues the request and + runs it through `owngil_execute_request` (despite its name it serves both + modes), which calls into Python with the GIL held. +4. The thread converts the result (`py_to_term`) and sends + `{py_result, Ref, Result}` to the `py_context` process, which replies + `From ! {MRef, Result}`. + +The older paths in `nif_context_call` (a blocking variant with an inline +"legacy" executor) are kept for the fallback +`{error, async_requires_worker_thread}` and are not taken by contexts +created today; see [code map](code-map.md) for the list of legacy code. + +### isolated + +1. Same first step: the message reaches the `py_context` pid, which in this + mode runs `py_isolated` (a `gen_statem` entered with `enter_loop`). +2. In `idle` the request is encoded with `term_to_binary` into a frame + `<>` and written to + the Unix socket; the state becomes `{busy, Id}` and other callers' + requests are postponed (served in order when the child is free). +3. In the child, the reader thread parses frames and queues the request; + the main thread runs it (`_erlang_impl/_isolated.py`, `Runtime._dispatch`) + and writes the reply frame. +4. Back in `py_isolated`, the reply for the busy id moves the state to + `idle` and answers the caller. A crash of the child is seen as the port's + `exit_status`; the state goes through `{restarting, Reason}` and a new + child is started within the restart budget. + +Protocol details: the module header of `src/py_isolated.erl` and the +docstring of `priv/_erlang_impl/_isolated.py`. + +## Python calling Erlang (`erlang.call`) + +`erlang_call_impl` in `c_src/py_callback.c` chooses one of these paths, in +this order (the comment above it is the authoritative version): + +1. **Suspension** (worker contexts). The Python call raises + `SuspensionRequired`; the context thread returns + `{suspended, CallbackId, State, {Name, Args}}` to `py_context`, which runs + the registered fun (`execute/2` in `py_callback`), possibly serving nested calls + meanwhile (`wait_for_callback/2`), and resumes with + the `resume_callback` NIF. +2. **Blocking callback pipe** (owngil contexts). The context thread writes + a request on a pipe and blocks; the `py_context` process has a dedicated + handler (`callback_handler_loop/1`) that runs the fun and writes the + response frame back with `context_write_callback_response`. +3. **Legacy worker handler** (`worker_*` NIFs): only used by + `examples/gen_test.erl`. +4. **Thread worker** (`c_src/py_thread_worker.c`): any Python thread that is + not a context thread (`threading.Thread`, executors) asks the + `py_thread_handler` coordinator for a handler process and talks to it + over a pipe. There is also an async variant (`erlang.async_call`) using a + per-interpreter async pipe. + +In isolated mode there is one path: a status-3 frame on the socket, answered +by a process the `py_isolated` state machine spawns; nested calls into the +same context are dispatched immediately because they come from that process. + +The frame format shared by the pipe and the socket, and the ETF conventions, +will get their own page (protocols); until then `c_src/py_convert.c` (type +mapping) and `priv/_erlang_impl/_etf.py` are the reference. + +## asyncio + +Three different machineries, on purpose: + +- Embedded contexts share an `ErlangEventLoop` (`priv/_erlang_impl/_loop.py`) + whose `add_reader`/`add_writer` map onto `enif_select` and `call_later` onto + `erlang:send_after`; readiness is delivered to a `py_event_worker` process + per loop (`c_src/py_event_loop.c`, `src/py_event_worker.erl`). Worker loops + (`py_context:start_loop/1`) run such a loop on the context thread. +- `py_event_loop`/`py_event_loop_pool` expose the main-interpreter loops for + `py:async_call/3` and friends. +- An isolated child runs a plain asyncio loop; the only integration is + delivering results over the socket. Nothing of the reactor is ported. + +See [event_loop_architecture](event_loop_architecture.md) for the embedded +loop and [asyncio](asyncio.md) for the API. + +## Data paths that avoid copies + +- `py_buffer` (native): a NIF resource Erlang writes into and Python reads + through the buffer protocol; embedded modes only. +- `py_shm` and `py_buffer:new(#{shared => true})`: a file mapped + `MAP_SHARED` by the VM (through iommap) and by any interpreter, embedded or + child, with flow control through callbacks (`_py_buffer_wait`, + `_py_buffer_consumed`). See [isolated](isolated.md#bulk-data-with-shared-memory). +- `py_channel`, `py_byte_channel`: message queues between Erlang and Python + coroutines, embedded modes only. + +## Where state lives + +| State | Owner | Notes | +|---|---|---| +| Registered callbacks | `py_callback` ETS `py_callbacks` | also mirrored in a C name registry so `erlang.` resolves; `py_state` exposes its store the same way (`erlang.state_get`) | +| Import and path registry | `py_import` ETS | applied to every new interpreter and to isolated children at start | +| Preload code | `py_preload` persistent_term | run once per interpreter | +| Context pid -> NIF ref | `py_context` ETS `py_context_refs` | lets `interrupt/1` reach a context blocked in a NIF; isolated contexts store the atom `isolated` | +| Per-Erlang-process Python env | `py` process dictionary + NIF env resource | `py:call(Ctx, ...)`; not applicable to isolated contexts | +| Shared regions | `py_shm` ETS `py_shm_regions` | closed on owner death | +| Python-side state | the interpreter | lost when an isolated child restarts | + +## Invariants worth knowing before editing + +- A context serves one request at a time. Embedded: the thread dequeues one + request; nested callbacks are served by the `py_context` process while the + thread waits. Isolated: enforced by the `{busy, Id}` state and `postpone`. +- Only the context's own thread touches its Python objects; NIF callers only + enqueue. `py_context_t` fields are documented in `c_src/py_nif.h`. +- Interrupts target the request executing now. Isolated mode also cancels a + queued request by id; embedded modes cannot. +- Everything that crosses the socket is a term; NIF resources (channels, + native buffers, object references) do not cross, and the API says so with + `{error, not_supported_in_isolated}`. +- `binary_to_term` on child data is not `safe`: the child can create atoms. + +## What is live and what is not + +Kept for now, not used by current contexts: the `worker_*` NIF API and its +single executor thread (`c_src/py_exec.c`), the `async_worker_*` NIFs (they +return `deprecated`), `c_src/py_worker_pool.c` (no caller), the inline +"legacy" executor branches in `nif_context_*`, and the test-only fd NIFs in +`c_src/py_event_loop.c`. They are listed in the [code map](code-map.md) so +nobody debugs them by mistake; removing them is planned. diff --git a/docs/code-map.md b/docs/code-map.md new file mode 100644 index 0000000..3139871 --- /dev/null +++ b/docs/code-map.md @@ -0,0 +1,98 @@ +# Code map + +Every source file, what it owns, and where to look for its behaviour. Status +is `live` (on the path of a context created today), `legacy` (kept for +compatibility, no current caller in `src/`), or `test` (only exercised by +suites). Guides are in `docs/`, suites in `test/`. Start with +[architecture](architecture.md). + +## Erlang (`src/`) + +| Module | Owns | Status | Guide | Suites | +|---|---|---|---|---| +| `py` | Public API facade: call/eval/exec, streams, async helpers, venvs, memory, function registration | live | README, getting-started | `py_SUITE`, `py_api_SUITE`, `py_stream_SUITE`, `py_venv_SUITE` | +| `py_context` | The context process for embedded modes and the API every mode answers (`call/eval/exec`, `interrupt`, `kill`, loops, `pass_fd`); dispatch to `py_isolated` for isolated mode | live | context-affinity, workers, interrupts | `py_context_SUITE`, `py_context_process_SUITE`, `py_interrupt_SUITE`, `py_worker_loop_SUITE` | +| `py_isolated` | `gen_statem` driving a child process over the socket; restart policy | live | isolated | `py_isolated_*_SUITE` | +| `py_context_router` | Pools and scheduler-affinity routing | live | pools, context-affinity | `py_context_router_SUITE`, `py_pool_SUITE` | +| `py_context_sup`, `py_context_init` | Supervisor of contexts; starts the default pool at boot | live | pools | (through the above) | +| `py_nif` | Erlang stubs and docs for every NIF | live | api-reference | all | +| `py_callback` | Registry of Erlang funs callable as `erlang.call('name', ...)` | live | README (callbacks) | `py_callback_encoding_SUITE`, `py_thread_callback_SUITE` | +| `py_thread_handler` | Coordinator that gives each Python thread calling Erlang a handler process and a pipe | live | threading | `py_thread_callback_SUITE`, `py_reentrant_SUITE` | +| `py_event_loop` | Main-interpreter asyncio loop: `run`, `create_task`, `await`, and the loop callbacks Python needs | live | asyncio | `py_event_loop_SUITE`, `py_async_task_SUITE` | +| `py_event_loop_pool` | Several main-interpreter loops with process affinity | live | asyncio | `py_event_loop_pool_SUITE` | +| `py_event_worker`, `_sup`, `_registry` | One process per running loop receiving `enif_select` readiness and timers | live | event_loop_architecture | `py_event_loop_SUITE`, `py_fd_ops_SUITE` | +| `py_reactor_context` | FD-owning context for the protocol-based reactor | live | reactor | `py_reactor_SUITE` | +| `py_channel`, `py_byte_channel` | Term and byte queues between Erlang and Python coroutines (NIF resources) | live | channel | `py_channel_SUITE`, `py_byte_channel_SUITE` | +| `py_buffer` | Native streaming input buffer; shared variant delegates to `py_shm` | live | buffer, isolated | `py_buffer_SUITE`, `py_isolated_buffer_SUITE` | +| `py_shm` | Shared memory regions over iommap and the ring behind shared buffers | live | isolated | `py_isolated_shm_SUITE` | +| `py_import` | Registry of imports and `sys.path` entries applied to every interpreter | live | imports | `py_import_SUITE` | +| `py_preload` | Code run once per interpreter at start | live | preload | `py_preload_SUITE` | +| `py_state` | Shared key/value store visible from Python as `erlang.state_get/set/delete/keys` | live | README (shared state) | `py_state_SUITE` | +| `py_semaphore` | ETS counting semaphore for rate limiting | live | scalability | (through `py_SUITE`) | +| `py_logger`, `py_tracer` | Python `logging` into Erlang logger; tracing hooks | live | logging | `py_logging_SUITE` | +| `erlang_python_app`, `erlang_python_sup` | Application start and the supervision tree | live | architecture | all | +| `py_util` | Small helpers | live | | | + +## C (`c_src/`) + +`py_nif.c` is the only translation unit: it `#include`s the other `.c` +files. Editing `py_convert.c` alone does not compile it alone; build with +`rebar3 compile`. See `c_src/README.md`. + +| File | Owns | Status | +|---|---|---| +| `py_nif.h` | Every shared type: `py_context_t`, request types, runtime state machine, atoms, globals | live | +| `py_nif.c` | Runtime init, context creation and destruction, the request queue and the two context thread mains, the process-per-context NIFs (`nif_context_*`), process-local envs, `py_ref`, the NIF table | live, with legacy branches | +| `py_convert.c` | `py_to_term` / `term_to_py`, the type mapping, tagged tuples (`{bytes, B}`, shared handles) | live | +| `py_exec.c` | Execution with suspension support; the legacy single executor thread | live (suspension), legacy (executor) | +| `py_callback.c` | The `erlang` Python module: `call`, `send`, `whereis`, `Atom`/`Pid`/`Ref` types, schedule markers, callback pipes, channel and shared dict methods | live | +| `py_thread_worker.c` | Python threads calling Erlang through `py_thread_handler` | live | +| `py_subinterp_thread.c` | Sub-interpreter thread pool used by owngil contexts and loop pools | live | +| `py_event_loop.c` | `ErlangEventLoop` support: `enif_select` readers/writers, timers, task injection, reactor dispatch, fd registry; also ~570 lines of test-only fd/TCP/UDP NIFs | live; test section | +| `py_channel.c`, `py_buffer.c`, `py_reactor_buffer.c`, `py_shared_dict.c` | The corresponding resources and their Python-facing methods | live | +| `py_logging.c` | Logging and tracing NIFs | live | +| `py_mem_limit.c` | Per-interpreter memory caps (owngil) | live | +| `py_worker_pool.c/.h` | An older worker pool | legacy, no caller | +| `py_util.c/.h` | Macros and helpers | live | + +Inside `py_nif.c`, these are legacy: the `worker_*` NIFs and the "Worker +management" section, the `async_worker_*` NIFs (return `deprecated`), the +inline executor branches marked "Legacy mode" in `nif_context_call`, +`nif_context_eval`, `nif_context_exec`, and the `cancel_reader/writer` +aliases. + +## Python (`priv/`) + +`priv/` is on `sys.path` of every interpreter. `_erlang_impl` is the Python +half of the `erlang` module; the embedded C module delegates to it for the +loop, channels and servers. + +| File | Owns | Used by | +|---|---|---| +| `_erlang_impl/__init__.py` | Public surface of `erlang` in embedded modes: `run`, `sleep`, `spawn_task`, loop policy, `atom`, channels, `server` | embedded | +| `_erlang_impl/_loop.py`, `_policy.py`, `_transport.py` | `ErlangEventLoop` (uvloop-compatible) over `enif_select` | embedded | +| `_erlang_impl/_reactor.py` | Protocol-based reactor over fds Erlang owns | embedded | +| `_erlang_impl/_channel.py`, `_byte_channel.py` | Python side of channels | embedded | +| `_erlang_impl/_server.py` | `serve`, `adopt`, `stop_serving` on fds handed over by Erlang; plain asyncio, works in every mode | all | +| `_erlang_impl/_sandbox.py`, `_subprocess.py` | Audit hook blocking fork/exec inside the VM | embedded | +| `_erlang_impl/_mode.py` | Detects how Python is running (embedded, free-threaded, child) | all | +| `_erlang_impl/_etf.py` | Pure-Python ETF codec with the `py_convert.c` mapping | isolated child | +| `_erlang_impl/_isolated.py` | Child runtime: socket frames, reader thread, re-entrant main loop, interrupt signal, asyncio loop, the `erlang` shim | isolated child | +| `_erlang_impl/_shm.py` | `SharedMemory` and `SharedBuffer` wrappers over mmap | all | +| `py_isolated_child.py` | Child launcher: rlimits, parent-death signal, cgroup join, connect | isolated child | +| `test_erlang_loop.py`, `tests/` | Python-side tests of the loop | test | + +## Tests (`test/`) + +Suites named `py__SUITE`. Cross-mode suites run the same cases in +`worker` and `isolated` groups (`py_isolated_SUITE`, `py_isolated_vm_SUITE`, +`py_isolated_shm_SUITE`, `py_isolated_buffer_SUITE`). Python helpers used by +suites are `test/py_test_*.py`. `test/coverage_audit.md` maps public APIs to +cases. `test/test.config` holds node-wide settings (memory limits flag). + +## Build and docs + +`rebar.config` runs `do_cmake.sh` / `do_build.sh` (CMake in `c_src/`) as +compile hooks; the NIF lands in `priv/py_nif.so`. `make lint-docs` checks +that Erlang snippets in the guides call real exports and that Python +snippets parse. `rebar3 ex_doc` builds the guides listed in `rebar.config`. diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 0000000..cf5864c --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,109 @@ +# Glossary + +The same words mean different things in different files of this project. +This page fixes one meaning per term and says where the other uses come +from, so a reader can translate as they go. + +## Context + +**Context**: one Python execution environment served by one Erlang process +(`py_context`), in order, one request at a time. The unit of `py:context/0`, +pools and modes. In C it is `py_context_t` (`c_src/py_nif.h`); in +`py_context.erl` it is the process; for isolated mode the process runs +`py_isolated` and the environment is a child OS process. + +Other uses: `py_reactor_context` is a context that also owns file +descriptors for the reactor; "coordinator context" in C comments means the +`py_thread_handler` side of the thread-worker channel. + +## Mode + +**Mode**: how a context runs Python. `worker` (main interpreter, one pthread +per context, shared GIL), `owngil` (a sub-interpreter with its own GIL per +context, one pthread), `isolated` (a child process). `py_context:new(#{mode => ...})`. + +Related flags on `py_context_t`: `uses_worker_thread` (has its own pthread; +true for worker and owngil contexts created today), `is_subinterp` (has its +own sub-interpreter), `uses_own_gil` (that sub-interpreter has its own GIL). +`subinterp` in file and NIF names (`py_subinterp_thread.c`, +`subinterp_supported/0`) refers to the machinery owngil mode is built on; +there is no separate "subinterp mode" any more. + +The runtime-wide `PY_MODE_FREE_THREADED` / `PY_MODE_GIL` in `py_nif.h` is +about the Python build (free-threaded or not), not about contexts. + +## Worker + +The most overloaded word. Meanings, by file: + +| Where | Meaning | Prefer to say | +|---|---|---| +| `py_context:new(#{mode => worker})` | the context mode above | worker mode | +| `worker_context_thread_main`, `uses_worker_thread` (`py_nif.c`) | the pthread that serves a context's queue | context thread | +| `worker_new/call/eval/exec` NIFs, `py_worker_t` | the legacy per-worker API, before contexts | legacy worker API | +| `py_worker_pool.c`, `py_pool_worker_t` | an older pool, no caller | legacy pool | +| `thread_worker`, `thread_worker_call` (`py_thread_worker.c`), `py_thread_handler` | the channel a Python thread uses to call Erlang | thread callback bridge | +| `py_event_worker` | the Erlang process that drives one asyncio loop (readiness, timers) | loop driver | +| `docs/workers.md`, "worker loop" | a long-running asyncio loop on a context thread, gunicorn-style | worker loop | + +## Pool + +`py_context_router` pools (`py:call(Pool, M, F, A)`): named sets of contexts +routed by scheduler. `py_event_loop_pool`: main-interpreter asyncio loops +with process affinity. `g_thread_pool` in `py_subinterp_thread.c`: the +threads behind owngil contexts. `g_pool` in `py_worker_pool.c`: legacy. + +## Callback + +An Erlang function registered with `py:register_function/2` or +`register/2` in `py_callback` and called from Python as `erlang.call('name', ...)` +or `erlang.name(...)`. Four delivery paths exist (suspension, blocking pipe, +thread worker, socket); see [architecture](architecture.md#python-calling-erlang-erlangcall). + +**Suspension**: worker-mode delivery where the Python call raises +`SuspensionRequired`, the context thread hands control to the Erlang process, +and execution resumes with the result (`resume_callback/2`). + +**Callback pipe**: owngil-mode delivery where the context thread blocks on a +pipe until the `py_context` handler process writes the response frame. + +## Frame + +The wire unit of the callback pipe and of the isolated socket: +`<>`, body `<>`. +Status 0 request, 1 error reply, 2 ok reply, 3 request from Python, +4 event, 5 control. + +## Loop + +`ErlangEventLoop`: the asyncio loop implementation backed by `enif_select` +(`_erlang_impl/_loop.py`), used by embedded modes. "Loop ref": the NIF +handle of such a loop (`py_context:loop_ref/1`, the `submit_task` NIF). +An isolated child uses the standard asyncio loop and has no loop ref. + +## Environment, process-local env + +The Python namespace a call runs in. Every context has globals; in embedded +modes each Erlang process can additionally get its own env inside a context +(`py:call(Ctx, ...)`, [process-bound-envs](process-bound-envs.md)); isolated +contexts have one namespace, the child's `__main__`. + +## Handle + +A term that stands for something living elsewhere: `{'$py_shm', Id, Path, Size}` +(shared region), `{'$py_buffer', Id, Path, Ring}` (shared buffer), NIF +resource references (native buffers, channels, `py_ref` object references). +Only the first two cross a process boundary. + +## Child + +The OS process an isolated context runs Python in, started from +`priv/py_isolated_child.py`. It is restarted on crash within the context's +restart budget; its state does not survive a restart. + +## Interrupt, kill + +`py_context:interrupt/1` stops the request executing now (at the next +bytecode in embedded modes, immediately in the child through a signal). +`py_context:kill/1` sends `SIGKILL` to an isolated child; there is no +equivalent for embedded contexts. diff --git a/priv/_erlang_impl/README.md b/priv/_erlang_impl/README.md new file mode 100644 index 0000000..36f5066 --- /dev/null +++ b/priv/_erlang_impl/README.md @@ -0,0 +1,29 @@ +# _erlang_impl + +The Python half of the `erlang` module. `priv/` is on `sys.path` of every +interpreter erlang_python starts. In embedded modes the C module `erlang` +(`c_src/py_callback.c`) is the primary and delegates to this package for the +asyncio loop, channels, servers and helpers; in an isolated child there is +no C module and `_isolated.py` builds the whole `erlang` module from here. + +| Module | What it is | Embedded | Child | +|---|---|---|---| +| `__init__.py` | Public surface: `run`, `sleep`, `spawn_task`, `new_event_loop`, `install`, `atom`, `channel`, `byte_channel`, `server`, `reactor` | yes | partly (the child shim re-exports `server` and reimplements the rest on the stdlib loop) | +| `_loop.py` | `ErlangEventLoop`: asyncio loop whose readiness comes from `enif_select` and timers from `erlang:send_after`, through the `py_event_loop` C module | yes | no | +| `_policy.py`, `_transport.py` | Loop policy and transports for the loop above | yes | no | +| `_reactor.py` | Protocol-based reactor on fds Erlang owns | yes | no | +| `_channel.py`, `_byte_channel.py` | `Channel`, `ByteChannel` over NIF resources | yes | no | +| `_server.py` | `serve(listen_fd, factory)`, `adopt(fd, factory)`, `stop_serving`: plain asyncio on an fd handed over by Erlang | yes | yes | +| `_sandbox.py`, `_subprocess.py` | Audit hook that blocks fork/exec inside the VM | yes | no | +| `_mode.py` | Detects the execution mode | yes | yes | +| `_etf.py` | ETF codec with the `py_convert.c` type mapping; opaque `Pid`, `Ref`, `Port` keep their raw bytes | no | yes | +| `_isolated.py` | Child runtime: frame parser, reader thread, re-entrant main loop, interrupt signal, execution stack, asyncio loop, `SharedMemory` conversion, the `erlang` shim | no | yes | +| `_shm.py` | `SharedMemory` and `SharedBuffer` over `mmap`; `from_term` cache; buffer flow control through Erlang callbacks | yes | yes | + +Conventions: modules starting with `_` are internal; user code imports +`erlang`. Anything a mode cannot support raises `RuntimeError` with the +mode in the message rather than degrading silently (`_isolated.py`, +`_subprocess.py`). + +Tests for the loop live in `priv/test_erlang_loop.py` and `priv/tests/`; +the Erlang suites drive everything else. diff --git a/rebar.config b/rebar.config index 0010a51..d3ee4e1 100644 --- a/rebar.config +++ b/rebar.config @@ -76,6 +76,9 @@ <<"docs/security.md">>, <<"docs/distributed.md">>, <<"docs/testing-free-threading.md">>, + <<"docs/architecture.md">>, + <<"docs/code-map.md">>, + <<"docs/glossary.md">>, <<"docs/preload.md">>, <<"docs/owngil_internals.md">>, <<"docs/event_loop_architecture.md">> @@ -111,6 +114,9 @@ <<"docs/testing-free-threading.md">> ]}, {<<"Internals">>, [ + <<"docs/architecture.md">>, + <<"docs/code-map.md">>, + <<"docs/glossary.md">>, <<"docs/preload.md">>, <<"docs/owngil_internals.md">>, <<"docs/event_loop_architecture.md">>