diff --git a/CHANGELOG.md b/CHANGELOG.md index f042595..a143464 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,7 +34,8 @@ such a region with ring backpressure, usable as `wsgi.input` in isolated contexts. Handles are plain terms and travel inside any argument or result; `py_shm:read_only/1` and `new(Size, #{writable => false})` hand Python a - read-only mapping. + read-only mapping. `py_buffer:write/3` takes a timeout for the case where + the ring is full and nobody reads (default 30 s). - `py:python_executable/0`, `py:kill/1`, `py_nif:os_kill/2`. - `py_isolated` is a `gen_statem` (states `idle`, `{busy, Id}`, `looping`, `stopping_loop`, `{restarting, Reason}`): `sys:get_state/1` and @@ -48,6 +49,11 @@ storms, loop churn, 60 s mixed workload with resource counters checked. - Guide: `docs/isolated.md`, with what each of the three modes guarantees. +### Fixed + +- `pthread_timedjoin_np` was called without `_GNU_SOURCE`, an implicit + declaration on Linux that newer compilers reject. + ## 4.1.0 (2026-08-15) ### Added diff --git a/README.md b/README.md index aa550d5..6045ac8 100644 --- a/README.md +++ b/README.md @@ -654,6 +654,7 @@ py:execution_mode(). %% => worker | owngil ## Documentation +- [Architecture](docs/architecture.md), [Code map](docs/code-map.md), [Glossary](docs/glossary.md) - how the pieces fit - [Getting Started](docs/getting-started.md) - [Process-Bound Environments](docs/process-bound-envs.md) - Isolated Python state per Erlang process - [AI Integration Guide](docs/ai-integration.md) diff --git a/c_src/README.md b/c_src/README.md new file mode 100644 index 0000000..a8f240e --- /dev/null +++ b/c_src/README.md @@ -0,0 +1,71 @@ +# c_src + +The NIF that embeds CPython in the VM. One translation unit: `py_nif.c` +`#include`s every other `.c` file (see the "Include module implementations" +section near line 265), so build with `rebar3 compile` (CMake through +`do_cmake.sh` / `do_build.sh`), never a single file. Headers declare what +the included files share. + +Read `docs/architecture.md` first for how a call travels; this file says +where things are. + +## Files + +| File | What it owns | Notes | +|---|---|---| +| `py_nif.h` | All shared types: `py_context_t` and its request queue, request types, callback and suspension state, runtime state machine, atoms, globals, declarations | 2.4k lines. The struct comments carry the locking rules; read `py_context_t` before touching threads | +| `py_nif.c` | Runtime init and finalize, resource types, context create/destroy, the request queue, `worker_context_thread_main` and `owngil_context_thread_main`, `owngil_execute_*` (used by both thread kinds), the `nif_context_*` NIFs, process-local envs, `py_ref`, the NIF function table at the end | Sections are banner-separated; `grep -n '^ \* ===\|^/\* ==='` lists them | +| `py_convert.c` | `py_to_term`, `term_to_py`, depth limits, tagged tuples (`{bytes, B}`, `{'$py_shm', ...}`), error tuples `{error, {Type, Msg}}` | The type mapping tables in the comments are the reference for `_etf.py` | +| `py_exec.c` | Executing a call/eval/exec with suspension support; the legacy single executor thread | | +| `py_callback.c` | The `erlang` Python module: `call`, `send`, `whereis`, `schedule*`, `Atom`/`Pid`/`Ref` types, callback delivery paths (suspension, blocking pipe, async pipe), channel and shared-dict methods, callback name registry | `erlang_call_impl` documents the path precedence | +| `py_thread_worker.c` | Python threads calling Erlang through `py_thread_handler` | | +| `py_subinterp_thread.c/.h` | Thread pool of sub-interpreters (owngil contexts, loop pools) | | +| `py_event_loop.c/.h` | `ErlangEventLoop` support: `enif_select` readers and writers, timers, task injection into loops, reactor dispatch, fd registry, Python module `py_event_loop`; plus test-only fd/TCP/UDP NIFs (section "Test Helper Functions") | Largest file | +| `py_channel.c/.h`, `py_buffer.c/.h`, `py_reactor_buffer.c/.h`, `py_shared_dict.c` | Resources with a Python-facing object each | | +| `py_logging.c` | Logging and tracing NIFs | | +| `py_mem_limit.c` | obmalloc arena accounting for owngil memory caps | | +| `py_worker_pool.c/.h` | Legacy pool, no caller in `src/` | Candidate for removal | +| `py_util.c/.h` | Macros, small helpers | | + +## Where the live paths are + +- `py:call/3` in worker or owngil mode: `nif_context_call_async` (`py_nif.c`) + enqueues; `worker_context_thread_main` or `owngil_context_thread_main` + dequeues and calls `owngil_execute_request`; the reply goes out as + `{py_result, Ref, Result}`. +- `erlang.call` from Python: `erlang_call_impl` (`py_callback.c`). +- Interrupt: `nif_context_interrupt` (`py_nif.c`), `interrupt_mutex` rules on + `py_context_t`. +- Type conversion: `py_to_term` / `term_to_py` (`py_convert.c`). +- Isolated mode has no C code of its own: `os_kill` is the only NIF it uses. + +## Rules that are easy to break + +- Only the context's thread touches the context's Python objects. NIFs + called from Erlang processes enqueue requests and return; they do not run + Python for a context that has a thread. +- `Py_BEGIN_ALLOW_THREADS` around every blocking wait; never block on an + Erlang-side resource with the GIL held. +- Never call a Future method while holding `async_futures_mutex` + (`py_callback.c`, comment above the struct explains the pattern). +- `interrupt_mutex` is taken only by threads that do not hold the GIL. +- `queue_mutex` protects the request queue of a context and is taken before + a request's own mutex (`ctx_queue_cancel_all`), never the other way. +- A NIF that can block or run Python is registered with a dirty scheduler + flag in the table at the end of `py_nif.c`. + +## Adding a NIF + +1. Implement `static ERL_NIF_TERM nif_x(ErlNifEnv*, int, const ERL_NIF_TERM[])` + next to related code. +2. Add `{"x", Arity, nif_x, Flags}` to `nif_funcs[]` at the end of `py_nif.c`. +3. Add the stub and its `-spec` and doc to `src/py_nif.erl`. +4. Cover it in a suite; `rebar3 dialyzer` and `rebar3 xref` must stay clean. + +## Legacy code, for orientation + +Not on the path of contexts created today: the `worker_*` NIFs and +"Worker management" section, `async_worker_*` NIFs (return `deprecated`), +the "Legacy mode" inline branches in `nif_context_call/eval/exec`, +`py_worker_pool.c`, and the `cancel_reader/writer` aliases. When in doubt, +follow `nif_context_call_async` and ignore the rest. diff --git a/c_src/py_nif.c b/c_src/py_nif.c index cb8f459..83f83df 100644 --- a/c_src/py_nif.c +++ b/c_src/py_nif.c @@ -29,11 +29,10 @@ * - Resource types for Python objects to ensure proper cleanup * - Dirty NIF flags for GIL-holding operations * - * This file is the main entry point. It includes the following modules: - * - py_nif.h: Shared header with types and declarations - * - py_convert.c: Type conversion (Python <-> Erlang) - * - py_exec.c: Python execution and GIL management - * - py_callback.c: Callback system and asyncio support + * This file is the main entry point and the single translation unit: it + * includes the other .c files (see "Include module implementations"). The + * file map is c_src/README.md; the request lifecycle per context mode is + * docs/architecture.md. */ /* pthread_timedjoin_np (used to bound the owngil worker join on Linux) diff --git a/c_src/py_nif.h b/c_src/py_nif.h index 4f2e37c..45020cb 100644 --- a/c_src/py_nif.h +++ b/c_src/py_nif.h @@ -21,54 +21,19 @@ * * @mainpage Python-Erlang NIF Integration * - * @section intro_sec Introduction - * - * This NIF (Native Implemented Function) library provides seamless integration - * between Erlang/OTP and Python. It embeds a Python interpreter within the - * Erlang VM and provides bidirectional communication capabilities. - * - * @section arch_sec Architecture - * - * The implementation follows a modular design with four main components: - * - * - **py_nif.h** - Shared types, macros, and declarations - * - **py_convert.c** - Bidirectional type conversion (Python ↔ Erlang) - * - **py_exec.c** - Python execution engine and GIL management - * - **py_callback.c** - Erlang callback support and asyncio integration - * - * @section modes_sec Execution Modes - * - * The library supports three execution modes based on Python version: - * - * | Mode | Python Version | Description | - * |------|----------------|-------------| - * | FREE_THREADED | 3.13+ (no-GIL) | Direct execution without GIL | - * | SUBINTERP | 3.12+ | Per-interpreter GIL isolation | - * | MULTI_EXECUTOR | Any | Multiple executor threads with GIL | - * - * @section gil_sec GIL Management - * - * The GIL (Global Interpreter Lock) is managed following PyO3/Granian patterns: - * - * - `Py_BEGIN_ALLOW_THREADS` / `Py_END_ALLOW_THREADS` around blocking ops - * - Executor threads hold the GIL and process queued requests - * - Dirty I/O schedulers are used for Python-calling NIFs - * - * @section callback_sec Callback Mechanism - * - * Python code can call back to Erlang using a suspension/resume pattern: - * - * 1. Python calls `erlang.call('func', args)` - * 2. NIF raises `SuspensionRequired` exception - * 3. Dirty scheduler is released, callback sent to Erlang - * 4. Erlang processes callback, calls `resume_callback/2` - * 5. Python execution resumes with cached result - * - * @section mem_sec Memory Management - * - * - Erlang resources wrap Python objects (prevent GC) - * - Thread-local storage for callback context - * - Proper cleanup in resource destructors + * This NIF embeds CPython in the Erlang VM. The map of the code, the life + * of a call in each context mode, the callback paths and the locking rules + * are documented in docs/architecture.md, c_src/README.md and + * docs/glossary.md; keep those current instead of this comment. + * + * In one paragraph: py_nif.c is the single translation unit and includes + * the other .c files; a context (py_context_t) owns a request queue and a + * pthread that runs Python for it (worker mode: main interpreter, shared + * GIL; owngil mode: a sub-interpreter with its own GIL); Erlang enqueues + * through nif_context_call_async and receives {py_result, Ref, Result}; + * Python calls Erlang through erlang_call_impl (py_callback.c), by + * suspension in worker mode and a blocking pipe in owngil mode. Isolated + * mode runs Python in a child process and uses no C code beyond os_kill. */ #ifndef PY_NIF_H diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..53e6426 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,198 @@ +# Architecture + +This page is the map of erlang_python. It +says which processes and threads exist, how a call travels from `py:call/3` +to Python and back in each context mode, how Python calls Erlang, and which +code paths are live. Read it before opening `c_src/` or `src/py_context.erl`. +The [code map](code-map.md) lists every file; the [glossary](glossary.md) +defines the overloaded words (worker, context, pool). + +## One picture + +``` +Erlang VM child OS process ++--------------------------------------------------+ (isolated mode only) +| erlang_python_sup | +| py_callback registry of Erlang funs Python | +| may call | +| py_shm shared memory regions | +| py_thread_handler spawns a handler process | +| per Python thread that calls | +| Erlang | +| py_logger / py_tracer Python logging, tracing | +| py_context_sup ---- py_context (one per | +| context; mode worker | owngil | isolated) | +| py_context_init starts the default pool | +| py_event_worker_sup / _registry loop drivers | +| py_event_loop, py_event_loop_pool main- | +| interpreter asyncio loops | ++--------------------------------------------------+ + | NIF calls (dirty schedulers) | Unix socket, ETF frames + v v ++------------------------+ +-------------------------+ +| libpython in the VM | | python3 py_isolated_ | +| main interpreter | | child.py | +| worker contexts: one | | reader thread + main | +| pthread each, shared | | thread, own asyncio | +| GIL | | loop | +| owngil contexts: one | +-------------------------+ +| pthread + own sub- | +| interpreter each | ++------------------------+ +``` + +A **context** is the unit of work: an Erlang process (`py_context`) that owns +one Python execution environment and serves calls in order. Pools +(`py_context_router`) route `py:call/3` to a context by scheduler affinity. + +## The three context modes + +| | `worker` | `owngil` | `isolated` | +|---|---|---|---| +| Python runs in | the VM, main interpreter | the VM, a sub-interpreter with its own GIL | a child process | +| Thread | one pthread per context (`worker_context_thread_main`) | one pthread per context (`owngil_context_thread_main`) | the child's main thread | +| Erlang process loop | the receive loop in `py_context` | the receive loop in `py_context` | `py_isolated` (`gen_statem`) | +| Transport | NIF request queue on `py_context_t` | same | Unix socket, frames of the callback pipe format | +| Python -> Erlang | suspension protocol | blocking callback pipe | socket frames | +| Interrupt | `PyThreadState_SetAsyncExc`, next bytecode | same | signal in the child, `SIGKILL` backstop | +| Guide | [context-affinity](context-affinity.md), [pools](pools.md) | [owngil_internals](owngil_internals.md) | [isolated](isolated.md) | + +## Life of a call, per mode + +### worker and owngil (embedded) + +1. `py:call(M, F, A)` picks a context through `py_context_router` and sends + `{call, From, MRef, M, F, Args, Kwargs}` to the `py_context` process + (`src/py.erl`, `src/py_context.erl:call/6`). The caller waits in + `await_reply/3`; a timeout there calls `interrupt/1`. +2. The `py_context` receive loop takes it and calls `handle_call_with_suspension/5`, + which calls the `context_call_async` NIF + (`c_src/py_nif.c`, `nif_context_call_async`). The NIF converts the + arguments (`term_to_py`, `c_src/py_convert.c`) into a request, enqueues it + on the context's queue (`ctx_queue_enqueue`) and returns `{enqueued, Ref}` + at once. The Erlang process is now free to serve callbacks. +3. The context's pthread (`worker_context_thread_main` or + `owngil_context_thread_main`, `c_src/py_nif.c`) dequeues the request and + runs it through `owngil_execute_request` (despite its name it serves both + modes), which calls into Python with the GIL held. +4. The thread converts the result (`py_to_term`) and sends + `{py_result, Ref, Result}` to the `py_context` process, which replies + `From ! {MRef, Result}`. + +The older paths in `nif_context_call` (a blocking variant with an inline +"legacy" executor) are kept for the fallback +`{error, async_requires_worker_thread}` and are not taken by contexts +created today; see [code map](code-map.md) for the list of legacy code. + +### isolated + +1. Same first step: the message reaches the `py_context` pid, which in this + mode runs `py_isolated` (a `gen_statem` entered with `enter_loop`). +2. In `idle` the request is encoded with `term_to_binary` into a frame + `<>` and written to + the Unix socket; the state becomes `{busy, Id}` and other callers' + requests are postponed (served in order when the child is free). +3. In the child, the reader thread parses frames and queues the request; + the main thread runs it (`_erlang_impl/_isolated.py`, `Runtime._dispatch`) + and writes the reply frame. +4. Back in `py_isolated`, the reply for the busy id moves the state to + `idle` and answers the caller. A crash of the child is seen as the port's + `exit_status`; the state goes through `{restarting, Reason}` and a new + child is started within the restart budget. + +Protocol details: the module header of `src/py_isolated.erl` and the +docstring of `priv/_erlang_impl/_isolated.py`. + +## Python calling Erlang (`erlang.call`) + +`erlang_call_impl` in `c_src/py_callback.c` chooses one of these paths, in +this order (the comment above it is the authoritative version): + +1. **Suspension** (worker contexts). The Python call raises + `SuspensionRequired`; the context thread returns + `{suspended, CallbackId, State, {Name, Args}}` to `py_context`, which runs + the registered fun (`execute/2` in `py_callback`), possibly serving nested calls + meanwhile (`wait_for_callback/2`), and resumes with + the `resume_callback` NIF. +2. **Blocking callback pipe** (owngil contexts). The context thread writes + a request on a pipe and blocks; the `py_context` process has a dedicated + handler (`callback_handler_loop/1`) that runs the fun and writes the + response frame back with `context_write_callback_response`. +3. **Legacy worker handler** (`worker_*` NIFs): only used by + `examples/gen_test.erl`. +4. **Thread worker** (`c_src/py_thread_worker.c`): any Python thread that is + not a context thread (`threading.Thread`, executors) asks the + `py_thread_handler` coordinator for a handler process and talks to it + over a pipe. There is also an async variant (`erlang.async_call`) using a + per-interpreter async pipe. + +In isolated mode there is one path: a status-3 frame on the socket, answered +by a process the `py_isolated` state machine spawns; nested calls into the +same context are dispatched immediately because they come from that process. + +The frame format shared by the pipe and the socket, and the ETF conventions, +will get their own page (protocols); until then `c_src/py_convert.c` (type +mapping) and `priv/_erlang_impl/_etf.py` are the reference. + +## asyncio + +Three different machineries, on purpose: + +- Embedded contexts share an `ErlangEventLoop` (`priv/_erlang_impl/_loop.py`) + whose `add_reader`/`add_writer` map onto `enif_select` and `call_later` onto + `erlang:send_after`; readiness is delivered to a `py_event_worker` process + per loop (`c_src/py_event_loop.c`, `src/py_event_worker.erl`). Worker loops + (`py_context:start_loop/1`) run such a loop on the context thread. +- `py_event_loop`/`py_event_loop_pool` expose the main-interpreter loops for + `py:async_call/3` and friends. +- An isolated child runs a plain asyncio loop; the only integration is + delivering results over the socket. Nothing of the reactor is ported. + +See [event_loop_architecture](event_loop_architecture.md) for the embedded +loop and [asyncio](asyncio.md) for the API. + +## Data paths that avoid copies + +- `py_buffer` (native): a NIF resource Erlang writes into and Python reads + through the buffer protocol; embedded modes only. +- `py_shm` and `py_buffer:new(#{shared => true})`: a file mapped + `MAP_SHARED` by the VM (through iommap) and by any interpreter, embedded or + child, with flow control through callbacks (`_py_buffer_wait`, + `_py_buffer_consumed`). See [isolated](isolated.md#bulk-data-with-shared-memory). +- `py_channel`, `py_byte_channel`: message queues between Erlang and Python + coroutines, embedded modes only. + +## Where state lives + +| State | Owner | Notes | +|---|---|---| +| Registered callbacks | `py_callback` ETS `py_callbacks` | also mirrored in a C name registry so `erlang.` resolves; `py_state` exposes its store the same way (`erlang.state_get`) | +| Import and path registry | `py_import` ETS | applied to every new interpreter and to isolated children at start | +| Preload code | `py_preload` persistent_term | run once per interpreter | +| Context pid -> NIF ref | `py_context` ETS `py_context_refs` | lets `interrupt/1` reach a context blocked in a NIF; isolated contexts store the atom `isolated` | +| Per-Erlang-process Python env | `py` process dictionary + NIF env resource | `py:call(Ctx, ...)`; not applicable to isolated contexts | +| Shared regions | `py_shm` ETS `py_shm_regions` | closed on owner death | +| Python-side state | the interpreter | lost when an isolated child restarts | + +## Invariants worth knowing before editing + +- A context serves one request at a time. Embedded: the thread dequeues one + request; nested callbacks are served by the `py_context` process while the + thread waits. Isolated: enforced by the `{busy, Id}` state and `postpone`. +- Only the context's own thread touches its Python objects; NIF callers only + enqueue. `py_context_t` fields are documented in `c_src/py_nif.h`. +- Interrupts target the request executing now. Isolated mode also cancels a + queued request by id; embedded modes cannot. +- Everything that crosses the socket is a term; NIF resources (channels, + native buffers, object references) do not cross, and the API says so with + `{error, not_supported_in_isolated}`. +- `binary_to_term` on child data is not `safe`: the child can create atoms. + +## What is live and what is not + +Kept for now, not used by current contexts: the `worker_*` NIF API and its +single executor thread (`c_src/py_exec.c`), the `async_worker_*` NIFs (they +return `deprecated`), `c_src/py_worker_pool.c` (no caller), the inline +"legacy" executor branches in `nif_context_*`, and the test-only fd NIFs in +`c_src/py_event_loop.c`. They are listed in the [code map](code-map.md) so +nobody debugs them by mistake; removing them is planned. diff --git a/docs/code-map.md b/docs/code-map.md new file mode 100644 index 0000000..3139871 --- /dev/null +++ b/docs/code-map.md @@ -0,0 +1,98 @@ +# Code map + +Every source file, what it owns, and where to look for its behaviour. Status +is `live` (on the path of a context created today), `legacy` (kept for +compatibility, no current caller in `src/`), or `test` (only exercised by +suites). Guides are in `docs/`, suites in `test/`. Start with +[architecture](architecture.md). + +## Erlang (`src/`) + +| Module | Owns | Status | Guide | Suites | +|---|---|---|---|---| +| `py` | Public API facade: call/eval/exec, streams, async helpers, venvs, memory, function registration | live | README, getting-started | `py_SUITE`, `py_api_SUITE`, `py_stream_SUITE`, `py_venv_SUITE` | +| `py_context` | The context process for embedded modes and the API every mode answers (`call/eval/exec`, `interrupt`, `kill`, loops, `pass_fd`); dispatch to `py_isolated` for isolated mode | live | context-affinity, workers, interrupts | `py_context_SUITE`, `py_context_process_SUITE`, `py_interrupt_SUITE`, `py_worker_loop_SUITE` | +| `py_isolated` | `gen_statem` driving a child process over the socket; restart policy | live | isolated | `py_isolated_*_SUITE` | +| `py_context_router` | Pools and scheduler-affinity routing | live | pools, context-affinity | `py_context_router_SUITE`, `py_pool_SUITE` | +| `py_context_sup`, `py_context_init` | Supervisor of contexts; starts the default pool at boot | live | pools | (through the above) | +| `py_nif` | Erlang stubs and docs for every NIF | live | api-reference | all | +| `py_callback` | Registry of Erlang funs callable as `erlang.call('name', ...)` | live | README (callbacks) | `py_callback_encoding_SUITE`, `py_thread_callback_SUITE` | +| `py_thread_handler` | Coordinator that gives each Python thread calling Erlang a handler process and a pipe | live | threading | `py_thread_callback_SUITE`, `py_reentrant_SUITE` | +| `py_event_loop` | Main-interpreter asyncio loop: `run`, `create_task`, `await`, and the loop callbacks Python needs | live | asyncio | `py_event_loop_SUITE`, `py_async_task_SUITE` | +| `py_event_loop_pool` | Several main-interpreter loops with process affinity | live | asyncio | `py_event_loop_pool_SUITE` | +| `py_event_worker`, `_sup`, `_registry` | One process per running loop receiving `enif_select` readiness and timers | live | event_loop_architecture | `py_event_loop_SUITE`, `py_fd_ops_SUITE` | +| `py_reactor_context` | FD-owning context for the protocol-based reactor | live | reactor | `py_reactor_SUITE` | +| `py_channel`, `py_byte_channel` | Term and byte queues between Erlang and Python coroutines (NIF resources) | live | channel | `py_channel_SUITE`, `py_byte_channel_SUITE` | +| `py_buffer` | Native streaming input buffer; shared variant delegates to `py_shm` | live | buffer, isolated | `py_buffer_SUITE`, `py_isolated_buffer_SUITE` | +| `py_shm` | Shared memory regions over iommap and the ring behind shared buffers | live | isolated | `py_isolated_shm_SUITE` | +| `py_import` | Registry of imports and `sys.path` entries applied to every interpreter | live | imports | `py_import_SUITE` | +| `py_preload` | Code run once per interpreter at start | live | preload | `py_preload_SUITE` | +| `py_state` | Shared key/value store visible from Python as `erlang.state_get/set/delete/keys` | live | README (shared state) | `py_state_SUITE` | +| `py_semaphore` | ETS counting semaphore for rate limiting | live | scalability | (through `py_SUITE`) | +| `py_logger`, `py_tracer` | Python `logging` into Erlang logger; tracing hooks | live | logging | `py_logging_SUITE` | +| `erlang_python_app`, `erlang_python_sup` | Application start and the supervision tree | live | architecture | all | +| `py_util` | Small helpers | live | | | + +## C (`c_src/`) + +`py_nif.c` is the only translation unit: it `#include`s the other `.c` +files. Editing `py_convert.c` alone does not compile it alone; build with +`rebar3 compile`. See `c_src/README.md`. + +| File | Owns | Status | +|---|---|---| +| `py_nif.h` | Every shared type: `py_context_t`, request types, runtime state machine, atoms, globals | live | +| `py_nif.c` | Runtime init, context creation and destruction, the request queue and the two context thread mains, the process-per-context NIFs (`nif_context_*`), process-local envs, `py_ref`, the NIF table | live, with legacy branches | +| `py_convert.c` | `py_to_term` / `term_to_py`, the type mapping, tagged tuples (`{bytes, B}`, shared handles) | live | +| `py_exec.c` | Execution with suspension support; the legacy single executor thread | live (suspension), legacy (executor) | +| `py_callback.c` | The `erlang` Python module: `call`, `send`, `whereis`, `Atom`/`Pid`/`Ref` types, schedule markers, callback pipes, channel and shared dict methods | live | +| `py_thread_worker.c` | Python threads calling Erlang through `py_thread_handler` | live | +| `py_subinterp_thread.c` | Sub-interpreter thread pool used by owngil contexts and loop pools | live | +| `py_event_loop.c` | `ErlangEventLoop` support: `enif_select` readers/writers, timers, task injection, reactor dispatch, fd registry; also ~570 lines of test-only fd/TCP/UDP NIFs | live; test section | +| `py_channel.c`, `py_buffer.c`, `py_reactor_buffer.c`, `py_shared_dict.c` | The corresponding resources and their Python-facing methods | live | +| `py_logging.c` | Logging and tracing NIFs | live | +| `py_mem_limit.c` | Per-interpreter memory caps (owngil) | live | +| `py_worker_pool.c/.h` | An older worker pool | legacy, no caller | +| `py_util.c/.h` | Macros and helpers | live | + +Inside `py_nif.c`, these are legacy: the `worker_*` NIFs and the "Worker +management" section, the `async_worker_*` NIFs (return `deprecated`), the +inline executor branches marked "Legacy mode" in `nif_context_call`, +`nif_context_eval`, `nif_context_exec`, and the `cancel_reader/writer` +aliases. + +## Python (`priv/`) + +`priv/` is on `sys.path` of every interpreter. `_erlang_impl` is the Python +half of the `erlang` module; the embedded C module delegates to it for the +loop, channels and servers. + +| File | Owns | Used by | +|---|---|---| +| `_erlang_impl/__init__.py` | Public surface of `erlang` in embedded modes: `run`, `sleep`, `spawn_task`, loop policy, `atom`, channels, `server` | embedded | +| `_erlang_impl/_loop.py`, `_policy.py`, `_transport.py` | `ErlangEventLoop` (uvloop-compatible) over `enif_select` | embedded | +| `_erlang_impl/_reactor.py` | Protocol-based reactor over fds Erlang owns | embedded | +| `_erlang_impl/_channel.py`, `_byte_channel.py` | Python side of channels | embedded | +| `_erlang_impl/_server.py` | `serve`, `adopt`, `stop_serving` on fds handed over by Erlang; plain asyncio, works in every mode | all | +| `_erlang_impl/_sandbox.py`, `_subprocess.py` | Audit hook blocking fork/exec inside the VM | embedded | +| `_erlang_impl/_mode.py` | Detects how Python is running (embedded, free-threaded, child) | all | +| `_erlang_impl/_etf.py` | Pure-Python ETF codec with the `py_convert.c` mapping | isolated child | +| `_erlang_impl/_isolated.py` | Child runtime: socket frames, reader thread, re-entrant main loop, interrupt signal, asyncio loop, the `erlang` shim | isolated child | +| `_erlang_impl/_shm.py` | `SharedMemory` and `SharedBuffer` wrappers over mmap | all | +| `py_isolated_child.py` | Child launcher: rlimits, parent-death signal, cgroup join, connect | isolated child | +| `test_erlang_loop.py`, `tests/` | Python-side tests of the loop | test | + +## Tests (`test/`) + +Suites named `py__SUITE`. Cross-mode suites run the same cases in +`worker` and `isolated` groups (`py_isolated_SUITE`, `py_isolated_vm_SUITE`, +`py_isolated_shm_SUITE`, `py_isolated_buffer_SUITE`). Python helpers used by +suites are `test/py_test_*.py`. `test/coverage_audit.md` maps public APIs to +cases. `test/test.config` holds node-wide settings (memory limits flag). + +## Build and docs + +`rebar.config` runs `do_cmake.sh` / `do_build.sh` (CMake in `c_src/`) as +compile hooks; the NIF lands in `priv/py_nif.so`. `make lint-docs` checks +that Erlang snippets in the guides call real exports and that Python +snippets parse. `rebar3 ex_doc` builds the guides listed in `rebar.config`. diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 0000000..cf5864c --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,109 @@ +# Glossary + +The same words mean different things in different files of this project. +This page fixes one meaning per term and says where the other uses come +from, so a reader can translate as they go. + +## Context + +**Context**: one Python execution environment served by one Erlang process +(`py_context`), in order, one request at a time. The unit of `py:context/0`, +pools and modes. In C it is `py_context_t` (`c_src/py_nif.h`); in +`py_context.erl` it is the process; for isolated mode the process runs +`py_isolated` and the environment is a child OS process. + +Other uses: `py_reactor_context` is a context that also owns file +descriptors for the reactor; "coordinator context" in C comments means the +`py_thread_handler` side of the thread-worker channel. + +## Mode + +**Mode**: how a context runs Python. `worker` (main interpreter, one pthread +per context, shared GIL), `owngil` (a sub-interpreter with its own GIL per +context, one pthread), `isolated` (a child process). `py_context:new(#{mode => ...})`. + +Related flags on `py_context_t`: `uses_worker_thread` (has its own pthread; +true for worker and owngil contexts created today), `is_subinterp` (has its +own sub-interpreter), `uses_own_gil` (that sub-interpreter has its own GIL). +`subinterp` in file and NIF names (`py_subinterp_thread.c`, +`subinterp_supported/0`) refers to the machinery owngil mode is built on; +there is no separate "subinterp mode" any more. + +The runtime-wide `PY_MODE_FREE_THREADED` / `PY_MODE_GIL` in `py_nif.h` is +about the Python build (free-threaded or not), not about contexts. + +## Worker + +The most overloaded word. Meanings, by file: + +| Where | Meaning | Prefer to say | +|---|---|---| +| `py_context:new(#{mode => worker})` | the context mode above | worker mode | +| `worker_context_thread_main`, `uses_worker_thread` (`py_nif.c`) | the pthread that serves a context's queue | context thread | +| `worker_new/call/eval/exec` NIFs, `py_worker_t` | the legacy per-worker API, before contexts | legacy worker API | +| `py_worker_pool.c`, `py_pool_worker_t` | an older pool, no caller | legacy pool | +| `thread_worker`, `thread_worker_call` (`py_thread_worker.c`), `py_thread_handler` | the channel a Python thread uses to call Erlang | thread callback bridge | +| `py_event_worker` | the Erlang process that drives one asyncio loop (readiness, timers) | loop driver | +| `docs/workers.md`, "worker loop" | a long-running asyncio loop on a context thread, gunicorn-style | worker loop | + +## Pool + +`py_context_router` pools (`py:call(Pool, M, F, A)`): named sets of contexts +routed by scheduler. `py_event_loop_pool`: main-interpreter asyncio loops +with process affinity. `g_thread_pool` in `py_subinterp_thread.c`: the +threads behind owngil contexts. `g_pool` in `py_worker_pool.c`: legacy. + +## Callback + +An Erlang function registered with `py:register_function/2` or +`register/2` in `py_callback` and called from Python as `erlang.call('name', ...)` +or `erlang.name(...)`. Four delivery paths exist (suspension, blocking pipe, +thread worker, socket); see [architecture](architecture.md#python-calling-erlang-erlangcall). + +**Suspension**: worker-mode delivery where the Python call raises +`SuspensionRequired`, the context thread hands control to the Erlang process, +and execution resumes with the result (`resume_callback/2`). + +**Callback pipe**: owngil-mode delivery where the context thread blocks on a +pipe until the `py_context` handler process writes the response frame. + +## Frame + +The wire unit of the callback pipe and of the isolated socket: +`<>`, body `<>`. +Status 0 request, 1 error reply, 2 ok reply, 3 request from Python, +4 event, 5 control. + +## Loop + +`ErlangEventLoop`: the asyncio loop implementation backed by `enif_select` +(`_erlang_impl/_loop.py`), used by embedded modes. "Loop ref": the NIF +handle of such a loop (`py_context:loop_ref/1`, the `submit_task` NIF). +An isolated child uses the standard asyncio loop and has no loop ref. + +## Environment, process-local env + +The Python namespace a call runs in. Every context has globals; in embedded +modes each Erlang process can additionally get its own env inside a context +(`py:call(Ctx, ...)`, [process-bound-envs](process-bound-envs.md)); isolated +contexts have one namespace, the child's `__main__`. + +## Handle + +A term that stands for something living elsewhere: `{'$py_shm', Id, Path, Size}` +(shared region), `{'$py_buffer', Id, Path, Ring}` (shared buffer), NIF +resource references (native buffers, channels, `py_ref` object references). +Only the first two cross a process boundary. + +## Child + +The OS process an isolated context runs Python in, started from +`priv/py_isolated_child.py`. It is restarted on crash within the context's +restart budget; its state does not survive a restart. + +## Interrupt, kill + +`py_context:interrupt/1` stops the request executing now (at the next +bytecode in embedded modes, immediately in the child through a signal). +`py_context:kill/1` sends `SIGKILL` to an isolated child; there is no +equivalent for embedded contexts. diff --git a/priv/_erlang_impl/README.md b/priv/_erlang_impl/README.md new file mode 100644 index 0000000..36f5066 --- /dev/null +++ b/priv/_erlang_impl/README.md @@ -0,0 +1,29 @@ +# _erlang_impl + +The Python half of the `erlang` module. `priv/` is on `sys.path` of every +interpreter erlang_python starts. In embedded modes the C module `erlang` +(`c_src/py_callback.c`) is the primary and delegates to this package for the +asyncio loop, channels, servers and helpers; in an isolated child there is +no C module and `_isolated.py` builds the whole `erlang` module from here. + +| Module | What it is | Embedded | Child | +|---|---|---|---| +| `__init__.py` | Public surface: `run`, `sleep`, `spawn_task`, `new_event_loop`, `install`, `atom`, `channel`, `byte_channel`, `server`, `reactor` | yes | partly (the child shim re-exports `server` and reimplements the rest on the stdlib loop) | +| `_loop.py` | `ErlangEventLoop`: asyncio loop whose readiness comes from `enif_select` and timers from `erlang:send_after`, through the `py_event_loop` C module | yes | no | +| `_policy.py`, `_transport.py` | Loop policy and transports for the loop above | yes | no | +| `_reactor.py` | Protocol-based reactor on fds Erlang owns | yes | no | +| `_channel.py`, `_byte_channel.py` | `Channel`, `ByteChannel` over NIF resources | yes | no | +| `_server.py` | `serve(listen_fd, factory)`, `adopt(fd, factory)`, `stop_serving`: plain asyncio on an fd handed over by Erlang | yes | yes | +| `_sandbox.py`, `_subprocess.py` | Audit hook that blocks fork/exec inside the VM | yes | no | +| `_mode.py` | Detects the execution mode | yes | yes | +| `_etf.py` | ETF codec with the `py_convert.c` type mapping; opaque `Pid`, `Ref`, `Port` keep their raw bytes | no | yes | +| `_isolated.py` | Child runtime: frame parser, reader thread, re-entrant main loop, interrupt signal, execution stack, asyncio loop, `SharedMemory` conversion, the `erlang` shim | no | yes | +| `_shm.py` | `SharedMemory` and `SharedBuffer` over `mmap`; `from_term` cache; buffer flow control through Erlang callbacks | yes | yes | + +Conventions: modules starting with `_` are internal; user code imports +`erlang`. Anything a mode cannot support raises `RuntimeError` with the +mode in the message rather than degrading silently (`_isolated.py`, +`_subprocess.py`). + +Tests for the loop live in `priv/test_erlang_loop.py` and `priv/tests/`; +the Erlang suites drive everything else. diff --git a/rebar.config b/rebar.config index 0010a51..d3ee4e1 100644 --- a/rebar.config +++ b/rebar.config @@ -76,6 +76,9 @@ <<"docs/security.md">>, <<"docs/distributed.md">>, <<"docs/testing-free-threading.md">>, + <<"docs/architecture.md">>, + <<"docs/code-map.md">>, + <<"docs/glossary.md">>, <<"docs/preload.md">>, <<"docs/owngil_internals.md">>, <<"docs/event_loop_architecture.md">> @@ -111,6 +114,9 @@ <<"docs/testing-free-threading.md">> ]}, {<<"Internals">>, [ + <<"docs/architecture.md">>, + <<"docs/code-map.md">>, + <<"docs/glossary.md">>, <<"docs/preload.md">>, <<"docs/owngil_internals.md">>, <<"docs/event_loop_architecture.md">>