A Rust library to load Python extension modules directly from memory. This project enables importing compiled Python extension modules (shared objects or DLLs) without requiring them to be present as files on disk.
- single-phase initialization modules loading fully implemented.
- multi-phase initialization modules loading WIP.
Python extension modules are usually loaded from files on disk. This project allows loading such modules directly from a binary blob in memory. This can be useful for embedding Python extensions in environments where writing files is undesirable or impossible.
On Unix-like systems, the library uses POSIX shared memory to simulate loading from a file. On Windows, it leverages the MemoryModule library to load PE modules from memory buffers via FFI.
Suppose you have a Python extension module add implemented in Cython:
add.py
def add(a, b):
return a + b
Build the extension module with Cython:
cythonize -i add.py
This produces add.pyd (Windows) or add.so (Unix).
Then, in Python, load the module from memory:
load.py
from memory_loader import PyMemoryModule
with open("add.pyd", "rb") as f:
data = f.read()
module = PyMemoryModule(data, "add")
assert module.add(2, 2) == 4
This code reads the compiled module into memory and loads it without writing it back to disk.
| Platform | Mechanism |
|---|---|
| Unix | Uses POSIX shared memory (shm) to back module loading. |
| Windows | Uses fancycode/MemoryModule via Rust FFI to load PE files from memory buffers. |
- Single-phase initialization modules are supported.
- Multi-phase initialization modules are under development.
This project uses and builds upon the excellent work of the MemoryModule library by Joachim Bauch for Windows PE loading from memory.