PEP 803 – “abi3t”: Stable ABI for Free-Threaded Builds

[encukou]

I've updated PEP 803 (“abi3t”: Stable ABI for Free-Threaded Builds) based on discussion that started in the previous issue.

Proposed API changes:

• Users set Py_TARGET_ABI3T to build against abi3t, which is like abi3 but for free-threaded builds.

  Implementation-wise, setting Py_TARGET_ABI3T will set:

  • Py_LIMITED_API (to the same version, or minimum if the user defined both) and
  • _Py_OPAQUE_PYOBJECT, which hides some API.

  If both Py_LIMITED_API and Py_GIL_DISABLED are set, CPython headers set Py_TARGET_ABI3T automatically.

• Py_mod_abi becomes mandatory if the new PyModExport_* hook is used

PyMutex/PyCriticalSection is left for a separate discussion. (Without locks, abi3t modules can work correctly but need Py_MOD_GIL_USED in practice.)

Should we recommend this to the SC?

• @encukou
• @erlend-aasland
• @serhiy-storchaka
• @vstinner
• @zooba

[encukou]

ping @erlend-aasland @serhiy-storchaka @vstinner @zooba

[erlend-aasland]

AFAICS, this issue is only for accepting Py_TARGET_ABI3T in the PEP?

[encukou]

Yes, only the API changes are relevant to this WG. That's Py_TARGET_ABI3T, and making Py_mod_abi mandatory.

(The latter wasn't discussed much, I hope it's because it's a detail rather than flying under the radar. I don't want to write ABI detection hacks again.)

[encukou]

ping @erlend-aasland @serhiy-storchaka @vstinner @zooba

[vstinner]

I support _Py_OPAQUE_PYOBJECT effort, PEP 697 types and PEP 793 modules. We have to migrate to these new APIs to have stable ABI working on regular Python and Free-Threaded Python. And we need to minimize the number of wheel files uploaded to PyPI.

---

Yes, only the API changes are relevant to this WG. That's Py_TARGET_ABI3T, and making Py_mod_abi mandatory.

Well, migrating an extension to Py_TARGET_ABI3T which implies _Py_OPAQUE_PYOBJECT requires to:

• Convert all types to PEP 697 API (Python 3.12).
• Implement the initialization function using PEP 793 module slots API (Python 3.15).

Keeping support for old Python versions in a single code base seems to be non trivial.

---

If _Py_OPAQUE_PYOBJECT is defined, PyModuleDef is no longer defined, and so PEP 793 PyModule_FromSlotsAndSpec() API must be used to implement a C extension, right?

It seems like _Py_OPAQUE_PYOBJECT is only usable with Py_LIMITED_API >= Python 3.15. I tried to use with Python 3.13 Py_LIMITED_API, I got build errors (ex: Py_TYPE() is not defined, which is surprising).

I hacked Modules/xxlimited.c to use _Py_OPAQUE_PYOBJECT. I had to copy/paste xx_slots which sounds bad in terms of maintenance :-( It would be nice to have a way to reuse xx_slots in both code paths (to avoid duplicating code). Now I better understand the nested slots feature of PEP 820! If PEP 820 is not accepted, I would suggest adding Py_mod_slots slot to refer to another PyModuleDef_Slot array.

#ifdef _Py_OPAQUE_PYOBJECT
PyABIInfo_VAR(abi_info);

static PyModuleDef_Slot xxmodule_slots[] = {
   {Py_mod_abi, &abi_info},
   {Py_mod_name, "xxlimited"},
   {Py_mod_doc, (void*)module_doc},
   {Py_mod_state_size, (void*)sizeof(xx_state)},
   {Py_mod_methods, xx_methods},
   {Py_mod_state_traverse, xx_traverse},
   {Py_mod_state_clear, xx_clear},
   {Py_mod_state_free, xx_free},

   {Py_mod_exec, xx_modexec},
   {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
   {Py_mod_gil, Py_MOD_GIL_NOT_USED},
   {0, NULL},
};
#else
static PyModuleDef_Slot xx_slots[] = {
    {Py_mod_exec, xx_modexec},
    {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
    {Py_mod_gil, Py_MOD_GIL_NOT_USED},
    {0, NULL}
};

static struct PyModuleDef xxmodule = {
    PyModuleDef_HEAD_INIT,
    .m_name = "xxlimited",
    .m_doc = module_doc,
    .m_size = sizeof(xx_state),
    .m_methods = xx_methods,
    .m_slots = xx_slots,
    .m_traverse = xx_traverse,
    .m_clear = xx_clear,
    .m_free = xx_free,
};
#endif

_Py_OPAQUE_PYOBJECT prevents defining types using PyObject_HEAD. Types should be converted to PEP 697 API. That's explained in PEP 803.

[ngoldbaum]

Keeping support for old Python versions in a single code base seems to be non trivial.

I agree that this is probably painful for a pure C or C++ extension that doesn't use any bindings generators.

However, for bindings generators, it's not so bad. Particularly if they've already added support for PEP 697 extensible types. This is true for both nanobind and PyO3, where the patches to support PEP 803 aren't all that large:

PyO3/pyo3#5807
wjakob/nanobind@master...abi3t

It's also not a problem for CFFI because CFFI doesn't produce Python types or modules in extensions - all the Python types it defines happen in CFFI itself, and CFFI doesn't currently ship limited API wheels.

In practice, the vast majority of projects shipping limited API wheels today do so through bindings generators. I'm not aware of very many (any?) pure C or C++ extensions that ship limited API wheels.

Note that nanobind also needs a new stable ABI function to get instance data without any knowledge of static types in CPython: https://discuss.python.org/t/pep-803-round-two-abi3t-stable-abi-for-free-threaded-builds/106181/18, which is a C API workgroup proposal that will come after this one I think.

[encukou]

ping @erlend-aasland @serhiy-storchaka @zooba

[encukou]

ping @serhiy-storchaka @zooba

Note that I have a PR open for the PEP.
The “hard” specification change is about filename tags (.abi3t.so), so out of WG scope.
The PR does reorganize a bunch of “fluff” (rationale, rejected ideas, &c.) -- if you want to know my current thinking, you can read it.
The PR downplays the possibility of abi3t&abi3 diverging (in response to informal private SC feedback).
It also argues against the term (not the concept) "Limited API". I plan to avoid that term in new documentation that I'll write, so please stop me if you like it.

[encukou]

The SC has accepted the PEP (and also the inactive voter timeout is over); I'll close the issue.

If you have any more concerns, there's still time for discussion of course.