Skip to content

feat: create types with PyType_FromMetaclass on CPython 3.12+ - #6231

Open
henryiii wants to merge 6 commits into
pybind:masterfrom
henryiii:feat/type-creation-via-spec
Open

henryiii wants to merge 6 commits into
pybind:masterfrom
henryiii:feat/type-creation-via-spec

Conversation

@henryiii

@henryiii henryiii commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Description

On CPython 3.12+, pybind11 now creates its type objects (pybind11_static_property, the pybind11_type metaclass, pybind11_object, and all bound classes) with PyType_FromMetaclass() and a PyType_Spec. It no longer fills in PyHeapTypeObject fields by hand there.

  • The PYBIND11_TYPE_CREATION_VIA_SPEC macro (0/1) controls this. It defaults to 1 on CPython 3.12+ and to 0 on older CPython, PyPy, and GraalPy. Define it as 0, or configure with -DPYBIND11_TYPE_CREATION_VIA_SPEC=OFF, to keep the legacy path. A new CI job tests the legacy path on 3.12+.
  • py::custom_type_setup(), metaclasses with a custom tp_new, and metaclass combinations that PyType_FromMetaclass() rejects still use the legacy path. One example of a rejected combination is a base whose metaclass is not a base or subtype of pybind11_type. Without the fallback, binding a class derived from such a base fails with a metaclass conflict. A new test covers this case.
  • If the user defines PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET, the legacy path is the default. The spec path uses Py_TPFLAGS_MANAGED_DICT, and all bound classes must have the same __dict__ layout. PYBIND11_TYPE_CREATION_VIA_SPEC=1 together with that macro is a compile error.
  • The instance layout and PYBIND11_INTERNALS_ID do not change. The macro is added to the precompiled config check.

This prepares for stable-ABI support (#6104), but it is also useful without it.

Suggested changelog entry:

  • On CPython 3.12+, type objects are now created with PyType_FromMetaclass(). Define PYBIND11_TYPE_CREATION_VIA_SPEC=0 (CMake: -DPYBIND11_TYPE_CREATION_VIA_SPEC=OFF) to use the previous PyHeapTypeObject path.

📚 Documentation preview 📚: https://pybind11--6231.org.readthedocs.build/

pybind11_static_property, the pybind11_type metaclass, pybind11_object
and the classes from make_new_python_type() are now created from a
PyType_Spec on CPython 3.12+ instead of by filling in PyHeapTypeObject
fields. PYBIND11_TYPE_CREATION_VIA_SPEC (0/1) controls this; define it
as 0, or configure with -DPYBIND11_TYPE_CREATION_VIA_SPEC=OFF, to keep
the legacy path. PyPy, GraalPy and Python < 3.12 keep the legacy path.

py::custom_type_setup(), metaclasses with a custom tp_new, and metaclass
combinations that PyType_FromMetaclass() rejects (such as a base whose
metaclass is not related to pybind11_type) still use the legacy path. A
user-defined PYBIND11_BACKWARD_COMPATIBILITY_TP_DICTOFFSET also selects
the legacy path, so all bound classes share one __dict__ layout.

The instance layout and the internals ID do not change. The macro is
encoded in the precompiled config check. A new CI job tests the legacy
path on CPython 3.12+.

See pybind#6104.

Assisted-by: ClaudeCode:claude-opus-5-5
@rwgk

rwgk commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

Initial codex GPT-6.1-Sol ultra review below. Before the review, I locally merged latest master.

I'll use the same codex session to generate a fix.


I found one reproducible regression, treating the Python 3.12+ switch as intentional.

[P2] Preserve supplied class names containing dots — class-inl.h:840–841.

For example:

py::class_<T>(m, "Wrapper[pkg.Value]");

The new path changes __name__ and top-level __qualname__ from Wrapper[pkg.Value] to Value]: PyType_FromMetaclass() interprets the last dot as the module/name separator. The module attribute retains the original name, leaving repr and introspection inconsistent.

This reproduces on Python 3.12, 3.14, and free-threaded 3.14. Master and the legacy opt-out preserve the supplied names. Explicitly restoring __name__ and __qualname__ after construction should address it.

Your local merged revision passed:

  • Default Python: 1,404 passed, 6 skipped.
  • Free-threaded Python: 1,402 passed, 8 skipped.
  • Both Catch2 runs: 39 cases, 11,444 assertions passed.

No other introduced actionable issues emerged from the independent reviews.

rwgk added 2 commits October 31, 2026 22:44
— codex GPT-6 on behalf of Ralf —

Use the existing legacy type-creation fallback when a bound name contains
a dot. PyType_FromMetaclass() treats the last dot as the module separator,
truncating names such as Wrapper[pkg.Value].

Preserve both the public __name__/__qualname__ and the fully qualified C
tp_name. Assigning __name__ after creation would also overwrite tp_name.

Add top-level and nested regression cases; both fail without the fix.
@rwgk

rwgk commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

In case anyone is looking at the moment: I'm having doubts myself about the direction taken with commit f9eda9d. I'm still working on making up my mind.

rwgk added 2 commits October 31, 2026 23:52
— codex GPT-6 on behalf of Ralf —

Revert f9eda9d so the naming behavior
introduced by spec-based type creation can be characterized across CI.
The naming policy will be discussed separately after collecting results.
— codex GPT-6 on behalf of Ralf —

Record name, qualified name, literal lookup, construction, reference-return
identity and generated signatures for dotted and bracketed class names.
Include nested scopes, a child of a dotted class, and custom type setup.

Keep expectations specific to the current creation path and Python runtime.
These tests collect evidence for the naming-policy discussion without
choosing whether to retain, reject or accommodate the changed behavior.
@rwgk

rwgk commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

Update: I'm using the CI here to fully characterize the behavior changes introduced by this PR.

I'm uncertain if we should care about them or not. I believe it's best to have complete ground truth, then we can decide.

— codex GPT-6 on behalf of Ralf —

Select true/false with the preprocessor instead of casting the numeric
configuration macro to bool. This satisfies modernize-use-bool-literals
in Clang-Tidy, including the Python3.11 macro-zero branch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants