Add R06, the C API tiers

README.md

@@ -164,6 +164,7 @@ A lesson can also end with a boss fight, which is a problem the text does not so
 | R03 | [What import does](lessons/r03-what-import-does/r03.ipynb) | An import statement compiles to one IMPORT_NAME plus zero or more IMPORT_FROM, and IMPORT_NAME looks the name __import__ up in builtins every single time, which is why replacing it works. Compiling the four spellings shows that import a.b binds a rather than a.b, and that a relative import is the empty string at a level above zero. A finder put on the front of sys.meta_path that answers nothing and writes down every question shows a dotted import searching for each part in turn from the outside in, with everything after the first part looked for in the parent package's __path__. A fresh interpreter has three finders, and import os stops at the second of them, so os.py is never opened. The module object goes into sys.modules before its body runs, which is what makes circular imports work and what decides how much of a half loaded module the other side can see, and if the body raises the entry is taken back out. Three caches sit in the way of a repeat import, and a directory created after it was first looked for stays invisible until importlib.invalidate_caches is called. Two recordings then settle the import lock: it is one lock per module name, so four threads importing four different modules keep one core busy on a build with the GIL and three and a half on a build without | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r03-what-import-does/r03.ipynb) |
 | R04 | [Frozen modules](lessons/r04-frozen-modules/r04.ipynb) | The import system is written in Python, so it cannot be imported, and the way out of that is to compile a handful of modules during CPython's own build and write the bytecode into the binary as C arrays. The module body of _frozen_importlib contains zero IMPORT_NAME opcodes, which is what makes it loadable with no import system running, and init_importlib in C hands sys and _imp to it as arguments. Thirty three names are frozen in a stock 3.15 build, in three groups that the flag treats differently: three for the import system that can never be switched off, nineteen for what a bare startup needs, and eleven hello world modules for the test suite. A frozen module still knows where it came from, because the loader puts the original path in loader_state and copies it onto __file__, which is why inspect.getsource and tracebacks still work while co_filename says <frozen os>. Switching the flag off in process with _imp._override_frozen_modules_for_tests moves os from FrozenImporter to SourceFileLoader and leaves the same constants behind. What freezing actually buys is the finder search and the file read, not the unmarshal, which is identical on both paths, and two recordings put it at about a tenth of a startup on a release build and a tenth on a debug build, where it is off by default | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r04-frozen-modules/r04.ipynb) |
 | R05 | [Lazy imports](lessons/r05-lazy-imports/r05.ipynb) | PEP 810 adds one word to the import statement in 3.15, and lazy import json binds the name now and does the finding, reading and running of the module the first time something reads that name back. It compiles to the same IMPORT_NAME opcode as a plain import, with the name index shifted up by two bits and the two bits underneath saying lazy, forced eager or ordinary, so dis prints json, json + lazy or json + eager. What gets bound is not a module but a five field placeholder that is not in sys.modules, that you can look at without setting it off, and whose name sits in sys.lazy_modules until it resolves. Only two opcodes resolve one, reading it as a bare name and reading it as an attribute of a module, so dict lookups, membership tests and reprs all leave it alone, and a placeholder copied into another variable resolves when that variable is read rather than when it was copied. The scope rules come from the symbol table rather than the code generator, which is why a lazy import inside a try block is refused with its own message. A failure inside a deferred module arrives with a second exception attached as its cause, pointing back at the lazy import line from information the placeholder was carrying. Three recordings put the startup saving at about three quarters of a run, and show that resolution takes the interpreter wide import lock rather than a lock per module name, so on a free threaded build four threads waking four different deferred imports keep 0.98 cores busy while four ordinary imports keep 3.60 | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r05-lazy-imports/r05.ipynb) |
+| R06 | [The C API tiers](lessons/r06-the-c-api-tiers/r06.ipynb) | The C API is three directories and two macros. Include is open to any extension, Include/cpython needs Py_LIMITED_API to be undefined, and Include/internal starts nearly every file with three lines that stop the compiler unless you define Py_BUILD_CORE. More than half the header lines are in that third directory. Defining Py_LIMITED_API hides 186 of the 766 functions the public headers declare and all 974 in the other two, and the part that costs is not the functions but the struct layouts, because with no fields to read Py_TYPE becomes a call and Py_DECREF becomes a call to _Py_DecRef. The naming convention nearly matches the directories and the exceptions have a reason: a private name has to be exported when a public macro expands to it, which is what the 17 underscore names in the public tier are. None of it survives the build. Every tier resolves through ctypes.pythonapi, and calling _PyDict_SizeOf by hand gives the same number dict.__sizeof__ does, 16 bytes short of sys.getsizeof because that adds the collector header. Two recordings show the split is deliberate: inside the internal headers 93 percent of the names spelled PyAPI_FUNC resolve against 0.4 percent of the ones spelled plain extern, with 168 comments naming which bundled extension needs each export | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r06-the-c-api-tiers/r06.ipynb) |
 
 More are landing in order. [lessons/README.md](lessons/README.md) explains how one is put together and how to run them locally.
 

citations.lock.json

@@ -1,5 +1,15 @@
 {
   "citations": {
+    "Doc/c-api/stable.rst:21-35@v3.15.0rc1": {
+      "digest": "9299cb98031d1463",
+      "first_line": "There are two tiers of C API with different stability expectations:",
+      "lines": 15
+    },
+    "Doc/c-api/stable.rst:44-52@v3.15.0rc1": {
+      "digest": "16edf2164cc5ad02",
+      "first_line": "Any API named with the ``PyUnstable`` prefix exposes CPython implementation",
+      "lines": 9
+    },
     "Grammar/python.gram:121-124@v3.15.0rc1": {
       "digest": "816f44568467b4ce",
       "first_line": "simple_stmt[stmt_ty] (memo):",
@@ -290,6 +300,11 @@
       "first_line": "#define E_EOF            11      /* End Of File */",
       "lines": 2
     },
+    "Include/exports.h:88-93@v3.15.0rc1": {
+      "digest": "a1f585b73b10bbb8",
+      "first_line": "#ifndef PyAPI_FUNC",
+      "lines": 6
+    },
     "Include/internal/mimalloc/mimalloc/types.h:203-211@v3.15.0rc1": {
       "digest": "ec93e4470862f3c7",
       "first_line": "#define MI_SEGMENT_SLICE_SHIFT            (13 + MI_INTPTR_SHIFT)         // 64KiB  (32KiB on 32-bit)",
@@ -400,6 +415,11 @@
       "first_line": "static inline uint8_t *",
       "lines": 17
     },
+    "Include/internal/pycore_dict.h:50-54@v3.15.0rc1": {
+      "digest": "67e2c9427216744a",
+      "first_line": "",
+      "lines": 5
+    },
     "Include/internal/pycore_dict.h:79-90@v3.15.0rc1": {
       "digest": "c3ccf876d4deb3d8",
       "first_line": "",
@@ -645,6 +665,11 @@
       "first_line": "/* Tries to incref the object op and ensures that *src still points to it. */",
       "lines": 16
     },
+    "Include/internal/pycore_object.h:7-9@v3.15.0rc1": {
+      "digest": "880508f79b0c0812",
+      "first_line": "#ifndef Py_BUILD_CORE",
+      "lines": 3
+    },
     "Include/internal/pycore_object.h:83-89@v3.15.0rc1": {
       "digest": "70fca12f791324cc",
       "first_line": "#if SIZEOF_VOID_P > 4",
@@ -935,6 +960,11 @@
       "first_line": "#define PyList_Check(op) \\",
       "lines": 3
     },
+    "Include/modsupport.h:85-101@v3.15.0rc1": {
+      "digest": "88ce91d467657676",
+      "first_line": "/* ABI info & checking (new in 3.15) */",
+      "lines": 17
+    },
     "Include/moduleobject.h:78-89@v3.15.0rc1": {
       "digest": "391087226de90dc4",
       "first_line": "/* for Py_mod_multiple_interpreters: */",
@@ -945,6 +975,11 @@
       "first_line": "/* for Py_mod_gil: */",
       "lines": 5
     },
+    "Include/object.h:124-126@v3.15.0rc1": {
+      "digest": "1f03faa6c6b4cdc0",
+      "first_line": "#ifdef _Py_OPAQUE_PYOBJECT",
+      "lines": 3
+    },
     "Include/object.h:127-149@v3.15.0rc1": {
       "digest": "c7e17988b3729573",
       "first_line": "struct _object {",
@@ -980,6 +1015,11 @@
       "first_line": "struct PyVarObject {",
       "lines": 5
     },
+    "Include/object.h:185-198@v3.15.0rc1": {
+      "digest": "82cf6802185e7cbd",
+      "first_line": "// Test if the 'x' object is the 'y' object, the same as \"x is y\" in Python.",
+      "lines": 14
+    },
     "Include/object.h:237-244@v3.15.0rc1": {
       "digest": "dc45c7bd6a91c2b3",
       "first_line": "// bpo-39573: The Py_SET_SIZE() function must be used to set an object size.",
@@ -1010,6 +1050,11 @@
       "first_line": "/* Macro for returning Py_None from a function.",
       "lines": 7
     },
+    "Include/object.h:740-744@v3.15.0rc1": {
+      "digest": "209c33f89cca4826",
+      "first_line": "#ifndef Py_LIMITED_API",
+      "lines": 5
+    },
     "Include/opcode_ids.h:1-4@v3.15.0rc1": {
       "digest": "427ac74efb62f52a",
       "first_line": "// This file is generated by Tools/cases_generator/opcode_id_generator.py",
@@ -1045,11 +1090,21 @@
       "first_line": "#if SIZEOF_VOID_P > 4",
       "lines": 28
     },
+    "Include/refcount.h:238-241@v3.15.0rc1": {
+      "digest": "da14159986fa309d",
+      "first_line": "#endif  // Py_REF_DEBUG && !Py_LIMITED_API",
+      "lines": 4
+    },
     "Include/refcount.h:285-292@v3.15.0rc1": {
       "digest": "8ae8e77b2ccded13",
       "first_line": "#elif SIZEOF_VOID_P > 4",
       "lines": 8
     },
+    "Include/refcount.h:327-338@v3.15.0rc1": {
+      "digest": "92fb9a053d91b091",
+      "first_line": "#if (defined(Py_LIMITED_API) && (Py_LIMITED_API+0 >= 0x030c0000 || defined(Py_REF_DEBUG))) \\",
+      "lines": 12
+    },
     "Include/refcount.h:417-429@v3.15.0rc1": {
       "digest": "a2af4e3c17465fe0",
       "first_line": "static inline Py_ALWAYS_INLINE void Py_DECREF(PyObject *op)",
@@ -4980,6 +5035,11 @@
       "first_line": "size_t",
       "lines": 16
     },
+    "Python/sysmodule.c:1970-1979@v3.15.0rc1": {
+      "digest": "5090fe7bc65883c4",
+      "first_line": "",
+      "lines": 10
+    },
     "Python/sysmodule.c:2015-2033@v3.15.0rc1": {
       "digest": "2aa5d7a83af6b2ee",
       "first_line": "/*[clinic input]",

experiments/README.md

@@ -47,6 +47,8 @@ So those programs run somewhere else. They run in the images this project publis
 | [r05-what-deferring-an-import-is-worth](tier1/r05-what-deferring-an-import-is-worth.md) | R05 | release | What does a program get back for not importing what it turns out not to need? |
 | [r05-how-much-of-a-wake-up-is-parallel](tier1/r05-how-much-of-a-wake-up-is-parallel.md) | R05 | release | Do two threads waking up two different deferred imports wait for each other? |
 | [r05-how-much-of-a-wake-up-is-parallel-without-the-lock](tier1/r05-how-much-of-a-wake-up-is-parallel-without-the-lock.md) | R05 | freethreaded | With the global interpreter lock gone, does waking up a deferred import scale? |
+| [r06-what-leaves-the-binary](tier1/r06-what-leaves-the-binary.md) | R06 | release | How much of the C API that the headers call private is callable anyway? |
+| [r06-what-leaves-the-binary-on-a-free-threaded-build](tier1/r06-what-leaves-the-binary-on-a-free-threaded-build.md) | R06 | freethreaded | Does dropping the global interpreter lock change what the C API exports? |
 
 ## The commands
 

experiments/tier1/r06-what-leaves-the-binary-on-a-free-threaded-build.md

@@ -0,0 +1,136 @@
+# The same sweep on the build that compiles a different half of the headers
+
+Generated by `just build-tier1`. Do not edit by hand, the change will be overwritten.
+
+Does dropping the global interpreter lock change what the C API exports?
+
+- Lesson: R06
+- Build: freethreaded
+- Image: ghcr.io/tamnd/cpython-internals/cpython:freethreaded@sha256:db72284e3a49f43c38b96bec2baed1380b8348e27ea6f54f6e8d0810b59c3144
+- Interpreter: 3.15.0rc1 free-threading build (37e98da:37e98da, Aug 29 2026, 09:25:00) [GCC 14.2.0]
+- Recorded: 2026-09-06
+
+Why this needs the freethreaded build: it needs a build configured with --disable-gil, because the question is whether the names behind Py_GIL_DISABLED are the ones that were missing on the ordinary build.
+
+## The program
+
+```python
+"""Which of the C API's private declarations actually leave the binary.
+
+The headers put the C API in three directories. `Include/` is what any extension may use,
+`Include/cpython/` is the part that only makes sense compiled against this exact CPython, and
+`Include/internal/` says at the top of nearly every file that it will not compile unless you
+claim to be CPython itself.
+
+That is all a compile time arrangement. This program asks what survives into the built
+interpreter, by taking every name the headers declare and asking the dynamic linker for it
+through `ctypes.pythonapi`. A name that resolves is a name any program can call, whatever the
+header said about it.
+
+The interesting split is inside the internal headers, which use two different spellings.
+`PyAPI_FUNC` means the symbol leaves the shared library. A plain `extern` means it does not.
+Both spellings sit in the same file, often two lines apart.
+"""
+
+import ctypes
+import pathlib
+import re
+import sys
+import sysconfig
+from collections import Counter
+
+API = re.compile(r"^PyAPI_FUNC\([^)]*\)\s*\**\s*(\w+)", re.M)
+EXTERN = re.compile(r"^extern\s+[\w *]+?\**\s*(\w+)\s*\(", re.M)
+NOTE = re.compile(r"^//\s*Export for (.+?)\.?$", re.M)
+
+TIERS = (("public", "*.h"), ("cpython only", "cpython/*.h"), ("internal", "internal/*.h"))
+
+api = ctypes.pythonapi
+include = pathlib.Path(sysconfig.get_paths()["include"])
+
+
+def resolves(name):
+    """Ask the linker for a name, the way any program with a handle on the process can."""
+    return hasattr(api, name)
+
+
+def read(pattern):
+    """Every header matching the pattern, as one blob of text per file."""
+    return [path.read_text(errors="replace") for path in sorted(include.glob(pattern))]
+
+
+print("include directory:", include)
+print("build has the gil disabled:", sysconfig.get_config_var("Py_GIL_DISABLED"))
+print("abi flags:", repr(sys.abiflags))
+print()
+
+for tier, pattern in TIERS:
+    blobs = read(pattern)
+    declared = set()
+    for blob in blobs:
+        declared |= set(API.findall(blob))
+    found = sum(1 for name in declared if resolves(name))
+    print(f"{tier}: {len(blobs)} header files")
+    print(f"  declared with PyAPI_FUNC: {len(declared)}")
+    print(f"  of those, resolve in this process: {found}")
+
+internal = read("internal/*.h")
+exported = set()
+kept_in = set()
+notes = []
+for blob in internal:
+    exported |= set(API.findall(blob))
+    kept_in |= set(EXTERN.findall(blob))
+    notes += NOTE.findall(blob)
+kept_in -= exported
+
+leaked = sum(1 for name in exported if resolves(name))
+held = sum(1 for name in kept_in if resolves(name))
+
+print()
+print("inside the internal headers")
+print("  names spelled PyAPI_FUNC:", len(exported))
+print("  names spelled plain extern:", len(kept_in))
+print("  comments naming who needs the export:", len(notes))
+for who, count in Counter(notes).most_common(5):
+    print(f"    {count} for {who}")
+
+print()
+exported_share = leaked / len(exported) * 100
+extern_share = held / len(kept_in) * 100
+print("~ private names that leave the binary: {}".format(leaked))
+print("~ share of PyAPI_FUNC internal names that resolve: {:.1f} percent".format(exported_share))
+print("~ share of plain extern internal names that resolve: {:.1f} percent".format(extern_share))
+```
+
+## What it printed
+
+```text
+include directory: /opt/python/include/python3.15t
+build has the gil disabled: 1
+abi flags: 't'
+
+public: 79 header files
+  declared with PyAPI_FUNC: 770
+  of those, resolve in this process: 754
+cpython only: 63 header files
+  declared with PyAPI_FUNC: 445
+  of those, resolve in this process: 440
+internal: 148 header files
+  declared with PyAPI_FUNC: 530
+  of those, resolve in this process: 499
+
+inside the internal headers
+  names spelled PyAPI_FUNC: 530
+  names spelled plain extern: 757
+  comments naming who needs the export: 168
+    25 for '_testinternalcapi' shared extension
+    12 for test_peg_generator
+    10 for '_datetime' shared extension
+    9 for '_asyncio' shared extension
+    7 for 'math' shared extension
+
+~ private names that leave the binary: 499
+~ share of PyAPI_FUNC internal names that resolve: 94.2 percent
+~ share of plain extern internal names that resolve: 0.4 percent
+```