R09: writing a C extension properly

[tamnd]

R09 is the last lesson in M8, and the first one in the whole set that writes C rather than reading it. Everything up to here has watched CPython from the Python side. This one crosses over, because the rules you have been reading about stop being descriptions the moment you are the one writing the extension.

There are four rules and none of them is hard. Hand back what you own and let go of what you borrowed. If your object can hold another object, tell the collector. Do your cleanup in the slot meant for it. Say honestly whether your code is safe without the lock. What makes them worth a lesson is that breaking any of the four produces no error, no warning and no crash. It produces a program that works and slowly stops working.

What the notebook runs

The error path first, because the other three are versions of it. Two nearly identical C functions are compiled side by side. Both pack their argument into a tuple twice over and then hash it, which works for a number and fails for a list. One of them drops the tuple on the way out through the failure and the other does not. A thousand failing calls later, one has left two thousand references behind and the other none, and nothing in the process has said a word. The fix is one line, and the habit behind it is a single cleanup block at the bottom reached by goto, which is the shape CPython's own source uses for exactly this reason.

Then the collector, which needs a container to show. A box holding one object is compiled three ways. Loose has no GC flag at all. Half has the flag and a tp_traverse that reports its type and forgets its field. Tracked reports both. Each one goes into a cycle, the collector runs, and only Tracked is ever freed. That is worth sitting with: Half is registered with the collector, gets asked what it is holding, gives an answer that is merely incomplete, and leaks exactly as thoroughly as the type that was never registered at all. The reason is subtract_refs, which takes one reference off per visit landing inside the candidate set and treats whatever is still above zero as held from outside. An incomplete answer is indistinguishable from a live reference.

The table in that section puts gc.get_referents next to whether the box was freed, which is the closest thing to a unit test you can write for a traverse handler: if the referents do not match the fields your object owns, you have the Half bug.

Then teardown. tp_clear belongs to the collector and only breaks links. tp_finalize is where your cleanup goes, and the cell shows it running exactly once on both routes to death, once for an object whose count hit zero and once for an object the collector found in a cycle.

Then the line at the bottom of the file. The boxes module in this lesson declares Py_MOD_GIL_USED, and that is honest rather than modest: state->freed++ really would lose counts under two threads. The lesson says what keeping the other promise would have cost.

The last section is a leak test, because four rules that fail silently need one. It is CPython's -R flag with the interesting parts removed, about a dozen lines, and it catches both broken boxes and clears the correct one.

Recordings

Two Tier 1 experiments, both on builds a reader does not have.

r09-what-the-leak-hunter-catches runs on the debug build. Four small tests, run once the way anybody runs a test suite and once under -R 3:3. All four pass the ordinary run. Two fail the hunter, and a third gets a note without failing, because it left something behind on the first measured run and then stopped, which is a cache filling rather than a leak. That last case is why the hunter only fails something when every measured run leaked.

r09-what-a-module-must-declare runs on the free threaded build. CPython's own test suite ships a shared object exporting two init functions that differ only in whether they declare the module safe without the lock, so the recording loads each one in a child of its own and prints sys._is_gil_enabled() before and after. One of the three loads turns the lock back on for the whole process and prints the warning naming the module. The same load under PYTHON_GIL=0 leaves it off and says nothing.

Running it in a browser

Every cell that needs a compiler is guarded the way C07 does it, so a reader on Pyodide gets a line saying there is nothing to build rather than a traceback, and the prose and the two recordings still carry the lesson. The probe confirms all eight cells run end to end.

The rest

Six diagrams, three new glossary terms (traverse function, clear function, module state), rows in all three READMEs, and sixteen citations into the pinned tree covering Py_VISIT, Py_CLEAR, Py_TPFLAGS_HAVE_GC, visit_decref, subtract_refs, delete_garbage, PyObject_GC_Track, _PyObject_GC_New, PyObject_CallFinalizerFromDealloc, subtype_traverse, PyType_GetModuleByDef, PyModule_GetState, the two halves of the refleak runner, and the alternate init function in _testmultiphase.

The _DuringGC family is new in 3.15, and the list of what a tp_traverse handler may call now includes it, so that got a diagram of its own.