Skip to content

Add R03, what import does - #181

Merged
tamnd merged 1 commit into
mainfrom
r03-what-import-does
Sep 5, 2026
Merged

Add R03, what import does#181
tamnd merged 1 commit into
mainfrom
r03-what-import-does

Conversation

@tamnd

@tamnd tamnd commented Sep 5, 2026

Copy link
Copy Markdown
Owner

R03 is the third of the nine runtime lessons. R02 ended by noting that a brand new interpreter imports dozens of modules of its own before it will run a line of your code, so this one opens the import system up.

The framing is that import is not a keyword doing something the language will not explain. It compiles to a call to an ordinary builtin, that builtin is written in Python in a file you can open, and every part of what it does is reachable from inside the language. So the lesson watches rather than describes.

Ten code cells. Compiling the four spellings of the statement shows there are only two opcodes involved, that import a.b binds a rather than a.b, and that a relative import is the empty string at a level above zero. A finder put on the front of sys.meta_path that answers nothing and writes down every question makes a dotted import visible as three separate searches, outermost first, with every part after the first looked for in the parent package's __path__ rather than on sys.path. Asking the three finders for the same three names side by side shows that a name can have more than one answer and the earlier finder wins, which is why import os never opens os.py. That is also the hand off to R04.

The payoff cell is the circular import. A module object goes into sys.modules before its body runs, which is what makes circular imports possible at all, and it decides exactly how much of a half loaded module the other side can see: whatever was defined by the line that triggered the import, and nothing after it. If the body raises, the entry is deleted again, so a failed import does not leave a broken half module behind. Both of those are _load_unlocked, lines 908 and 918.

Then a fourteen line class serves a module out of a string with no file on disk anywhere, which is the mechanism behind every import hook anybody has ever used, and a cell walks the three caches an import passes through. The third of those is the one that bites: a directory created after it was first looked for stays invisible, because the failed lookup was cached as None, until importlib.invalidate_caches() is called.

The two Tier 1 recordings settle the thing almost everybody has wrong about the import lock. It is one lock per module name, not one lock for the process, and it has been that way since 3.3. Four threads importing four different modules keep 1.02 cores busy on a release build and 3.63 on a free threaded one, so what serialises them is the GIL rather than the import lock. Four threads importing the same module keep almost exactly one core busy on both builds and the body runs exactly once, which is the per module lock doing precisely what it says.

That experiment took two goes and the first one was wrong in an interesting way. It compared wall clock for four sequential imports against four threaded ones and reported a ratio that would have meant bytecode running in parallel with the GIL held. Adding time.process_time() showed what was really happening: one busy thread gets put on an efficiency core by macOS and four runnable threads get promoted, so the sequential case was being handicapped. Measuring processor time divided by wall clock instead takes the machine back out of the answer, and it reads 1.00 on the control case on both builds.

Also in here: four new glossary terms, module spec, meta path finder, path entry finder and module lock. Six diagrams, seventeen citations, both READMEs updated. GLOSSARY.md is 222 terms, CLAIMS.md is 580 claims across 74 lessons, and citations.lock.json is 983 entries.

All ten cells run end to end on Pyodide with nothing skipped, which is the first R lesson where that is true. The browser run is more interesting than the native one in two places: json comes back as loaded by a zipimporter instance because the standard library is in a zip file, and the directory listing has no __pycache__ in it because nothing is writing bytecode. Both are covered by the varies= note on that cell.

Part of #27.

The third runtime lesson. R02 ended by noting that a new interpreter imports
dozens of modules of its own before it will run a line, so this one opens the
import system up. Ten code cells, seventeen citations, six diagrams, four
glossary terms and two Tier 1 recordings.

The lesson works by watching rather than describing. Compiling the four
spellings of the statement shows there are only two opcodes involved, that
import a.b binds a, and that a relative import is the empty string at a level
above zero. A finder put on the front of sys.meta_path that answers nothing and
writes down every question shows a dotted import searching for each part in
turn from the outside in, with every part after the first looked for in the
parent package's __path__. Asking the three finders for the same three names
side by side shows that a name can have more than one answer and the earlier
finder wins, which is why import os never opens os.py.

The payoff cell is the circular import. A module object goes into sys.modules
before its body runs, so the second of two modules that import each other reads
back whatever the first had defined by the line that triggered the import, and
if a body raises the entry is taken back out again. That is _load_unlocked,
lines 908 and 918, and it explains more real bugs than anything else here.

After that a fourteen line class serves a module out of a string, and a cell
walks the three caches an import passes through, including the one that keeps a
directory you have just created invisible until invalidate_caches is called.

The two recordings settle the import lock. It is one lock per module name, not
one lock for the process, so four threads importing four different modules keep
1.02 cores busy on a release build and 3.63 on a free threaded one. What
serialises them is the GIL. Four threads importing the same module keep almost
exactly one core busy on both builds and the body runs once, which is the per
module lock doing its job.

Writing that experiment took two goes. The first version compared wall clock
for four sequential imports against four threaded ones and reported a ratio
that would have meant bytecode running in parallel with the GIL held. It was
measuring macOS core assignment rather than Python: one busy thread gets put on
an efficiency core and four runnable threads get promoted. Measuring processor
time over wall clock instead takes the machine back out of the answer.

New glossary terms: module spec, meta path finder, path entry finder and module
lock. GLOSSARY.md is 222 terms, CLAIMS.md is 580 claims across 74 lessons, and
citations.lock.json is 983 entries.

Part of #27.
@tamnd tamnd added this to the M8 Concurrency and runtime milestone Sep 5, 2026
@tamnd tamnd added kind/lesson A chapter: prose, notebook, experiments, boss fight area/runtime Startup, shutdown, import, extension modules and the C API labels Sep 5, 2026
@tamnd
tamnd merged commit a02ad6b into main Sep 5, 2026
16 checks passed
@tamnd
tamnd deleted the r03-what-import-does branch September 5, 2026 21:00
@tamnd tamnd mentioned this pull request Sep 5, 2026
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/runtime Startup, shutdown, import, extension modules and the C API kind/lesson A chapter: prose, notebook, experiments, boss fight

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant