Add R03, what import does - #181
Merged
Merged
Conversation
The third runtime lesson. R02 ended by noting that a new interpreter imports dozens of modules of its own before it will run a line, so this one opens the import system up. Ten code cells, seventeen citations, six diagrams, four glossary terms and two Tier 1 recordings. The lesson works by watching rather than describing. Compiling the four spellings of the statement shows there are only two opcodes involved, that import a.b binds a, and that a relative import is the empty string at a level above zero. A finder put on the front of sys.meta_path that answers nothing and writes down every question shows a dotted import searching for each part in turn from the outside in, with every part after the first looked for in the parent package's __path__. Asking the three finders for the same three names side by side shows that a name can have more than one answer and the earlier finder wins, which is why import os never opens os.py. The payoff cell is the circular import. A module object goes into sys.modules before its body runs, so the second of two modules that import each other reads back whatever the first had defined by the line that triggered the import, and if a body raises the entry is taken back out again. That is _load_unlocked, lines 908 and 918, and it explains more real bugs than anything else here. After that a fourteen line class serves a module out of a string, and a cell walks the three caches an import passes through, including the one that keeps a directory you have just created invisible until invalidate_caches is called. The two recordings settle the import lock. It is one lock per module name, not one lock for the process, so four threads importing four different modules keep 1.02 cores busy on a release build and 3.63 on a free threaded one. What serialises them is the GIL. Four threads importing the same module keep almost exactly one core busy on both builds and the body runs once, which is the per module lock doing its job. Writing that experiment took two goes. The first version compared wall clock for four sequential imports against four threaded ones and reported a ratio that would have meant bytecode running in parallel with the GIL held. It was measuring macOS core assignment rather than Python: one busy thread gets put on an efficiency core and four runnable threads get promoted. Measuring processor time over wall clock instead takes the machine back out of the answer. New glossary terms: module spec, meta path finder, path entry finder and module lock. GLOSSARY.md is 222 terms, CLAIMS.md is 580 claims across 74 lessons, and citations.lock.json is 983 entries. Part of #27.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
R03 is the third of the nine runtime lessons. R02 ended by noting that a brand new interpreter imports dozens of modules of its own before it will run a line of your code, so this one opens the import system up.
The framing is that
importis not a keyword doing something the language will not explain. It compiles to a call to an ordinary builtin, that builtin is written in Python in a file you can open, and every part of what it does is reachable from inside the language. So the lesson watches rather than describes.Ten code cells. Compiling the four spellings of the statement shows there are only two opcodes involved, that
import a.bbindsarather thana.b, and that a relative import is the empty string at a level above zero. A finder put on the front ofsys.meta_paththat answers nothing and writes down every question makes a dotted import visible as three separate searches, outermost first, with every part after the first looked for in the parent package's__path__rather than onsys.path. Asking the three finders for the same three names side by side shows that a name can have more than one answer and the earlier finder wins, which is whyimport osnever opensos.py. That is also the hand off to R04.The payoff cell is the circular import. A module object goes into
sys.modulesbefore its body runs, which is what makes circular imports possible at all, and it decides exactly how much of a half loaded module the other side can see: whatever was defined by the line that triggered the import, and nothing after it. If the body raises, the entry is deleted again, so a failed import does not leave a broken half module behind. Both of those are_load_unlocked, lines 908 and 918.Then a fourteen line class serves a module out of a string with no file on disk anywhere, which is the mechanism behind every import hook anybody has ever used, and a cell walks the three caches an import passes through. The third of those is the one that bites: a directory created after it was first looked for stays invisible, because the failed lookup was cached as
None, untilimportlib.invalidate_caches()is called.The two Tier 1 recordings settle the thing almost everybody has wrong about the import lock. It is one lock per module name, not one lock for the process, and it has been that way since 3.3. Four threads importing four different modules keep 1.02 cores busy on a release build and 3.63 on a free threaded one, so what serialises them is the GIL rather than the import lock. Four threads importing the same module keep almost exactly one core busy on both builds and the body runs exactly once, which is the per module lock doing precisely what it says.
That experiment took two goes and the first one was wrong in an interesting way. It compared wall clock for four sequential imports against four threaded ones and reported a ratio that would have meant bytecode running in parallel with the GIL held. Adding
time.process_time()showed what was really happening: one busy thread gets put on an efficiency core by macOS and four runnable threads get promoted, so the sequential case was being handicapped. Measuring processor time divided by wall clock instead takes the machine back out of the answer, and it reads 1.00 on the control case on both builds.Also in here: four new glossary terms,
module spec,meta path finder,path entry finderandmodule lock. Six diagrams, seventeen citations, both READMEs updated. GLOSSARY.md is 222 terms, CLAIMS.md is 580 claims across 74 lessons, andcitations.lock.jsonis 983 entries.All ten cells run end to end on Pyodide with nothing skipped, which is the first R lesson where that is true. The browser run is more interesting than the native one in two places:
jsoncomes back as loaded by azipimporterinstance because the standard library is in a zip file, and the directory listing has no__pycache__in it because nothing is writing bytecode. Both are covered by thevaries=note on that cell.Part of #27.