CI: select tests by what a pull request's changes reach #369

CI could run only the tests a pull request's changes can reach, chosen from a map of which source files each test used on main. This builds on the path filter in #368, which skips test jobs when the docs, examples or tools alone change.

Measurement (2026-11-05)

How it was measured

• The map: the whole CPU suite on main (dc4ca7a2), recorded per test. Each test's map lists:
  • the repository functions it ran, recorded with sys.monitoring (PY_START, each location disabled after its first hit and re-enabled at the next test);
  • the files of every kernel it launched, and of everything lowering inlined into those kernels. Python never runs kernel bodies, so these come from hooks on Kernel.__call__, Kernel._lower and CallBindings.resolve;
  • what its subprocesses ran, through a sitecustomize.
• Fixture setup is credited to every test in its file, since a module-scoped fixture is built during whichever test needs it first.
• The recording run: 6755 passed, 371 skipped, 6 min 17 s on 8 workers on an M1 Max. The overhead is negligible.
• The replay:
  • the 278 PRs merged from 2026-10-14 to 2026-11-05, each diffed from its merge commit's first parent;
  • every CI run on a pull request in that period that failed with named tests: 18 runs, each diffed against main as it was when the run started.
• Weights: test-durations.json, with each file's seconds spread over its tests by their local durations.
• Rules:
  • a changed test file selects all of its tests;
  • the build and CI files (pyproject.toml, uv.lock, conftest.py, .github/, test-durations.json) run everything;
  • a file under packages/*/src/ that no test used runs everything;
  • a file the PR adds selects nothing by itself; the changed files that use it select the tests.

Results

With selection by test, the 278 PRs split as follows:

Failures: none missed. 9 of the 18 failing runs got a partial selection, with 13 failing tests between them, and every one was selected. The vis failures would have been found by running 12–31% of the suite:

The core and runtime failures were selected at 99–100%. The other 9 runs would have run everything.

What keeps runs broad

• Files nearly every test really uses. In the 90%+ PRs, the broadest changed file was:

  File	PRs	Share of suite that uses it
  dataset/views.py	16	96%
  runtime/cpu.py	11	99%
  dataset/topology.py	10	96%
  lang/array.py	9	99%

  The rest were dataset.py, shapes.py, _common.py, ghosts.py and units.py. Narrowing these would need function granularity: map a diff's changed lines to the functions they're in, and select the tests that ran those functions.

• Full runs came from ci.yml (28 PRs, this period's CI work) and from tack/dataset/__init__.py (29 PRs). That file holds only module-level code, so no test is credited with it; usually the change is a new algorithm's export, whose implementing files are selected anyway. A rule for module-only files could recover most of those.

What it would take

1. The recorder, as a test-only pytest plugin. The prototype is about 150 lines: sys.monitoring, plus the three Tack hooks above.
2. A map refreshed on every push to main, for example by one Linux share uploading it as an artifact, or as a committed file like test-durations.json. Pushes to main keep running everything, as the safety net that refreshes the map.
3. Selection by node ID in the shares (conftest's --shard), with each share dealt the selected tests by their weights.
4. Rules for unmapped files, module-only files and data files under tests/.
5. Metal: run it on the same selection, or only when Metal's backend or the core changes. The map comes from CPU runs, so GPU-only files are unmapped and force a full run, which is safe.

Risks

• A result cached in Python (an lru_cache, a module-level dataset) credits only the first test that computes it, so a later test that depends on it could be missed. None of the 13 failures hit this, but the sample is small.
• A dependency added by a PR is unknown to main's map. The full run on the push to main catches it, after the merge.
• Wall time is set mostly by Metal's runner queue and the longest share, so it falls less than test time does.

Not before 0.7.