pytest.ini is dead config that silently shadows pyproject.toml (--strict-markers, testpaths, markers all inert)

Part of #108 · Phase 1 · Discovered while red-teaming #109.

Problem

pytest.ini uses the section header [tool:pytest]. That header is only valid in setup.cfg. In a file named pytest.ini, pytest requires [pytest].

pytest still selects the file as its config — and having selected it, stops looking, so pyproject.toml's [tool.pytest.ini_options] is never read.

Verified:

$ pytest tests/unit/test_billable_safety.py --co
rootdir: /Users/jmanning/clustrix
configfile: pytest.ini              <- not pyproject.toml

$ pytest --markers | grep -cE "^@pytest.mark.(expensive|real_world|dartmouth_network)"
0                                   <- no project markers registered at all

$ pytest <file with @pytest.mark.definitely_not_registered_xyz> --co -q
1 test collected                    <- --strict-markers is NOT active

Everything currently inert

Both files declare these; neither takes effect:

Setting	Intended	Actual
`addopts` (`-v --tb=short --strict-markers`, and `-n 4 --dist loadfile` in pyproject)	applied to every run	ignored
`testpaths`	limit collection to `tests`	ignored — bare `pytest` collects the whole repo
`markers` (13 of them)	registered	none registered
`filterwarnings`	suppress paramiko/crypto noise	ignored
`--strict-markers`	typo in a marker = error	inert

Why this matters beyond tidiness

It invalidates a premise of several other issues. pip install -e ".[dev]" produces an environment where pytest cannot start #110 reports that pip install -e ".[dev]" cannot start pytest because addopts requires xdist. That is true only on branches where pytest.ini is absent — the epic branch deleted it. On master, addopts never applies. Any statement about "the addopts" needs to say which config was live.
Marker-based safety is illusory. tests/integration/ is unmarked, so the documented unit-test command provisions billable AWS EKS clusters #109's gate deliberately does not rely on markers, which turns out to have been necessary: applying pytest.mark.expensive today produces PytestUnknownMarkWarning, not a usable selector.
--strict-markers being inert hides typos. scripts/test_discovery.py (on the closed epic branch) found 1,532 marker-hygiene issues that a live --strict-markers would have caught at source.

Fix — needs care, do not just flip it

The obvious change ([tool:pytest] → [pytest], or delete pytest.ini so pyproject wins) will activate --strict-markers, testpaths, and possibly -n 4. Any test using an unregistered mark then becomes a hard error, and there are 1,532 known marker issues. Sequence it:

Decide on ONE config source. Recommendation: delete pytest.ini, keep pyproject.toml (it has the fuller marker list — 13 vs 10 — and is the modern convention).
Before activating, inventory every mark actually used: grep -rhoE "@pytest\.mark\.[a-z_]+" tests/ | sort -u and reconcile against the registered list.
Confirm pytest-xdist is installed wherever addopts will now apply (ties to pip install -e ".[dev]" produces an environment where pytest cannot start #110).
Verify testpaths activation does not pull tests/real_world/ into default runs.
Land --strict-markers last, once the marker inventory is clean.
Add a test asserting Config.inifile is the expected file, so a future stray config file cannot silently shadow it again.

Verification

pytest --co -q tests/unit/ 2>&1 | grep -i configfile
pytest --markers | grep -cE "^@pytest.mark."

Issue #130: Add handoff plan for the pytest config-shadowing fix

Handoff plan written — `notes/handoff_130_pytest_config.md` (commit `de09f1a`)

A fresh session can pick this up cold. Everything in it was verified on master at a9393b7; every fact carries the command that established it.

⚠️ Landmine discovered while writing the plan

Naively fixing this breaks bare pytest outright. Verified:

$ mv pytest.ini /tmp && pytest --co -q
ERROR: Refusing to run 'tests/integration': tests/integration provisions real,
billable cloud resources (AWS EKS/EC2). Set CLUSTRIX_ALLOW_BILLABLE=1 ...

Cause: master's pyproject sets testpaths = ["tests/unit", "tests/integration"], and pytest populates config.args from testpaths when no paths are given on the command line. The #109 guard inspects config.args, so the moment pyproject becomes live, every bare run aborts.

This is a trap I created in #109 and it must be defused first. The clean fix — verified — is to key the guard off config.invocation_params.args (what the user actually typed) instead:

invocation	`config.args`	`config.invocation_params.args`
`pytest` (testpaths live)	`['tests/unit', 'tests/integration']`	(no path entries)
`pytest tests/unit/`	`['tests/unit/']`	`['tests/unit/', ...flags]`

Corrections to this issue's original body

pyproject declares 4 markers on master (real_world, slow, unit, integration), not 13. The longer list I cited was from the closed epic branch. pytest.ini declares 14.
master's pyproject addopts is just -v --tb=short — no -n 4. So activating pyproject does not require xdist here. The xdist problem is specific to the epic branch, which deletes pytest.ini. pip install -e ".[dev]" produces an environment where pytest cannot start #110 needs the same correction.

Measured marker gap (the thing that gates `--strict-markers`)

marker	uses	files	declared in pyproject?
`real_world`	224	—	yes
`dartmouth_network`	11	4	no
`slow`	6	—	yes
`expensive`	5	5	no
`performance`	4	4	no
`integration`	2	—	yes
`cleanup`	1	1	declared nowhere — `test_kubernetes_performance_benchmarks.py:1038`, likely a typo

Plan shape

6 steps, one commit each, verification commands after every one:

Baseline the numbers
Defuse the landmine (guard → invocation_params.args) — must be first
Delete pytest.ini, set testpaths = ["tests"]
Declare the 3 missing markers; decide cleanup
Enable --strict-markers last
Add a regression test so a stray config can never shadow again
Full scripts/pre_push_check.py, then correct pip install -e ".[dev]" produces an environment where pytest cannot start #110, MIGRATION.md:80, .claude/commands/testing/prime.md:95

The regression test uses pytestconfig.inipath and getini("markers"); both were verified against pytest 8.4.2 in the broken and fixed states, so the snippet is known-working rather than assumed. It fails today and passes after Step 2 — write it first and watch it fail.

Fixed in #134 (branch fix/130-pytest-config). Before/after evidence below; the issue closes when that PR merges.

Before — `master` @ `a9393b7`

$ head -1 pytest.ini
[tool:pytest]

$ pytest tests/unit/ --co 2>&1 | grep configfile
configfile: pytest.ini

$ pytest --markers | grep -c '^@pytest.mark.real_world'
0

$ printf 'import pytest\n@pytest.mark.bogus_xyz\ndef test_x(): assert True\n' > tests/unit/test_probe_tmp.py
$ pytest tests/unit/test_probe_tmp.py --co -q
1 test collected            # --strict-markers inert

$ pytest tests/ -m "not real_world" --co -q -o addopts= | tail -1
1214/1592 tests collected (378 deselected), 6 errors

After

$ ls pytest.ini tox.ini setup.cfg 2>/dev/null | wc -l
0

$ pytest --co 2>&1 | grep configfile
configfile: pyproject.toml

$ pytest --markers | grep -cE '^@pytest.mark.(real_world|slow|unit|integration|expensive|dartmouth_network|performance):'
7

$ pytest tests/unit/test_probe_tmp.py --co -q
ERROR ... Failed: 'bogus_xyz' not found in `markers` configuration option

$ pytest --co -q | tail -1
1670 tests collected        # bare pytest, 0 errors, no abort

$ pytest tests/unit/ -q | tail -1
75 passed

Checklist from this issue

Decide on ONE config source — deleted pytest.ini, kept pyproject.toml
Inventory every mark in use before activating — 7 in use, 4 were declared; added expensive, dartmouth_network, performance; removed the single dead cleanup
Confirm pytest-xdist is installed wherever addopts applies — not needed: master's addopts has no -n. A comment above addopts now records that adding one requires moving xdist from [test] into [dev] first (pip install -e ".[dev]" produces an environment where pytest cannot start #110)
Verify testpaths activation does not pull tests/real_world/ into default runs — it does collect them (testpaths = ["tests"]), which is correct and matches the previous no-config behaviour. Nothing runs them by default: CI uses pytest tests/unit/ -m "not real_world", and scripts/pre_push_check.py now does the same, which it previously did not
Land --strict-markers last — Step 4, after the marker inventory was clean
Add a test asserting the config file is the expected one — tests/unit/test_pytest_config.py, four assertions, all failing on master

Two corrections to this issue's own text

"13 markers" / "1,532 marker issues." pytest.ini declared 10 markers and pyproject.toml 4. The suite uses 7 distinct non-builtin markers across 253 applications. The 1,532 figure came from scripts/test_discovery.py on the closed epic branch and does not correspond to anything measurable on master.

The landmine was not in the issue. Deleting pytest.ini alone breaks pytest entirely, because testpaths then feeds config.args and the #109 guard refuses the run. That had to be fixed first, in its own commit. Details in #134 and in notes/handoff_130_pytest_config.md.

Found along the way

The Step 0 baseline had 6 collection errors, not zero: an f-string backslash SyntaxError (fatal on Python <3.12 while requires-python is >=3.8) plus undeclared numpy/pandas. Fixed here, since Step 4's gate is "0 errors".
scripts/pre_push_check.py cannot exit 0 for unrelated reasons, and never could — filed as pre_push_check.py can never pass: 92 flake8 findings, incl. a real f-string bug in test_direct_gpu_detection.py #133, which also documents a real f-string bug in tests/integration/test_direct_gpu_detection.py.
pip install -e ".[dev]" produces an environment where pytest cannot start #110's addopts premise and its sklearn claim are both wrong; corrected there.

Issue #130: Close the guard bypasses red-teaming found, and fix my ow…

Issue #130: Revert the fast_ci.yml change; the diagnosis was wrong

Issue #130: pytest.ini was dead config shadowing pyproject.toml (#134)

Update notes for handoff to a fresh session (#136)

Navigation Menu

Repository navigation

pytest.ini is dead config that silently shadows pyproject.toml (--strict-markers, testpaths, markers all inert) #130

Problem

Everything currently inert

Why this matters beyond tidiness

Fix — needs care, do not just flip it

Verification

Handoff plan written — `notes/handoff_130_pytest_config.md` (commit `de09f1a`)

⚠️ Landmine discovered while writing the plan

Corrections to this issue's original body

Measured marker gap (the thing that gates `--strict-markers`)

Plan shape

Before — `master` @ `a9393b7`

After

Checklist from this issue

Two corrections to this issue's own text

Found along the way

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pytest.ini is dead config that silently shadows pyproject.toml (--strict-markers, testpaths, markers all inert) #130

Description

Problem

Everything currently inert

Why this matters beyond tidiness

Fix — needs care, do not just flip it

Verification

Activity

jeremymanning commented on Aug 17, 2026

Handoff plan written — notes/handoff_130_pytest_config.md (commit de09f1a)

⚠️ Landmine discovered while writing the plan

Corrections to this issue's original body

Measured marker gap (the thing that gates --strict-markers)

Plan shape

jeremymanning commented on Aug 17, 2026

Before — master @ a9393b7

After

Checklist from this issue

Two corrections to this issue's own text

Found along the way

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Handoff plan written — `notes/handoff_130_pytest_config.md` (commit `de09f1a`)

Measured marker gap (the thing that gates `--strict-markers`)

Before — `master` @ `a9393b7`