diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 38305897..065e36db 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -3922,13 +3922,21 @@ record and the cell's current score live in the vault scorecard and are not rest ## 1006. A mutation that matches is not a mutation that bites: the absence-claim gate proves syntax, never behaviour -> ๐Ÿ”ข **Filed 2026-08-04 โ€” not started. Scored 2026-08-04 โ†’ P2.** Value **6/10** ยท Difficulty -> **3/10** ยท _quick win_. `check_absences` admits an ASVS absence claim on `re.search(a.pattern, -> a.mutation)` (`scripts/asvs/scorecard.py:395`) โ€” one string field of a TOML row matched against -> another โ€” so a well-formed reintroduction that would change nothing if applied passes all three -> of the gate's failure modes and certifies a non-control into the compliance record; the -> remainder is a required per-claim observable plus a mode that applies the mutation and requires -> that observable to go red, in one stdlib script and its fixture tests. +> โœ… **SHIPPED 2026-08-06 โ€” a new opt-in mode can prove an absence claim BITES, which the pattern +> check structurally cannot.** Value **6/10** ยท Difficulty **3/10** ยท _quick win_. +> `scripts/asvs/scorecard.py` gains a `--prove-absences` mode: per claim it applies the `mutation` to +> a scratch copy of the tree and requires a named `observable` (a pytest node id) to go RED, failing +> closed on any exit code that is not an honest test failure (an already-red baseline, an +> uncollectable node, or a mutation that only breaks import is a PROVE-ERROR, never a proof). So a +> well-formed reintroduction that would change nothing if applied CAN be caught the moment its claim +> carries an `observable` โ€” but the default `verify` path is byte-unchanged and no authored claim +> carries one yet, so nothing new is blocked by this alone today. Two optional `Absence` fields +> (`mutation_path`, `observable`) feed it, a coarse same-file static backstop screens claims that +> carry no observable, the scratch copy refuses secrets / the store / `docs/security` (defence for the +> eventual vault run), and fixture negative controls plus a CLI exit-code test prove the mode itself +> can go red. Public repo script + fixtures only; wiring the mode over the vault's ~81 existing +> absence claims (untouched) and backfilling their observables is the owner's follow-up +> (`scorecard.py:14-16`, ADR 0156 ยง7). **Cluster:** Security & Compliance. **Priority:** P2. **Verdict:** build. **Severity:** medium โ€” the defect is in the instrument, not the engine, and a green instrument that cannot go red is the diff --git a/scripts/asvs/scorecard.py b/scripts/asvs/scorecard.py index 33bc1d52..2526fa01 100644 --- a/scripts/asvs/scorecard.py +++ b/scripts/asvs/scorecard.py @@ -19,8 +19,12 @@ from __future__ import annotations import argparse +import ast import re +import shutil +import subprocess import sys +import tempfile import tomllib from collections import Counter from dataclasses import dataclass, field @@ -102,11 +106,43 @@ class Absence: Do NOT derive ``mutation`` from ``pattern``. A value generated from the thing it validates satisfies the check by construction, which would make this the most authoritative-looking vacuous gate in the file โ€” the same defect class it exists to close, arriving through the fix. + + ``mutation_path`` and ``observable`` feed the ``--prove-absences`` mode (:func:`prove_absences`), + which closes a mode the pattern check cannot see: a ``mutation`` that matches its pattern, whose + control speaks and whose corpus is quiet, yet **changes nothing observable when applied** โ€” a + reintroduction raised into a swallowing handler, a field nobody reads, a flag nobody branches on. + ``re.search(pattern, mutation)`` proves the mutation is *well-formed*; it never proves it *bites*. + + - ``mutation_path`` โ€” the file the reintroduction lands in, relative to ``root``. + - ``observable`` โ€” the named artifact that must go red when the mutation is applied: a + ``tests/test_x.py::test_y`` pytest node id. When both fields are set the mode PROVES the claim + by execution โ€” it runs the observable on a scratch copy of the tree (baseline must be green), + applies the mutation, and requires the observable to FAIL (and to fail as a test failure, not a + collection/usage error, which fails closed). When only ``mutation_path`` is set the mode falls + back to a coarse static backstop. + + Both fields default empty, and their absence means **"not yet proven by execution"** โ€” never + "proven vacuous". They are opt-in per claim because a live proof spawns a pytest subprocess per + claim; a claim with neither field is reported as SKIPPED, not failed. + + Honest limits of the proving mode, stated so they are not overclaimed: + + - Application is **append-based**: the mutation text is appended to the scratch target, so it + breaks a fixture by *redefinition shadowing*. That faithfully reddens a well-formed + reintroduction in a fixture; it does not reproduce every in-function reintroduction a real claim + might describe. + - The static backstop is a **coarse same-file heuristic** โ€” a ``raise`` in the mutation landing + lexically in a ``try`` whose every handler swallows (bare/``Exception``, log-only body). It + proves **nothing** in general: it cannot see a swallow in a *caller* rather than at the landing + site, so it would miss the very cross-file instance that motivated this item. It is a screen, + not a proof, and must not be written up as one. """ pattern: str positive_control: str mutation: str + mutation_path: str = "" + observable: str = "" @dataclass(frozen=True) @@ -143,6 +179,13 @@ class Findings: checked_anchors: int = 0 checked_absences: int = 0 skipped_anchors: int = 0 + #: Populated only by :func:`prove_absences`. ``proved_absences`` counts claims whose observable + #: went red under the applied mutation (a live proof); ``static_screened`` counts claims that took + #: the static backstop (a screen, not a proof); ``skipped_absences`` counts claims carrying no + #: ``mutation_path`` (nothing to apply). UNPROVEN and PROVE-ERROR outcomes go into ``problems``. + proved_absences: int = 0 + static_screened: int = 0 + skipped_absences: int = 0 @property def ok(self) -> bool: @@ -271,6 +314,12 @@ def load_scorecard(path: Path) -> list[Cell]: # No default. A missing mutation must be authored, not inferred โ€” see the # Absence docstring on why deriving one from the pattern is worse than none. mutation=str(a["mutation"]), + # Optional, and deliberately NOT refused at load. A hard requirement here would + # void every already-authored absence claim (none carry these yet), and their + # re-authoring is out of this script's reach (ADR 0156 ยง7). Absent means "not + # yet proven by execution", surfaced by --prove-absences, not "proven vacuous". + mutation_path=str(a.get("mutation_path", "")), + observable=str(a.get("observable", "")), ) for a in raw.get("absence", []) ), @@ -431,6 +480,244 @@ def _grep_count(pattern: str, files: list[Path]) -> int: return sum(1 for f in files if rx.search(f.read_text(encoding="utf-8", errors="replace"))) +# --- proving an absence by mutation (--prove-absences) -------------------------------------------- +# +# check_absences proves a mutation is well-formed (its pattern fires on it). It cannot prove the +# mutation BITES: applied, does anything observable go red? A reintroduction raised into a swallowing +# handler passes every check in check_absences and changes nothing. This mode closes that hole by +# EXECUTING the claim โ€” mutate a scratch copy of the tree, run the named observable, require it to go +# red โ€” and it fails closed on every code that is not an honest test failure, so a typo'd node or an +# already-red observable can never masquerade as "the control bit". The whole pass runs inside a +# TemporaryDirectory scratch copy, so it never mutates `root` and never trips the committed-tree scan +# on itself. + + +def _scratch_ignore(dirpath: str, names: list[str]) -> set[str]: + """Names to skip when copying `root` into the scratch tree. Beyond the usual VCS/venv/cache noise + this refuses secrets and posture data โ€” ``.env*``, ``*.db`` and its WAL sidecars (the local + store), and the vault's ``docs/security`` tree (ADR 0156 ยง7). The vault runs this module against + the REAL tree, so a scratch copy carrying those would spill them into a world-default temp dir, + which CLAUDE.md ยง9 forbids this module reading at all. Public-repo runs never see them (no + committed ``.env``/``*.db``, ``docs/security`` absent), so this is defence for the vault run.""" + ignored = set( + shutil.ignore_patterns( + ".git", + ".venv", + "__pycache__", + "node_modules", + ".env", + ".env.*", + "*.db", + "*.db-wal", + "*.db-shm", + )(dirpath, names) + ) + # `docs/security` is path-specific, not a basename glob: skip a `security` entry only directly + # under `docs`, leaving any unrelated `security` elsewhere in the tree copied. + if Path(dirpath).name == "docs" and "security" in names: + ignored.add("security") + return ignored + + +def _copy_scratch(root: Path, dest: Path) -> Path: + """Copy `root` into `dest`, skipping VCS/venv/cache dirs and โ€” defensively, for the vault run โ€” + secrets, the local store, and vault posture data (:func:`_scratch_ignore`). Never writes to + `root`.""" + shutil.copytree(root, dest, ignore=_scratch_ignore) + return dest + + +def _is_within_tree(rel: str) -> bool: + """True only for a repo-relative path with no anchor and no ``..`` component โ€” one that cannot + escape the scratch copy when joined onto it. ``mutation_path`` comes from the authored scorecard, + so it is untrusted for this purpose: an absolute or ``..``-bearing value is refused, not resolved.""" + p = Path(rel) + if p.is_absolute() or p.anchor: + return False + return ".." not in p.parts + + +def _apply_mutation(scratch: Path, mutation_path: str, mutation: str) -> None: + """Append the reintroduction to the scratch target โ€” redefinition shadowing is what makes a + well-formed reintroduction actually break an observable. Never called against `root`.""" + target = scratch / mutation_path + with target.open("a", encoding="utf-8") as fh: + fh.write("\n" + mutation + "\n") + + +def _run_node(scratch: Path, node_id: str, python: str, timeout: float) -> int: + """Run one pytest node inside the scratch copy and return its exit code. + + Invoked with ``--rootdir `` and ``cwd=scratch`` and ``-o addopts=`` so no repo + ``conftest``/``pyproject``/addopts leaks into the child run, and ``-p no:cacheprovider`` so it + writes nothing back. A timeout is treated as a non-{0,1} code โ€” fail closed, never a proof. + """ + try: + proc = subprocess.run( # nosec B603 B607 - fixed argv, no shell; python is sys.executable, node id is scorecard-authored not shell-interpreted + [ + python, + "-m", + "pytest", + "-q", + "--rootdir", + str(scratch), + "-p", + "no:cacheprovider", + "-o", + "addopts=", + node_id, + ], + cwd=scratch, + capture_output=True, + text=True, + check=False, + timeout=timeout, + ) + except subprocess.TimeoutExpired: + return 124 # non-zero and non-1: fails closed as a PROVE-ERROR, never counted as a proof + return proc.returncode + + +def _handler_swallows(handler: ast.ExceptHandler) -> bool: + """A handler that catches broadly (bare or ``Exception``/``BaseException``) with a log-only/``pass`` + body and no re-raise โ€” the shape that eats a reintroduced exception.""" + caught = handler.type + if not ( + caught is None + or (isinstance(caught, ast.Name) and caught.id in {"Exception", "BaseException"}) + ): + return False + if any(isinstance(n, ast.Raise) for stmt in handler.body for n in ast.walk(stmt)): + return False # a re-raise is not a swallow + return all(isinstance(stmt, (ast.Pass, ast.Expr)) for stmt in handler.body) + + +def _landing_swallows(source: str) -> bool: + """True if `source` contains a ``try`` whose EVERY handler swallows (see :func:`_handler_swallows`). + + A coarse same-file heuristic โ€” it proves nothing in general and cannot see a swallow in a caller. + """ + try: + tree = ast.parse(source) + except SyntaxError: + return False + return any( + isinstance(node, ast.Try) + and bool(node.handlers) + and all(_handler_swallows(h) for h in node.handlers) + for node in ast.walk(tree) + ) + + +def _prove_one( + a: Absence, + cell_id: str, + root: Path, + scratch_dir: Path, + findings: Findings, + *, + python: str, + timeout: float, +) -> None: + if not a.mutation_path: + # Nothing to apply. Reported, not failed: opt-in per claim (a live proof spawns a subprocess). + findings.skipped_absences += 1 + return + if not _is_within_tree(a.mutation_path): + # mutation_path is authored data. An absolute path or a `..` escape would let _apply_mutation + # write outside the scratch copy (and the is_file probe below read outside `root`), defeating + # the 'never touches root' guarantee. Refuse it rather than resolve it. + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR โ€” mutation_path {a.mutation_path!r} is not a " + "repo-relative path inside the tree (it is absolute or contains '..'), so applying the " + "mutation could escape the scratch copy" + ) + return + target_in_root = root / a.mutation_path + if not target_in_root.is_file(): + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR โ€” mutation_path {a.mutation_path!r} is not a file " + "in the tree, so the mutation cannot be applied" + ) + return + + if a.observable: + scratch = _copy_scratch(root, scratch_dir) + baseline = _run_node(scratch, a.observable, python, timeout) + if baseline != 0: + # An already-red or uncollectable observable cannot attribute its red to the mutation. + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR โ€” observable {a.observable!r} is not green on " + f"the pristine tree (pytest exit {baseline}); a red or uncollectable baseline cannot " + "be attributed to the mutation" + ) + return + _apply_mutation(scratch, a.mutation_path, a.mutation) + mutated = _run_node(scratch, a.observable, python, timeout) + if mutated == 1: + findings.proved_absences += 1 # live proof: the control bit + elif mutated == 0: + findings.problems.append( + f"{cell_id}: absence claim UNPROVEN โ€” applying the mutation to {a.mutation_path} left " + f"observable {a.observable!r} green (pytest exit 0); the mutation changes nothing the " + "control catches, so this claim is syntax without behaviour" + ) + else: + # exit 2/3/4/5/...: collection or usage error. NEVER a proof โ€” a typo'd node or a mutation + # that merely breaks import must not masquerade as the control biting. + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR โ€” mutated run of {a.observable!r} errored " + f"(pytest exit {mutated}) rather than failing; a collection or usage error must not " + "count as the control biting" + ) + return + + # Static backstop: mutation_path but no observable. A screen, not a proof (see Absence docstring). + findings.static_screened += 1 + if re.search(r"\braise\b", a.mutation) and _landing_swallows( + target_in_root.read_text(encoding="utf-8", errors="replace") + ): + findings.problems.append( + f"{cell_id}: absence claim SUSPECT (static heuristic) โ€” its reintroduction raises into " + f"{a.mutation_path}, which has a try/except that swallows (bare or Exception, log-only " + "body), so a live raise there may be caught and prove nothing. Supply an `observable` to " + "prove it by execution" + ) + + +def prove_absences( + cells: list[Cell], + root: Path, + *, + python: str = sys.executable, + timeout: float = 120.0, +) -> Findings: + """Prove each absence claim BITES: apply its mutation to a scratch copy and require its observable + to go red. Fails closed on anything that is not an honest baseline-green / mutated-fail pair. + + This is separate from :func:`verify` and opt-in (``--prove-absences``) because it spawns a pytest + subprocess per provable claim. It never touches `root`. + """ + findings = Findings() + resolved_root = root.resolve() + with tempfile.TemporaryDirectory(prefix="asvs_prove_") as td_base: + base = Path(td_base) + i = 0 + for c in cells: + for a in c.absence: + i += 1 + _prove_one( + a, + c.id, + resolved_root, + base / f"scratch_{i}", + findings, + python=python, + timeout=timeout, + ) + return findings + + def _sort_key(cell_id: str) -> tuple[int, ...]: try: return tuple(int(p) for p in cell_id.split(".")) @@ -596,20 +883,59 @@ def render_current(cells: list[Cell], *, anchor_sha: str) -> str: return chr(10).join(lines) + chr(10) +def _run_prove_absences(scorecard: Path, root: Path) -> int: + """The ``--prove-absences`` entry point: execute-prove every absence claim (see + :func:`prove_absences`). Needs no corpus โ€” it applies mutations, it does not grep for patterns.""" + try: + cells = load_scorecard(scorecard) + except ScorecardError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 # could not measure โ€” never 0, never confused with "clean" + findings = prove_absences(cells, root) + print( + f"prove-absences: proved {findings.proved_absences} by mutation; " + f"{findings.static_screened} static-screened; {findings.skipped_absences} skipped; " + f"{len(findings.problems)} problem(s)" + ) + for p in findings.problems: + print(f" FAIL {p}", file=sys.stderr) + return 0 if findings.ok else 1 + + def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description="Verify or render the ASVS scorecard (ADR 0156).") ap.add_argument("--scorecard", type=Path, required=True) - ap.add_argument("--corpus", type=Path, required=True) + # Not required: --prove-absences applies mutations and never greps for patterns, so it needs no + # corpus. Verify mode still does; that is enforced after parsing, not by argparse. + ap.add_argument("--corpus", type=Path, required=False) ap.add_argument( "--root", type=Path, default=Path.cwd(), help="tree the evidence anchors point into" ) ap.add_argument("--render", type=Path, help="write the generated CURRENT.md here") + # A separate, opt-in mode: prove each absence claim BITES by applying its mutation to a scratch + # copy and requiring its observable to go red. Opt-in because it spawns a pytest subprocess per + # provable claim; kept out of the default verify path, which stays purely static. + ap.add_argument( + "--prove-absences", + action="store_true", + help="execute-prove absence claims (apply mutation to a scratch tree, require observable red)", + ) # NO --anchor-sha injected by CI. The anchor is the commit the EVIDENCE was read on โ€” a property # of the assessment, recorded in [scorecard].anchor_commit. Passing ${{ github.sha }} made the # rendered file differ on every run, so the drift check could never pass: a gate that cannot go # green is as useless as one that cannot go red, and this one shipped that way. args = ap.parse_args(argv) + if args.prove_absences: + return _run_prove_absences(args.scorecard, args.root) + + if args.corpus is None: + print( + "error: --corpus is required to verify the scorecard (only --prove-absences may omit it)", + file=sys.stderr, + ) + return 2 + try: findings = verify(args.scorecard, args.corpus, args.root) except ScorecardError as exc: diff --git a/tests/test_asvs_scorecard.py b/tests/test_asvs_scorecard.py index 0c3b15fb..58fd0bc3 100644 --- a/tests/test_asvs_scorecard.py +++ b/tests/test_asvs_scorecard.py @@ -23,6 +23,7 @@ Cell, Findings, ScorecardError, + _copy_scratch, check_absences, check_anchors, check_completeness, @@ -31,6 +32,8 @@ count, load_corpus, load_scorecard, + main, + prove_absences, render_current, verify, ) @@ -675,3 +678,383 @@ def test_completeness_accepts_a_decided_cell_evidenced_only_by_an_absence_claim( Cell(id="2.1.1", level=3, verdict="unverified"), ] assert not [p for p in check_completeness(cells, CORPUS) if "carry NO anchor" in p] + + +# --- --prove-absences: a mutation that matches is not a mutation that BITES (#1006) --------------- +# +# check_absences proves a mutation's pattern fires on it; it never applies the mutation. So a +# well-formed reintroduction that would change nothing observable passes every check. prove_absences +# closes that hole by EXECUTING the claim: mutate a scratch copy, run the named observable, require it +# to go red -- and fail closed on any exit code that is not an honest test failure. Every fixture tree +# lives in tmp_path (never in the scanned packages), and the mutation is applied only to a scratch +# copy in a system TemporaryDirectory, so nothing here touches the committed corpus or tmp_path. + +_SCANNER = "def scan(p):\n return 'clean'\n" +_OBS_TEST = "from scanner import scan\n\n\ndef test_clean():\n assert scan('x') == 'clean'\n" + + +def _module(tmp_path: Path, name: str, body: str) -> Path: + """Write a code module fixture into tmp_path (the code the reintroduction lands in).""" + p = tmp_path / name + p.write_text(body, encoding="utf-8") + return p + + +def _obs_test(tmp_path: Path, name: str, body: str) -> Path: + """Write an observable pytest module fixture into tmp_path.""" + p = tmp_path / name + p.write_text(body, encoding="utf-8") + return p + + +def _live_claim(mutation: str, mutation_path: str, observable: str) -> Cell: + # pattern/positive_control are irrelevant to prove_absences (they drive check_absences); supply + # harmless values so the required fields are present. + return Cell( + id="1.1.1", + level=1, + verdict="fail", + absence=( + Absence( + pattern="x", + positive_control="y", + mutation=mutation, + mutation_path=mutation_path, + observable=observable, + ), + ), + ) + + +def test_prove_absences_proves_a_claim_when_the_mutation_reddens_its_observable( + tmp_path: Path, +) -> None: + """The positive half: a mutation that shadows `scan` reddens the observable, so the claim BITES. + + Falsified by making `_apply_mutation` a no-op: the observable stays green, the mode reports + UNPROVEN, and the `.ok`/`proved_absences == 1` assertions go RED. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + claim = _live_claim( + 'def scan(p): return "infected"', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert findings.ok, findings.problems + assert findings.proved_absences == 1 + + +def test_prove_absences_fails_when_the_mutation_reddens_nothing(tmp_path: Path) -> None: + """The negative control the brief requires: a mutation to a file the observable never imports + reddens nothing, so the claim is UNPROVEN and the mode FAILS. + + Falsified by making the mode accept a mutated exit==0 as a pass (dropping the exit==1 + requirement): the non-biting claim then reports ok, and `not findings.ok` goes RED -- proving the + mode can actually fail. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _module( + tmp_path, "unrelated.py", "VALUE = 1\n" + ) # exists, but the observable does not import it + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + claim = _live_claim( + 'def scan(p): return "infected"', "unrelated.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("UNPROVEN" in p for p in findings.problems), findings.problems + assert findings.proved_absences == 0 + + +def test_prove_absences_fails_closed_when_the_observable_is_already_red(tmp_path: Path) -> None: + """An observable that fails on the pristine tree cannot attribute its red to the mutation. + + Falsified by removing the baseline-green check: the already-red observable stays red under the + mutation, is miscounted as `proved`, and this test's `not findings.ok` goes RED. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test( + tmp_path, + "test_scanner.py", + "from scanner import scan\n\n\ndef test_clean():\n assert scan('x') == 'DIFFERENT'\n", + ) + claim = _live_claim( + 'def scan(p): return "infected"', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("PROVE-ERROR" in p and "baseline" in p for p in findings.problems), findings.problems + assert findings.proved_absences == 0 + + +def test_prove_absences_fails_closed_when_the_mutation_errors_instead_of_failing( + tmp_path: Path, +) -> None: + """A mutation that breaks IMPORT of the observable's module errors at collection (pytest exit 4), + not a test failure (exit 1). A collection/usage error must NEVER count as the control biting -- + otherwise a typo'd node or an import-breaking mutation rebuilds the exact vacuity being fixed. + + Falsified by changing the mutated-run requirement from `exit == 1` to `exit != 0`: the exit-4 + collection error then masquerades as `proved`, and this test's `not findings.ok` goes RED. + Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + # Appended at module level, this raises when `scanner` is imported -> collection error, not a + # failing assertion. + claim = _live_claim( + 'raise RuntimeError("reintroduced")', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("PROVE-ERROR" in p and "errored" in p for p in findings.problems), findings.problems + assert findings.proved_absences == 0 + + +def test_prove_absences_static_backstop_flags_a_raise_into_a_swallowing_file( + tmp_path: Path, +) -> None: + """With no observable, the static backstop flags a `raise` landing in a file whose try/except + swallows (bare/Exception, log-only body). A screen, not a proof -- but it fails the mode. + + Falsified by forcing `_landing_swallows` to return False: the swallow is not flagged, `findings.ok` + becomes True, and this test's `not findings.ok` goes RED. Restored. + """ + _module( + tmp_path, + "caller.py", + "import logging\n\nlog = logging.getLogger(__name__)\n\n\n" + "def reconcile():\n try:\n work()\n except Exception:\n" + " log.exception('reconcile failed')\n", + ) + claim = Cell( + id="13.3.4", + level=3, + verdict="fail", + absence=( + Absence( + pattern="x", + positive_control="y", + mutation='raise RuntimeError("reintroduced")', + mutation_path="caller.py", + observable="", # no observable -> static backstop + ), + ), + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("SUSPECT" in p and "swallow" in p for p in findings.problems), findings.problems + assert findings.static_screened == 1 + + +def test_prove_absences_static_backstop_passes_a_non_swallowing_file(tmp_path: Path) -> None: + """REACH control: the static backstop must NOT flag a `raise` into a handler that re-raises -- + proving the heuristic reads the handler body, not merely the presence of a try/except. + + Falsified by forcing `_landing_swallows` to return True: the re-raising file is flagged, and this + test's `assert findings.ok` goes RED. Restored. + """ + _module( + tmp_path, + "plain.py", + "def reconcile():\n try:\n work()\n except Exception:\n raise\n", + ) + claim = Cell( + id="13.3.4", + level=3, + verdict="fail", + absence=( + Absence( + pattern="x", + positive_control="y", + mutation='raise RuntimeError("reintroduced")', + mutation_path="plain.py", + observable="", + ), + ), + ) + findings = prove_absences([claim], tmp_path) + assert findings.ok, findings.problems + assert findings.static_screened == 1 + assert not any("SUSPECT" in p for p in findings.problems) + + +def test_prove_absences_leaves_the_root_tree_untouched(tmp_path: Path) -> None: + """The mode must run OUT of the tracked tree: mutation lands only on the scratch copy. + + Falsified by pointing `_apply_mutation` at `root` instead of the scratch copy: root's scanner.py + changes, `after == before` goes RED (and the claim also drops to UNPROVEN). Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + before = {p: p.read_bytes() for p in sorted(tmp_path.rglob("*")) if p.is_file()} + claim = _live_claim( + 'def scan(p): return "infected"', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert findings.proved_absences == 1, findings.problems + after = {p: p.read_bytes() for p in sorted(tmp_path.rglob("*")) if p.is_file()} + assert after == before + + +def test_load_reads_optional_mutation_path_and_observable_and_omitting_them_still_loads( + tmp_path: Path, +) -> None: + """Round-trip: the two new fields load when present, and OMITTING them still loads (vault-safety -- + the ~81 existing absence claims carry neither and must stay loadable). + + Falsified by dropping the `.get` wiring in load_scorecard (hardcoding ``mutation_path=""``): half + (a) then reads "" and its assertion goes RED, while half (b) stays green -- proving the round-trip + is actually asserted. Restored. + """ + with_fields = tmp_path / "with.toml" + with_fields.write_text( + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "clamd"\n' + ' positive_control = "ScanRejected"\n' + ' mutation = "import clamd"\n' + ' mutation_path = "messagefoundry/scan.py"\n' + ' observable = "tests/test_scan.py::test_rejects"\n', + encoding="utf-8", + ) + a = load_scorecard(with_fields)[0].absence[0] + assert a.mutation_path == "messagefoundry/scan.py" + assert a.observable == "tests/test_scan.py::test_rejects" + + without_fields = tmp_path / "without.toml" + without_fields.write_text( + '[[cell]]\nid = "1.1.2"\nlevel = 2\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "clamd"\n' + ' positive_control = "ScanRejected"\n' + ' mutation = "import clamd"\n', + encoding="utf-8", + ) + b = load_scorecard(without_fields)[0].absence[0] + assert b.mutation_path == "" and b.observable == "" + + +def test_prove_absences_refuses_a_mutation_path_that_escapes_the_scratch_tree( + tmp_path: Path, +) -> None: + """`mutation_path` is authored data. An absolute path or a `..` escape would let the mutation land + OUTSIDE the scratch copy (defeating the 'never touches root' guarantee), so the mode refuses it as + a PROVE-ERROR before applying anything -- it never counts as a proof. + + Falsified by making `_is_within_tree` return True unconditionally: the `..` path is no longer + refused, the run falls through to the is_file probe with a different message, and this test's + `any("repo-relative" in p ...)` assertion goes RED. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + claim = _live_claim( + 'def scan(p): return "infected"', + "../escape.py", # a `..` that would climb out of the scratch copy + "test_scanner.py::test_clean", + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("PROVE-ERROR" in p and "repo-relative" in p for p in findings.problems), ( + findings.problems + ) + assert findings.proved_absences == 0 + + +def test_copy_scratch_excludes_secrets_store_and_vault_posture(tmp_path: Path) -> None: + """The scratch copy the vault mutation-run reads must never carry secrets, the local store, or the + vault posture tree -- CLAUDE.md ยง9 forbids this module reading them at all. `_copy_scratch` skips + `.env*`, `*.db`(+WAL sidecars), and `docs/security`, while ordinary sources are still copied. + + Falsified by reverting `_scratch_ignore` to the bare VCS/venv/cache patterns: the `.env`, `*.db` + and `docs/security` fixtures are then copied into the scratch dir and every `not (dest/...).exists()` + assertion goes RED, while the `keep.py` assertion stays green. Restored. + """ + (tmp_path / ".env").write_text("EXAMPLE_PLACEHOLDER=not-a-secret\n", encoding="utf-8") + (tmp_path / "local.db").write_text("binary-store\n", encoding="utf-8") + (tmp_path / "local.db-wal").write_text("wal\n", encoding="utf-8") + (tmp_path / "docs" / "security").mkdir(parents=True) + (tmp_path / "docs" / "security" / "posture.toml").write_text("real = true\n", encoding="utf-8") + (tmp_path / "docs" / "PUBLIC.md").write_text("# public\n", encoding="utf-8") + (tmp_path / "keep.py").write_text("KEEP = 1\n", encoding="utf-8") + + dest = tmp_path.parent / "scratch_out" + _copy_scratch(tmp_path, dest) + + assert not (dest / ".env").exists() + assert not (dest / "local.db").exists() + assert not (dest / "local.db-wal").exists() + assert not (dest / "docs" / "security").exists() + # ordinary sources and other docs survive the copy + assert (dest / "keep.py").read_text(encoding="utf-8") == "KEEP = 1\n" + assert (dest / "docs" / "PUBLIC.md").exists() + + +def _biting_scorecard(sc: Path, mutation_path: str) -> None: + """Write a one-claim scorecard whose live absence claim points at `mutation_path`.""" + sc.write_text( + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "x"\n' + ' positive_control = "y"\n' + " mutation = 'def scan(p): return \"infected\"'\n" + f' mutation_path = "{mutation_path}"\n' + ' observable = "test_scanner.py::test_clean"\n', + encoding="utf-8", + ) + + +def test_main_prove_absences_returns_0_on_a_biting_claim(tmp_path: Path) -> None: + """The CLI contract CI depends on: `--prove-absences` exits 0 when every claim's mutation reddens + its observable. Exercises `main` -> `_run_prove_absences` end to end, not just `prove_absences`. + + Falsified by changing `_run_prove_absences`'s `return 0 if findings.ok else 1` to `return 1`: this + test's `rc == 0` goes RED while the non-biting test below stays green. Restored. + """ + tree = tmp_path / "tree" + tree.mkdir() + _module(tree, "scanner.py", _SCANNER) + _obs_test(tree, "test_scanner.py", _OBS_TEST) + sc = tmp_path / "sc.toml" + _biting_scorecard(sc, "scanner.py") + rc = main(["--scorecard", str(sc), "--root", str(tree), "--prove-absences"]) + assert rc == 0 + + +def test_main_prove_absences_returns_1_on_a_nonbiting_claim(tmp_path: Path) -> None: + """The other half of the contract: `--prove-absences` exits 1 when a claim is UNPROVEN (its + mutation reddens nothing). Proves the CLI's non-zero failure path, not only the library's. + + Falsified by changing `_run_prove_absences`'s `return 0 if findings.ok else 1` to `return 0`: + this test's `rc == 1` goes RED while the biting test above stays green. Restored. + """ + tree = tmp_path / "tree" + tree.mkdir() + _module(tree, "scanner.py", _SCANNER) + _module(tree, "unrelated.py", "VALUE = 1\n") # present, but the observable never imports it + _obs_test(tree, "test_scanner.py", _OBS_TEST) + sc = tmp_path / "sc.toml" + _biting_scorecard(sc, "unrelated.py") + rc = main(["--scorecard", str(sc), "--root", str(tree), "--prove-absences"]) + assert rc == 1 + + +def test_main_verify_without_corpus_returns_exit_2(tmp_path: Path) -> None: + """Verify mode needs the corpus; omitting `--corpus` (without `--prove-absences`) must exit 2 -- + could-not-measure, never confused with a clean 0. Proves the argparse-independent guard in `main`. + + Falsified by deleting the `if args.corpus is None: ... return 2` branch in `main`: it then falls + through to `verify(...)` with `corpus=None`, raising instead of returning 2, and this test's + `rc == 2` goes RED. Restored. + """ + sc = tmp_path / "sc.toml" + sc.write_text( + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "x"\n' + ' positive_control = "y"\n' + ' mutation = "import x"\n', + encoding="utf-8", + ) + rc = main(["--scorecard", str(sc), "--root", str(tmp_path)]) + assert rc == 2