From aa11acdd0bb271fe20a12d752107efe45fa3e59c Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Thu, 6 Aug 2026 02:03:09 -0500 Subject: [PATCH 1/2] feat(asvs): prove an absence claim bites, not just that its pattern matches (BACKLOG #1006) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `check_absences` admits an ASVS absence claim on `re.search(a.pattern, a.mutation)` -- one TOML field matched against another. That proves the mutation is well-formed; it never proves the mutation BITES. A reintroduction raised into a swallowing handler, written to a field nobody reads, or behind a flag nobody branches on satisfies every failure mode `check_absences` has and changes nothing observable. A green gate that is not evidence. Add an opt-in `--prove-absences` mode (`scripts/asvs/scorecard.py`) that executes the claim rather than grepping it: - Two optional `Absence` fields, `mutation_path` and `observable` (a pytest node id). When both are set the mode copies the tree to a scratch dir, runs the observable (baseline must be green), appends the mutation, and requires the observable to go RED -- and to fail as a test failure (exit 1). It fails closed on every other code: an already-red baseline, an uncollectable node, or a mutation that only breaks import is a PROVE-ERROR, never a proof. A claim that reddens nothing is UNPROVEN and fails the mode. - A coarse same-file static backstop screens claims carrying `mutation_path` but no `observable`: a `raise` landing in a file whose every handler swallows. It is a screen, not a proof (it cannot see a swallow in a caller), documented as such. - The whole pass runs in a TemporaryDirectory scratch copy, so it never mutates the tracked tree and never trips the committed-tree scan on itself. Both fields default empty and load without being refused: the vault's ~81 existing absence claims carry neither and must stay loadable (ADR 0156 §7). Absent means "not yet proven by execution", surfaced by the mode, never "proven vacuous". Review hardening carried in this change (the mode's own helpers): - `_scratch_ignore` refuses `.env*`, `*.db` (+ WAL sidecars) and `docs/security` when copying the tree. The vault runs this module against the REAL tree (ADR 0156 §7); a scratch copy carrying those would spill secrets / the local store / vault posture data into a world-default temp dir, which CLAUDE.md §9 forbids. The public-repo path never sees them; this is defence for the eventual vault run. - `_is_within_tree` refuses a `mutation_path` that is absolute or contains `..` before anything is applied, so an authored path cannot escape the scratch copy. Tests (tests/test_asvs_scorecard.py): eight fixture tests drive `prove_absences` directly (proved / UNPROVEN / already-red baseline / collection-error / two static backstop arms including a re-raise reach control / root-untouched / load round-trip), plus three that drive the CLI contract CI depends on -- `main([..., "--prove-absences"])` exit 0 on a biting fixture and 1 on a non-biting one, and `main([...])` without `--corpus` exit 2 -- plus the secrets-exclusion and path-traversal guards. Every new test was falsified (broken on purpose, watched red, restored). MessageFoundry is a not-deployed beta: the mode is opt-in, the default `verify` path is byte-unchanged, and no authored claim carries an `observable` yet, so nothing new is blocked by this alone today. Wiring the mode over the vault claims and backfilling their observables is the owner's follow-up. --- scripts/asvs/scorecard.py | 328 +++++++++++++++++++++++++++++- tests/test_asvs_scorecard.py | 383 +++++++++++++++++++++++++++++++++++ 2 files changed, 710 insertions(+), 1 deletion(-) diff --git a/scripts/asvs/scorecard.py b/scripts/asvs/scorecard.py index 33bc1d52..2526fa01 100644 --- a/scripts/asvs/scorecard.py +++ b/scripts/asvs/scorecard.py @@ -19,8 +19,12 @@ from __future__ import annotations import argparse +import ast import re +import shutil +import subprocess import sys +import tempfile import tomllib from collections import Counter from dataclasses import dataclass, field @@ -102,11 +106,43 @@ class Absence: Do NOT derive ``mutation`` from ``pattern``. A value generated from the thing it validates satisfies the check by construction, which would make this the most authoritative-looking vacuous gate in the file — the same defect class it exists to close, arriving through the fix. + + ``mutation_path`` and ``observable`` feed the ``--prove-absences`` mode (:func:`prove_absences`), + which closes a mode the pattern check cannot see: a ``mutation`` that matches its pattern, whose + control speaks and whose corpus is quiet, yet **changes nothing observable when applied** — a + reintroduction raised into a swallowing handler, a field nobody reads, a flag nobody branches on. + ``re.search(pattern, mutation)`` proves the mutation is *well-formed*; it never proves it *bites*. + + - ``mutation_path`` — the file the reintroduction lands in, relative to ``root``. + - ``observable`` — the named artifact that must go red when the mutation is applied: a + ``tests/test_x.py::test_y`` pytest node id. When both fields are set the mode PROVES the claim + by execution — it runs the observable on a scratch copy of the tree (baseline must be green), + applies the mutation, and requires the observable to FAIL (and to fail as a test failure, not a + collection/usage error, which fails closed). When only ``mutation_path`` is set the mode falls + back to a coarse static backstop. + + Both fields default empty, and their absence means **"not yet proven by execution"** — never + "proven vacuous". They are opt-in per claim because a live proof spawns a pytest subprocess per + claim; a claim with neither field is reported as SKIPPED, not failed. + + Honest limits of the proving mode, stated so they are not overclaimed: + + - Application is **append-based**: the mutation text is appended to the scratch target, so it + breaks a fixture by *redefinition shadowing*. That faithfully reddens a well-formed + reintroduction in a fixture; it does not reproduce every in-function reintroduction a real claim + might describe. + - The static backstop is a **coarse same-file heuristic** — a ``raise`` in the mutation landing + lexically in a ``try`` whose every handler swallows (bare/``Exception``, log-only body). It + proves **nothing** in general: it cannot see a swallow in a *caller* rather than at the landing + site, so it would miss the very cross-file instance that motivated this item. It is a screen, + not a proof, and must not be written up as one. """ pattern: str positive_control: str mutation: str + mutation_path: str = "" + observable: str = "" @dataclass(frozen=True) @@ -143,6 +179,13 @@ class Findings: checked_anchors: int = 0 checked_absences: int = 0 skipped_anchors: int = 0 + #: Populated only by :func:`prove_absences`. ``proved_absences`` counts claims whose observable + #: went red under the applied mutation (a live proof); ``static_screened`` counts claims that took + #: the static backstop (a screen, not a proof); ``skipped_absences`` counts claims carrying no + #: ``mutation_path`` (nothing to apply). UNPROVEN and PROVE-ERROR outcomes go into ``problems``. + proved_absences: int = 0 + static_screened: int = 0 + skipped_absences: int = 0 @property def ok(self) -> bool: @@ -271,6 +314,12 @@ def load_scorecard(path: Path) -> list[Cell]: # No default. A missing mutation must be authored, not inferred — see the # Absence docstring on why deriving one from the pattern is worse than none. mutation=str(a["mutation"]), + # Optional, and deliberately NOT refused at load. A hard requirement here would + # void every already-authored absence claim (none carry these yet), and their + # re-authoring is out of this script's reach (ADR 0156 §7). Absent means "not + # yet proven by execution", surfaced by --prove-absences, not "proven vacuous". + mutation_path=str(a.get("mutation_path", "")), + observable=str(a.get("observable", "")), ) for a in raw.get("absence", []) ), @@ -431,6 +480,244 @@ def _grep_count(pattern: str, files: list[Path]) -> int: return sum(1 for f in files if rx.search(f.read_text(encoding="utf-8", errors="replace"))) +# --- proving an absence by mutation (--prove-absences) -------------------------------------------- +# +# check_absences proves a mutation is well-formed (its pattern fires on it). It cannot prove the +# mutation BITES: applied, does anything observable go red? A reintroduction raised into a swallowing +# handler passes every check in check_absences and changes nothing. This mode closes that hole by +# EXECUTING the claim — mutate a scratch copy of the tree, run the named observable, require it to go +# red — and it fails closed on every code that is not an honest test failure, so a typo'd node or an +# already-red observable can never masquerade as "the control bit". The whole pass runs inside a +# TemporaryDirectory scratch copy, so it never mutates `root` and never trips the committed-tree scan +# on itself. + + +def _scratch_ignore(dirpath: str, names: list[str]) -> set[str]: + """Names to skip when copying `root` into the scratch tree. Beyond the usual VCS/venv/cache noise + this refuses secrets and posture data — ``.env*``, ``*.db`` and its WAL sidecars (the local + store), and the vault's ``docs/security`` tree (ADR 0156 §7). The vault runs this module against + the REAL tree, so a scratch copy carrying those would spill them into a world-default temp dir, + which CLAUDE.md §9 forbids this module reading at all. Public-repo runs never see them (no + committed ``.env``/``*.db``, ``docs/security`` absent), so this is defence for the vault run.""" + ignored = set( + shutil.ignore_patterns( + ".git", + ".venv", + "__pycache__", + "node_modules", + ".env", + ".env.*", + "*.db", + "*.db-wal", + "*.db-shm", + )(dirpath, names) + ) + # `docs/security` is path-specific, not a basename glob: skip a `security` entry only directly + # under `docs`, leaving any unrelated `security` elsewhere in the tree copied. + if Path(dirpath).name == "docs" and "security" in names: + ignored.add("security") + return ignored + + +def _copy_scratch(root: Path, dest: Path) -> Path: + """Copy `root` into `dest`, skipping VCS/venv/cache dirs and — defensively, for the vault run — + secrets, the local store, and vault posture data (:func:`_scratch_ignore`). Never writes to + `root`.""" + shutil.copytree(root, dest, ignore=_scratch_ignore) + return dest + + +def _is_within_tree(rel: str) -> bool: + """True only for a repo-relative path with no anchor and no ``..`` component — one that cannot + escape the scratch copy when joined onto it. ``mutation_path`` comes from the authored scorecard, + so it is untrusted for this purpose: an absolute or ``..``-bearing value is refused, not resolved.""" + p = Path(rel) + if p.is_absolute() or p.anchor: + return False + return ".." not in p.parts + + +def _apply_mutation(scratch: Path, mutation_path: str, mutation: str) -> None: + """Append the reintroduction to the scratch target — redefinition shadowing is what makes a + well-formed reintroduction actually break an observable. Never called against `root`.""" + target = scratch / mutation_path + with target.open("a", encoding="utf-8") as fh: + fh.write("\n" + mutation + "\n") + + +def _run_node(scratch: Path, node_id: str, python: str, timeout: float) -> int: + """Run one pytest node inside the scratch copy and return its exit code. + + Invoked with ``--rootdir `` and ``cwd=scratch`` and ``-o addopts=`` so no repo + ``conftest``/``pyproject``/addopts leaks into the child run, and ``-p no:cacheprovider`` so it + writes nothing back. A timeout is treated as a non-{0,1} code — fail closed, never a proof. + """ + try: + proc = subprocess.run( # nosec B603 B607 - fixed argv, no shell; python is sys.executable, node id is scorecard-authored not shell-interpreted + [ + python, + "-m", + "pytest", + "-q", + "--rootdir", + str(scratch), + "-p", + "no:cacheprovider", + "-o", + "addopts=", + node_id, + ], + cwd=scratch, + capture_output=True, + text=True, + check=False, + timeout=timeout, + ) + except subprocess.TimeoutExpired: + return 124 # non-zero and non-1: fails closed as a PROVE-ERROR, never counted as a proof + return proc.returncode + + +def _handler_swallows(handler: ast.ExceptHandler) -> bool: + """A handler that catches broadly (bare or ``Exception``/``BaseException``) with a log-only/``pass`` + body and no re-raise — the shape that eats a reintroduced exception.""" + caught = handler.type + if not ( + caught is None + or (isinstance(caught, ast.Name) and caught.id in {"Exception", "BaseException"}) + ): + return False + if any(isinstance(n, ast.Raise) for stmt in handler.body for n in ast.walk(stmt)): + return False # a re-raise is not a swallow + return all(isinstance(stmt, (ast.Pass, ast.Expr)) for stmt in handler.body) + + +def _landing_swallows(source: str) -> bool: + """True if `source` contains a ``try`` whose EVERY handler swallows (see :func:`_handler_swallows`). + + A coarse same-file heuristic — it proves nothing in general and cannot see a swallow in a caller. + """ + try: + tree = ast.parse(source) + except SyntaxError: + return False + return any( + isinstance(node, ast.Try) + and bool(node.handlers) + and all(_handler_swallows(h) for h in node.handlers) + for node in ast.walk(tree) + ) + + +def _prove_one( + a: Absence, + cell_id: str, + root: Path, + scratch_dir: Path, + findings: Findings, + *, + python: str, + timeout: float, +) -> None: + if not a.mutation_path: + # Nothing to apply. Reported, not failed: opt-in per claim (a live proof spawns a subprocess). + findings.skipped_absences += 1 + return + if not _is_within_tree(a.mutation_path): + # mutation_path is authored data. An absolute path or a `..` escape would let _apply_mutation + # write outside the scratch copy (and the is_file probe below read outside `root`), defeating + # the 'never touches root' guarantee. Refuse it rather than resolve it. + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR — mutation_path {a.mutation_path!r} is not a " + "repo-relative path inside the tree (it is absolute or contains '..'), so applying the " + "mutation could escape the scratch copy" + ) + return + target_in_root = root / a.mutation_path + if not target_in_root.is_file(): + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR — mutation_path {a.mutation_path!r} is not a file " + "in the tree, so the mutation cannot be applied" + ) + return + + if a.observable: + scratch = _copy_scratch(root, scratch_dir) + baseline = _run_node(scratch, a.observable, python, timeout) + if baseline != 0: + # An already-red or uncollectable observable cannot attribute its red to the mutation. + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR — observable {a.observable!r} is not green on " + f"the pristine tree (pytest exit {baseline}); a red or uncollectable baseline cannot " + "be attributed to the mutation" + ) + return + _apply_mutation(scratch, a.mutation_path, a.mutation) + mutated = _run_node(scratch, a.observable, python, timeout) + if mutated == 1: + findings.proved_absences += 1 # live proof: the control bit + elif mutated == 0: + findings.problems.append( + f"{cell_id}: absence claim UNPROVEN — applying the mutation to {a.mutation_path} left " + f"observable {a.observable!r} green (pytest exit 0); the mutation changes nothing the " + "control catches, so this claim is syntax without behaviour" + ) + else: + # exit 2/3/4/5/...: collection or usage error. NEVER a proof — a typo'd node or a mutation + # that merely breaks import must not masquerade as the control biting. + findings.problems.append( + f"{cell_id}: absence claim PROVE-ERROR — mutated run of {a.observable!r} errored " + f"(pytest exit {mutated}) rather than failing; a collection or usage error must not " + "count as the control biting" + ) + return + + # Static backstop: mutation_path but no observable. A screen, not a proof (see Absence docstring). + findings.static_screened += 1 + if re.search(r"\braise\b", a.mutation) and _landing_swallows( + target_in_root.read_text(encoding="utf-8", errors="replace") + ): + findings.problems.append( + f"{cell_id}: absence claim SUSPECT (static heuristic) — its reintroduction raises into " + f"{a.mutation_path}, which has a try/except that swallows (bare or Exception, log-only " + "body), so a live raise there may be caught and prove nothing. Supply an `observable` to " + "prove it by execution" + ) + + +def prove_absences( + cells: list[Cell], + root: Path, + *, + python: str = sys.executable, + timeout: float = 120.0, +) -> Findings: + """Prove each absence claim BITES: apply its mutation to a scratch copy and require its observable + to go red. Fails closed on anything that is not an honest baseline-green / mutated-fail pair. + + This is separate from :func:`verify` and opt-in (``--prove-absences``) because it spawns a pytest + subprocess per provable claim. It never touches `root`. + """ + findings = Findings() + resolved_root = root.resolve() + with tempfile.TemporaryDirectory(prefix="asvs_prove_") as td_base: + base = Path(td_base) + i = 0 + for c in cells: + for a in c.absence: + i += 1 + _prove_one( + a, + c.id, + resolved_root, + base / f"scratch_{i}", + findings, + python=python, + timeout=timeout, + ) + return findings + + def _sort_key(cell_id: str) -> tuple[int, ...]: try: return tuple(int(p) for p in cell_id.split(".")) @@ -596,20 +883,59 @@ def render_current(cells: list[Cell], *, anchor_sha: str) -> str: return chr(10).join(lines) + chr(10) +def _run_prove_absences(scorecard: Path, root: Path) -> int: + """The ``--prove-absences`` entry point: execute-prove every absence claim (see + :func:`prove_absences`). Needs no corpus — it applies mutations, it does not grep for patterns.""" + try: + cells = load_scorecard(scorecard) + except ScorecardError as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 # could not measure — never 0, never confused with "clean" + findings = prove_absences(cells, root) + print( + f"prove-absences: proved {findings.proved_absences} by mutation; " + f"{findings.static_screened} static-screened; {findings.skipped_absences} skipped; " + f"{len(findings.problems)} problem(s)" + ) + for p in findings.problems: + print(f" FAIL {p}", file=sys.stderr) + return 0 if findings.ok else 1 + + def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description="Verify or render the ASVS scorecard (ADR 0156).") ap.add_argument("--scorecard", type=Path, required=True) - ap.add_argument("--corpus", type=Path, required=True) + # Not required: --prove-absences applies mutations and never greps for patterns, so it needs no + # corpus. Verify mode still does; that is enforced after parsing, not by argparse. + ap.add_argument("--corpus", type=Path, required=False) ap.add_argument( "--root", type=Path, default=Path.cwd(), help="tree the evidence anchors point into" ) ap.add_argument("--render", type=Path, help="write the generated CURRENT.md here") + # A separate, opt-in mode: prove each absence claim BITES by applying its mutation to a scratch + # copy and requiring its observable to go red. Opt-in because it spawns a pytest subprocess per + # provable claim; kept out of the default verify path, which stays purely static. + ap.add_argument( + "--prove-absences", + action="store_true", + help="execute-prove absence claims (apply mutation to a scratch tree, require observable red)", + ) # NO --anchor-sha injected by CI. The anchor is the commit the EVIDENCE was read on — a property # of the assessment, recorded in [scorecard].anchor_commit. Passing ${{ github.sha }} made the # rendered file differ on every run, so the drift check could never pass: a gate that cannot go # green is as useless as one that cannot go red, and this one shipped that way. args = ap.parse_args(argv) + if args.prove_absences: + return _run_prove_absences(args.scorecard, args.root) + + if args.corpus is None: + print( + "error: --corpus is required to verify the scorecard (only --prove-absences may omit it)", + file=sys.stderr, + ) + return 2 + try: findings = verify(args.scorecard, args.corpus, args.root) except ScorecardError as exc: diff --git a/tests/test_asvs_scorecard.py b/tests/test_asvs_scorecard.py index 0c3b15fb..58fd0bc3 100644 --- a/tests/test_asvs_scorecard.py +++ b/tests/test_asvs_scorecard.py @@ -23,6 +23,7 @@ Cell, Findings, ScorecardError, + _copy_scratch, check_absences, check_anchors, check_completeness, @@ -31,6 +32,8 @@ count, load_corpus, load_scorecard, + main, + prove_absences, render_current, verify, ) @@ -675,3 +678,383 @@ def test_completeness_accepts_a_decided_cell_evidenced_only_by_an_absence_claim( Cell(id="2.1.1", level=3, verdict="unverified"), ] assert not [p for p in check_completeness(cells, CORPUS) if "carry NO anchor" in p] + + +# --- --prove-absences: a mutation that matches is not a mutation that BITES (#1006) --------------- +# +# check_absences proves a mutation's pattern fires on it; it never applies the mutation. So a +# well-formed reintroduction that would change nothing observable passes every check. prove_absences +# closes that hole by EXECUTING the claim: mutate a scratch copy, run the named observable, require it +# to go red -- and fail closed on any exit code that is not an honest test failure. Every fixture tree +# lives in tmp_path (never in the scanned packages), and the mutation is applied only to a scratch +# copy in a system TemporaryDirectory, so nothing here touches the committed corpus or tmp_path. + +_SCANNER = "def scan(p):\n return 'clean'\n" +_OBS_TEST = "from scanner import scan\n\n\ndef test_clean():\n assert scan('x') == 'clean'\n" + + +def _module(tmp_path: Path, name: str, body: str) -> Path: + """Write a code module fixture into tmp_path (the code the reintroduction lands in).""" + p = tmp_path / name + p.write_text(body, encoding="utf-8") + return p + + +def _obs_test(tmp_path: Path, name: str, body: str) -> Path: + """Write an observable pytest module fixture into tmp_path.""" + p = tmp_path / name + p.write_text(body, encoding="utf-8") + return p + + +def _live_claim(mutation: str, mutation_path: str, observable: str) -> Cell: + # pattern/positive_control are irrelevant to prove_absences (they drive check_absences); supply + # harmless values so the required fields are present. + return Cell( + id="1.1.1", + level=1, + verdict="fail", + absence=( + Absence( + pattern="x", + positive_control="y", + mutation=mutation, + mutation_path=mutation_path, + observable=observable, + ), + ), + ) + + +def test_prove_absences_proves_a_claim_when_the_mutation_reddens_its_observable( + tmp_path: Path, +) -> None: + """The positive half: a mutation that shadows `scan` reddens the observable, so the claim BITES. + + Falsified by making `_apply_mutation` a no-op: the observable stays green, the mode reports + UNPROVEN, and the `.ok`/`proved_absences == 1` assertions go RED. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + claim = _live_claim( + 'def scan(p): return "infected"', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert findings.ok, findings.problems + assert findings.proved_absences == 1 + + +def test_prove_absences_fails_when_the_mutation_reddens_nothing(tmp_path: Path) -> None: + """The negative control the brief requires: a mutation to a file the observable never imports + reddens nothing, so the claim is UNPROVEN and the mode FAILS. + + Falsified by making the mode accept a mutated exit==0 as a pass (dropping the exit==1 + requirement): the non-biting claim then reports ok, and `not findings.ok` goes RED -- proving the + mode can actually fail. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _module( + tmp_path, "unrelated.py", "VALUE = 1\n" + ) # exists, but the observable does not import it + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + claim = _live_claim( + 'def scan(p): return "infected"', "unrelated.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("UNPROVEN" in p for p in findings.problems), findings.problems + assert findings.proved_absences == 0 + + +def test_prove_absences_fails_closed_when_the_observable_is_already_red(tmp_path: Path) -> None: + """An observable that fails on the pristine tree cannot attribute its red to the mutation. + + Falsified by removing the baseline-green check: the already-red observable stays red under the + mutation, is miscounted as `proved`, and this test's `not findings.ok` goes RED. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test( + tmp_path, + "test_scanner.py", + "from scanner import scan\n\n\ndef test_clean():\n assert scan('x') == 'DIFFERENT'\n", + ) + claim = _live_claim( + 'def scan(p): return "infected"', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("PROVE-ERROR" in p and "baseline" in p for p in findings.problems), findings.problems + assert findings.proved_absences == 0 + + +def test_prove_absences_fails_closed_when_the_mutation_errors_instead_of_failing( + tmp_path: Path, +) -> None: + """A mutation that breaks IMPORT of the observable's module errors at collection (pytest exit 4), + not a test failure (exit 1). A collection/usage error must NEVER count as the control biting -- + otherwise a typo'd node or an import-breaking mutation rebuilds the exact vacuity being fixed. + + Falsified by changing the mutated-run requirement from `exit == 1` to `exit != 0`: the exit-4 + collection error then masquerades as `proved`, and this test's `not findings.ok` goes RED. + Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + # Appended at module level, this raises when `scanner` is imported -> collection error, not a + # failing assertion. + claim = _live_claim( + 'raise RuntimeError("reintroduced")', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("PROVE-ERROR" in p and "errored" in p for p in findings.problems), findings.problems + assert findings.proved_absences == 0 + + +def test_prove_absences_static_backstop_flags_a_raise_into_a_swallowing_file( + tmp_path: Path, +) -> None: + """With no observable, the static backstop flags a `raise` landing in a file whose try/except + swallows (bare/Exception, log-only body). A screen, not a proof -- but it fails the mode. + + Falsified by forcing `_landing_swallows` to return False: the swallow is not flagged, `findings.ok` + becomes True, and this test's `not findings.ok` goes RED. Restored. + """ + _module( + tmp_path, + "caller.py", + "import logging\n\nlog = logging.getLogger(__name__)\n\n\n" + "def reconcile():\n try:\n work()\n except Exception:\n" + " log.exception('reconcile failed')\n", + ) + claim = Cell( + id="13.3.4", + level=3, + verdict="fail", + absence=( + Absence( + pattern="x", + positive_control="y", + mutation='raise RuntimeError("reintroduced")', + mutation_path="caller.py", + observable="", # no observable -> static backstop + ), + ), + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("SUSPECT" in p and "swallow" in p for p in findings.problems), findings.problems + assert findings.static_screened == 1 + + +def test_prove_absences_static_backstop_passes_a_non_swallowing_file(tmp_path: Path) -> None: + """REACH control: the static backstop must NOT flag a `raise` into a handler that re-raises -- + proving the heuristic reads the handler body, not merely the presence of a try/except. + + Falsified by forcing `_landing_swallows` to return True: the re-raising file is flagged, and this + test's `assert findings.ok` goes RED. Restored. + """ + _module( + tmp_path, + "plain.py", + "def reconcile():\n try:\n work()\n except Exception:\n raise\n", + ) + claim = Cell( + id="13.3.4", + level=3, + verdict="fail", + absence=( + Absence( + pattern="x", + positive_control="y", + mutation='raise RuntimeError("reintroduced")', + mutation_path="plain.py", + observable="", + ), + ), + ) + findings = prove_absences([claim], tmp_path) + assert findings.ok, findings.problems + assert findings.static_screened == 1 + assert not any("SUSPECT" in p for p in findings.problems) + + +def test_prove_absences_leaves_the_root_tree_untouched(tmp_path: Path) -> None: + """The mode must run OUT of the tracked tree: mutation lands only on the scratch copy. + + Falsified by pointing `_apply_mutation` at `root` instead of the scratch copy: root's scanner.py + changes, `after == before` goes RED (and the claim also drops to UNPROVEN). Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + before = {p: p.read_bytes() for p in sorted(tmp_path.rglob("*")) if p.is_file()} + claim = _live_claim( + 'def scan(p): return "infected"', "scanner.py", "test_scanner.py::test_clean" + ) + findings = prove_absences([claim], tmp_path) + assert findings.proved_absences == 1, findings.problems + after = {p: p.read_bytes() for p in sorted(tmp_path.rglob("*")) if p.is_file()} + assert after == before + + +def test_load_reads_optional_mutation_path_and_observable_and_omitting_them_still_loads( + tmp_path: Path, +) -> None: + """Round-trip: the two new fields load when present, and OMITTING them still loads (vault-safety -- + the ~81 existing absence claims carry neither and must stay loadable). + + Falsified by dropping the `.get` wiring in load_scorecard (hardcoding ``mutation_path=""``): half + (a) then reads "" and its assertion goes RED, while half (b) stays green -- proving the round-trip + is actually asserted. Restored. + """ + with_fields = tmp_path / "with.toml" + with_fields.write_text( + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "clamd"\n' + ' positive_control = "ScanRejected"\n' + ' mutation = "import clamd"\n' + ' mutation_path = "messagefoundry/scan.py"\n' + ' observable = "tests/test_scan.py::test_rejects"\n', + encoding="utf-8", + ) + a = load_scorecard(with_fields)[0].absence[0] + assert a.mutation_path == "messagefoundry/scan.py" + assert a.observable == "tests/test_scan.py::test_rejects" + + without_fields = tmp_path / "without.toml" + without_fields.write_text( + '[[cell]]\nid = "1.1.2"\nlevel = 2\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "clamd"\n' + ' positive_control = "ScanRejected"\n' + ' mutation = "import clamd"\n', + encoding="utf-8", + ) + b = load_scorecard(without_fields)[0].absence[0] + assert b.mutation_path == "" and b.observable == "" + + +def test_prove_absences_refuses_a_mutation_path_that_escapes_the_scratch_tree( + tmp_path: Path, +) -> None: + """`mutation_path` is authored data. An absolute path or a `..` escape would let the mutation land + OUTSIDE the scratch copy (defeating the 'never touches root' guarantee), so the mode refuses it as + a PROVE-ERROR before applying anything -- it never counts as a proof. + + Falsified by making `_is_within_tree` return True unconditionally: the `..` path is no longer + refused, the run falls through to the is_file probe with a different message, and this test's + `any("repo-relative" in p ...)` assertion goes RED. Restored. + """ + _module(tmp_path, "scanner.py", _SCANNER) + _obs_test(tmp_path, "test_scanner.py", _OBS_TEST) + claim = _live_claim( + 'def scan(p): return "infected"', + "../escape.py", # a `..` that would climb out of the scratch copy + "test_scanner.py::test_clean", + ) + findings = prove_absences([claim], tmp_path) + assert not findings.ok + assert any("PROVE-ERROR" in p and "repo-relative" in p for p in findings.problems), ( + findings.problems + ) + assert findings.proved_absences == 0 + + +def test_copy_scratch_excludes_secrets_store_and_vault_posture(tmp_path: Path) -> None: + """The scratch copy the vault mutation-run reads must never carry secrets, the local store, or the + vault posture tree -- CLAUDE.md §9 forbids this module reading them at all. `_copy_scratch` skips + `.env*`, `*.db`(+WAL sidecars), and `docs/security`, while ordinary sources are still copied. + + Falsified by reverting `_scratch_ignore` to the bare VCS/venv/cache patterns: the `.env`, `*.db` + and `docs/security` fixtures are then copied into the scratch dir and every `not (dest/...).exists()` + assertion goes RED, while the `keep.py` assertion stays green. Restored. + """ + (tmp_path / ".env").write_text("EXAMPLE_PLACEHOLDER=not-a-secret\n", encoding="utf-8") + (tmp_path / "local.db").write_text("binary-store\n", encoding="utf-8") + (tmp_path / "local.db-wal").write_text("wal\n", encoding="utf-8") + (tmp_path / "docs" / "security").mkdir(parents=True) + (tmp_path / "docs" / "security" / "posture.toml").write_text("real = true\n", encoding="utf-8") + (tmp_path / "docs" / "PUBLIC.md").write_text("# public\n", encoding="utf-8") + (tmp_path / "keep.py").write_text("KEEP = 1\n", encoding="utf-8") + + dest = tmp_path.parent / "scratch_out" + _copy_scratch(tmp_path, dest) + + assert not (dest / ".env").exists() + assert not (dest / "local.db").exists() + assert not (dest / "local.db-wal").exists() + assert not (dest / "docs" / "security").exists() + # ordinary sources and other docs survive the copy + assert (dest / "keep.py").read_text(encoding="utf-8") == "KEEP = 1\n" + assert (dest / "docs" / "PUBLIC.md").exists() + + +def _biting_scorecard(sc: Path, mutation_path: str) -> None: + """Write a one-claim scorecard whose live absence claim points at `mutation_path`.""" + sc.write_text( + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "x"\n' + ' positive_control = "y"\n' + " mutation = 'def scan(p): return \"infected\"'\n" + f' mutation_path = "{mutation_path}"\n' + ' observable = "test_scanner.py::test_clean"\n', + encoding="utf-8", + ) + + +def test_main_prove_absences_returns_0_on_a_biting_claim(tmp_path: Path) -> None: + """The CLI contract CI depends on: `--prove-absences` exits 0 when every claim's mutation reddens + its observable. Exercises `main` -> `_run_prove_absences` end to end, not just `prove_absences`. + + Falsified by changing `_run_prove_absences`'s `return 0 if findings.ok else 1` to `return 1`: this + test's `rc == 0` goes RED while the non-biting test below stays green. Restored. + """ + tree = tmp_path / "tree" + tree.mkdir() + _module(tree, "scanner.py", _SCANNER) + _obs_test(tree, "test_scanner.py", _OBS_TEST) + sc = tmp_path / "sc.toml" + _biting_scorecard(sc, "scanner.py") + rc = main(["--scorecard", str(sc), "--root", str(tree), "--prove-absences"]) + assert rc == 0 + + +def test_main_prove_absences_returns_1_on_a_nonbiting_claim(tmp_path: Path) -> None: + """The other half of the contract: `--prove-absences` exits 1 when a claim is UNPROVEN (its + mutation reddens nothing). Proves the CLI's non-zero failure path, not only the library's. + + Falsified by changing `_run_prove_absences`'s `return 0 if findings.ok else 1` to `return 0`: + this test's `rc == 1` goes RED while the biting test above stays green. Restored. + """ + tree = tmp_path / "tree" + tree.mkdir() + _module(tree, "scanner.py", _SCANNER) + _module(tree, "unrelated.py", "VALUE = 1\n") # present, but the observable never imports it + _obs_test(tree, "test_scanner.py", _OBS_TEST) + sc = tmp_path / "sc.toml" + _biting_scorecard(sc, "unrelated.py") + rc = main(["--scorecard", str(sc), "--root", str(tree), "--prove-absences"]) + assert rc == 1 + + +def test_main_verify_without_corpus_returns_exit_2(tmp_path: Path) -> None: + """Verify mode needs the corpus; omitting `--corpus` (without `--prove-absences`) must exit 2 -- + could-not-measure, never confused with a clean 0. Proves the argparse-independent guard in `main`. + + Falsified by deleting the `if args.corpus is None: ... return 2` branch in `main`: it then falls + through to `verify(...)` with `corpus=None`, raising instead of returning 2, and this test's + `rc == 2` goes RED. Restored. + """ + sc = tmp_path / "sc.toml" + sc.write_text( + '[[cell]]\nid = "1.1.1"\nlevel = 1\nverdict = "fail"\n' + " [[cell.absence]]\n" + ' pattern = "x"\n' + ' positive_control = "y"\n' + ' mutation = "import x"\n', + encoding="utf-8", + ) + rc = main(["--scorecard", str(sc), "--root", str(tmp_path)]) + assert rc == 2 From 089bcdaeb32bf5c10cc55c3e46b3ade277500df7 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Thu, 6 Aug 2026 02:03:25 -0500 Subject: [PATCH 2/2] docs(backlog): flip #1006 to shipped -- the absence gate can now prove behaviour (BACKLOG #1006) Flip the #1006 banner from filed to shipped. It is written as a capability claim, not a closure claim: the `--prove-absences` mode CAN catch a well-formed-but-vacuous reintroduction once a claim carries an `observable`, but the default `verify` path is byte-unchanged and no authored claim carries one yet, so nothing new is blocked by this alone today -- the honest present-tense state for a not-deployed beta. Banner lines of #1006 ONLY. The ranked table, the four census distribution lines, and every other item's banner are untouched. The status census was NOT recomputed. --- docs/BACKLOG.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 38305897..065e36db 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -3922,13 +3922,21 @@ record and the cell's current score live in the vault scorecard and are not rest ## 1006. A mutation that matches is not a mutation that bites: the absence-claim gate proves syntax, never behaviour -> 🔢 **Filed 2026-08-04 — not started. Scored 2026-08-04 → P2.** Value **6/10** · Difficulty -> **3/10** · _quick win_. `check_absences` admits an ASVS absence claim on `re.search(a.pattern, -> a.mutation)` (`scripts/asvs/scorecard.py:395`) — one string field of a TOML row matched against -> another — so a well-formed reintroduction that would change nothing if applied passes all three -> of the gate's failure modes and certifies a non-control into the compliance record; the -> remainder is a required per-claim observable plus a mode that applies the mutation and requires -> that observable to go red, in one stdlib script and its fixture tests. +> ✅ **SHIPPED 2026-08-06 — a new opt-in mode can prove an absence claim BITES, which the pattern +> check structurally cannot.** Value **6/10** · Difficulty **3/10** · _quick win_. +> `scripts/asvs/scorecard.py` gains a `--prove-absences` mode: per claim it applies the `mutation` to +> a scratch copy of the tree and requires a named `observable` (a pytest node id) to go RED, failing +> closed on any exit code that is not an honest test failure (an already-red baseline, an +> uncollectable node, or a mutation that only breaks import is a PROVE-ERROR, never a proof). So a +> well-formed reintroduction that would change nothing if applied CAN be caught the moment its claim +> carries an `observable` — but the default `verify` path is byte-unchanged and no authored claim +> carries one yet, so nothing new is blocked by this alone today. Two optional `Absence` fields +> (`mutation_path`, `observable`) feed it, a coarse same-file static backstop screens claims that +> carry no observable, the scratch copy refuses secrets / the store / `docs/security` (defence for the +> eventual vault run), and fixture negative controls plus a CLI exit-code test prove the mode itself +> can go red. Public repo script + fixtures only; wiring the mode over the vault's ~81 existing +> absence claims (untouched) and backfilling their observables is the owner's follow-up +> (`scorecard.py:14-16`, ADR 0156 §7). **Cluster:** Security & Compliance. **Priority:** P2. **Verdict:** build. **Severity:** medium — the defect is in the instrument, not the engine, and a green instrument that cannot go red is the