Token path: convert HTML once, retire the always-on bug ledger row, stage-scoped runbook reads, docs-ready fast path

CHANGELOG.md

@@ -18,6 +18,12 @@ Credit the person or project that surfaced the change — the credit line is the
 - process(skill-routing): `learned-css-that-never-applied` routed **baseline** for mdl/review/gate at stages 5–6, trigger-phrased on *"a style change that appears to have done nothing, or an app still grey after a design port every instrument called green"*. Baseline rather than situational on this repo's own rule that a skill living only in the situational table goes unnoticed by projects that are not hunting for it — and not being reached is precisely what this one records — Maurits Visser, from a MOC/PSSR app replacement
 
 ## 2026-09-09
+- fix(bin/sync-project.sh + bin/check-portability.sh): the retire-ledger-row rewrite passed its regex to awk as `-v re="^\\|..."`; gawk (the GitHub runner's awk) processes escapes in -v values, so the pattern became an alternation that matched every line and the whole CLAUDE.local.md was replaced with copies of one row — green under mawk locally, red only in CI. Bracket expressions now; check-portability flags any backslash inside an awk -v value — the toolkit merge queue
+- process(tests/ab + process/token-path-ab-2026-09-09.md): the token-path A/B — answer-key corpus generator and scorer (synthetic Harbour Berth Booking, stdlib only), and the report: wave 1 Sonnet P–2 (branch 31% cheaper, recall 54%→81–94%), wave 2 Haiku P–4 on three corpora (4–8 min, <$1.10 each), the non-reproducing hollow run, the per-stage mandated reading load, and the proportionality finding — the toolkit merge queue
+- process(skills/conversion-runbook.md → Entry Modes): **the docs-ready fast path for a requirements-driven, text-native corpus (Markdown + HTML, no legacy code, no Office/PDF containers)** — measured cause four on the same project: the ledger wanted one disposition per file, every sidecar png/css/js included, for a corpus nothing extracts. Stage 0 = `source-sufficiency.sh init` + CAC-1 + sign-off, triage rows written `N/A (text-native corpus)`; Stage 1 = `bin/html-to-md.sh` once, the KB IS the converted text + `documents-index.md`, ONE ledger mark (`mark <p> 'sources/**' --artifact analysis/knowledge-base/documents-index.md --evidence ... --by ...`), images still read by vision with the index's count as denominator; Stage 2 = one use case per section with a `sourceRef`, rules lifted as stated, validation once; Stages 3–4 unchanged and not skippable (module boundaries, grants, script order). Every command was run against the real gates first, and two things had to change for it to pass: (1) `--adopt 1` is the wrong instrument — adoption waives the stages *before* the point (Stage 0, which ran) and leaves Stage 1 demanding an extraction report no extractor wrote — so the path records `--waive 1 --reason "text-native corpus: ..."` (Stage 2 still runs the ledger: index hidden → Stage 2 exits 1, verified); (2) `source-ledger.sh mark` now strips the source-root segment from a target, so `'sources/**'` — the natural spelling — matches the corpus instead of being refused. Verified that a glob mark whose artifact names every basename yields EXTRACTED for every row, css/js/png included, and that image rows owe no `--media`. `bin/lib/source-formats.tsv` gains route `chrome` for css|map|woff|woff2|ttf|otf|eot|ico ("web export chrome — listed, never content"; `.js` deliberately stays on the code route — one extension, one route, and a `.js` in a migration corpus is application source); `bin/html-to-md.sh` is routed baseline (ba, stages 0–1: "HTML in the source corpus — convert once before anyone reads it"), surfaces re-rendered, baseline 74,147 of 80,000 words. Fixture: tests/wave2/test-source-ledger.sh T12 — the toolkit merge queue, from a Markdown+HTML requirements project
+- new(bin/gate-check.sh + skills/conversion-runbook.md): **a stage run names the runbook lines to read — `Read for this gate: skills/conversion-runbook.md §"Stage N — …" (lines A–B) + §1b Live Checklist (lines C–D)` — instead of the whole 11,700-word file.** Measured cause three on the same Markdown+HTML requirements project: every project's CLAUDE.local.md says "read the runbook FIRST — every session", so every session read all of it. What a stage session needs is §1b (applies to every stage) plus its own stage's §2 section; the spans are derived on every run by grepping the runbook's actual headings (`### Stage N — `, `## 1b.`; keyed on the em dash because `### Stage 0 runs in every entry mode` also starts with `### Stage 0`), never hard-coded, and `build-ready` reads Stage 5's. The runbook's top now carries the matching "How to read this file: not whole" paragraph. Fixture: tests/wave2/test-bug03-gates.sh T10 (line numbers checked against the file's headings, span ends at the next heading, build-ready → Stage 5, whole-project run prints none); test-source-ledger.sh and test-stage-p.sh still pass — the toolkit merge queue, from a Markdown+HTML requirements project
+- fix(bin/sync-project.sh): **retires the always-on `bug-logs/mxcli-bugs.md` row in a project's CLAUDE.local.md → `bin/bug-lookup.sh`, and says so.** Measured cause two on the same Markdown+HTML requirements project: master replaced the 47,500-word ledger's baseline row with bin/bug-lookup.sh on 2026-09-08, but every project scaffolded before that still read the whole ledger every session. A marked routing block was already re-rendered by section 2d — silently; now the retirement is announced ("retired the 47k-word ledger row → bin/bug-lookup.sh"). A hand-written, UNMARKED `## Baseline routing` table (every project bootstrapped before 2026-08-18) got 2d's whole-table refusal and kept the row; new section 2e rewrites that ONE row in place with the row's own path prefix kept and every neighbouring row byte-identical (new `routing_row <name> [prefix]` in bin/lib/skill-routing.sh renders it from the table). CLAUDE.md's "mxcli-project-toolkit Integration" block is NOT edited — it is bootstrap-project.md's LLM merge, not something init-project.sh writes — the row is reported with the exact replacement instead. --dry-run announces and writes nothing; a second run is silent. Fixture: tests/wave2/test-bug12-sync.sh T11 (marked, unmarked, dry-run, idempotence, CLAUDE.md report) — the toolkit merge queue, from a Markdown+HTML requirements project
+- new(bin/html-to-md.sh + bin/lib/html-to-md.py): **convert a saved-webpage corpus to Markdown once, so no session reads raw HTML again.** A requirements-driven project whose sources were Markdown plus browser "save page as" exports (page + `_files/` sidecar) was slow and token-hungry through Stages 1–4; measured cause one: agents opened the .html themselves, at 3–10× the tokens of the page's own text, every session. Now `bin/html-to-md.sh <project>` walks the source root (the inventory's own walk — bin/lib/source-inventory.py, same prune rules) and writes each page to `analysis/knowledge-base/text/<same path>.md` — headings, lists, tables, links, pre/code, bold/italic kept; script/style/noscript/svg/comments and nav/sidebar/footer/cookie/breadcrumb chrome dropped, conservatively (never `<main>`, `<article>`, or anything holding an h1–h3); every `<img>` as `![alt](src)`, inline base64 images decoded to `<page>_images/NN.ext` so a vision read needs no HTML. Each .md opens with source, raw size, words, image count and the section list with md line numbers — the read's denominator. Plus `documents-index.md`: ONE row per file under the root, sidecar png/css/js included, with role (page · markdown · image of <page> · chrome of <page> · other), so a single ledger glob mark can point at it. Idempotent (skips up-to-date pages; `--force`). Prints raw KB → md words and the ratio per page. Stdlib `html.parser` only, bash 3.2. Fixture: tests/wave2/test-html-to-md.sh over a synthetic export (its CAPTURE.md says why synthetic is acceptable for a format with no tool contract), 67 assertions: structure kept, every chrome sentinel absent, base64 written and referenced, section line numbers verified against the file, index names every basename, idempotence, --force, ratio printed — the toolkit merge queue, from a Markdown+HTML requirements project
 - new(project-bin/page-fidelity.js): **`class="bound"` — the wireframe author saying "this text is a sample of bound data".** The scorer already had this concept and applied it structurally: `<td>` is excluded from the text corpus because a wireframe's table cells are sample ROWS and a page that correctly BINDS them contains none of that literal text. The same is true of a bound value anywhere else on the screen, and there the structure gives nothing away — an `<h1>` holding a record's number and name looks exactly like an `<h1>` holding page copy, and a chip holding a filename looks exactly like a chip holding a label. Measured on a MOC/PSSR app replacement's project detail page, 2026-09-09: **54%**, and all three misses were sample values the page could not contain — the record's own title, and a mocked attachment's filename, twice. The page was right; the denominator was wrong, and the number was about to be read off as a gate verdict. So the marker is **DECLARED, not guessed**: an element carrying `bound` leaves both the text corpus and the class denominator, and guessing was the alternative and does not work — a heading is bound or it is not, and only the person who drew the screen knows which. Put it on the smallest element that holds only the value (`<h1 class="bound">` leaves no heading behind; `<span class="bound">` inside a chip leaves the chip). Fixture grows to **18 assertions** including that a bound heading leaves the DENOMINATOR rather than merely stopping being reported as missed — otherwise the score would be 1 of 2 and still look clean. Known-bad control without the marker: **5 of 18 fail**. Same page scores 100% (3/3 headings, 2/2 content) with it — Maurits Visser, from a MOC/PSSR app replacement
 - process(bug-logs): BUG-114 filed upstream as mendixlabs/mxcli#1076 with BUG-118 as its second reproduction; both drafts and the ledger header stamped FILED — the toolkit merge queue
 - fix(project-bin/page-fidelity.js): **structure classed as bound data, and the score of record silently going `null%` again.** A wireframe's own `<style>` defines two very different kinds of class — BOUND-DATA MOCKS (a repeated region of sample rows, whose literal text a correctly *binding* page contains none of) and STRUCTURE (the wrapper, the page header, the toolbar — furniture the page script genuinely must declare) — and the classifier **drops a mock's whole subtree**, so a misclassified structural class deletes page content from the denominator. The test was `uses === 1 && kept < 0.4`, once-used AND dominant, which contradicts the paragraph directly above it stating that a mock is a **repeated** region: a once-used class whose subtree was *small* fell through the second conjunct and was dropped as sample data. Measured on a MOC/PSSR app replacement's project overview, 2026-09-09: `.page-head` (uses=1, kept=0.513) was deleted **with the page's only `<h1>` inside it**, `.toolbar` (uses=1, kept=0.825) with it, and every dimension reported `0 of 0` — which normalizes to **`null%`**, so the run printed a clean report over an empty corpus. That is the same silent-null failure this file's own header note says was worth fixing, arriving a second time through the other conjunct, on a page that scores **100%** once the page is not gutted. The threshold is **removed rather than retuned**: it was a guess, and any value for it splits once-used wrappers on a measurement that has nothing to do with whether their content is bound data; a repeated region is still a mock at any size, which is the half that was always right. New fixture `tests/wave2/test-page-fidelity-mocks.sh`, **14 assertions**, with two captured wireframe shapes and a `CAPTURE.md` stating that only literal text and class prefixes were changed. Its **first assertion is that the score is not `null`, not that it is high** — a null on a wireframe that draws an `<h1>` means the scorer gutted the page, whatever the eventual percentage. Proven against a known-bad control carrying the old rule: **8 of 14 fail** — Maurits Visser, from a MOC/PSSR app replacement

README.md

@@ -733,6 +733,7 @@ The "When to use which skill" table above is *situational* — load a skill when
 | Deciding whether to extract at all, before any BRD gets generated | `skills/source-triage.md` |
 | Taking in a new source — before generating anything from it. Grades what the source can support; nothing else in this toolkit reads a source | `bin/source-sufficiency.sh` |
 | Closing Stage 1, or adding files to a source folder — every inventoried file must name the artifact that consumed it (text AND embedded diagrams), or carry a waiver; blocks Stages 1–2 until it does | `bin/source-ledger.sh` |
+| HTML in the source corpus — convert once before anyone reads it: `bin/html-to-md.sh <project>` writes each page as Markdown under analysis/knowledge-base/text/ (3–10× fewer tokens than the raw export; inline images decoded to files; section list with line numbers as the read's denominator) plus documents-index.md over EVERY file, which one ledger glob mark points at. A session that opens a .html itself has skipped this | `bin/html-to-md.sh` |
 | Deciding who answers a question — before putting any batch to the user. gap/conflict/choice/user-only is what keeps a gate batch at four questions instead of 127 | `bin/question-kinds.sh` |
 | Writing BRDs, especially several in parallel — "build" before the fan-out, "check" before any BRD is called done | `bin/facts-lock.sh` |
 | Building any module — before the first script. The mdl-agent's single per-module input | `skills/module-brief.md` |

ROUTING.md

@@ -64,6 +64,7 @@ picks the row up. That is the whole procedure — there is no second list to rem
 | Deciding whether to extract at all, before any BRD gets generated | `skills/source-triage.md` | ba | 0 | baseline |
 | Taking in a new source — before generating anything from it. Grades what the source can support; nothing else in this toolkit reads a source | `bin/source-sufficiency.sh` | ba | 0,1 | baseline |
 | Closing Stage 1, or adding files to a source folder — every inventoried file must name the artifact that consumed it (text AND embedded diagrams), or carry a waiver; blocks Stages 1–2 until it does | `bin/source-ledger.sh` | ba | 0,1,2 | baseline |
+| HTML in the source corpus — convert once before anyone reads it: `bin/html-to-md.sh <project>` writes each page as Markdown under analysis/knowledge-base/text/ (3–10× fewer tokens than the raw export; inline images decoded to files; section list with line numbers as the read's denominator) plus documents-index.md over EVERY file, which one ledger glob mark points at. A session that opens a .html itself has skipped this | `bin/html-to-md.sh` | ba | 0,1 | baseline |
 | Rendering a filled triage.md for review — the triage.html surface Stage 0 names. Renders only; the Stage 0 verdict stays with gate-check and the judgement with source-triage.md | `bin/triage-report.sh` | ba | 0 | ondemand |
 | Reviewing what the extraction actually produced — the Stage 1 surface, and the file the Stage 1 gate looks for. Renders a code-extracted and a document knowledge base alike, so a requirements-driven project gets the surface too; prints no zero that a second record does not agree with | `bin/extraction-report.sh` | ba | 1 | ondemand |
 | Assessing or planning a migration up front, before any pipeline is chosen | `skills/assess-migration.md` | ba | 0 | ondemand |

agents/ba-agent.md

@@ -44,6 +44,7 @@ You run discovery and the interview gates for {{PROJECT}}. You never touch the `
 | `skills/source-triage.md` | Deciding whether to extract at all, before any BRD gets generated |
 | `bin/source-sufficiency.sh` | Taking in a new source — before generating anything from it. Grades what the source can support; nothing else in this toolkit reads a source |
 | `bin/source-ledger.sh` | Closing Stage 1, or adding files to a source folder — every inventoried file must name the artifact that consumed it (text AND embedded diagrams), or carry a waiver; blocks Stages 1–2 until it does |
+| `bin/html-to-md.sh` | HTML in the source corpus — convert once before anyone reads it: `bin/html-to-md.sh <project>` writes each page as Markdown under analysis/knowledge-base/text/ (3–10× fewer tokens than the raw export; inline images decoded to files; section list with line numbers as the read's denominator) plus documents-index.md over EVERY file, which one ledger glob mark points at. A session that opens a .html itself has skipped this |
 | `bin/question-kinds.sh` | Deciding who answers a question — before putting any batch to the user. gap/conflict/choice/user-only is what keeps a gate batch at four questions instead of 127 |
 | `bin/facts-lock.sh` | Writing BRDs, especially several in parallel — "build" before the fan-out, "check" before any BRD is called done |
 | `skills/module-brief.md` | Building any module — before the first script. The mdl-agent's single per-module input |

bin/check-portability.sh

@@ -127,6 +127,20 @@ while IFS= read -r f; do
           'Use: arr=(); while IFS= read -r x; do arr+=("$x"); done < <(cmd)' ;;
     esac
 
+    # awk -v processes escape sequences in the VALUE (gawk warns and rewrites, mawk keeps the
+    # backslash): a regex passed as -v re="^\\|..." became "^|..." under gawk — the GitHub
+    # runner's awk — and matched every line (sync-project 2e rewrote a whole CLAUDE.local.md
+    # with copies of one row, CI-only, 2026-09-12). Bracket expressions carry no backslash.
+    case "$txt" in
+      *awk*)
+        # A backslash inside a $(...) substitution is the shell's, not the value's (e.g.
+        # -v ok="$(... | tr '\n' ' ')"): strip substitutions before looking.
+        if printf '%s' "$txt" | sed 's/\$([^)]*)//g' | grep -Eq -- '-v [A-Za-z_][A-Za-z_0-9]*="[^"]*\\'; then
+          report "$f" "$ln" "backslash inside an awk -v value (gawk rewrites escapes, mawk does not)" \
+            'Pass patterns without backslashes: [|] for a literal pipe, [.] for a dot, or read the regex from ENVIRON["re"].'
+        fi ;;
+    esac
+
     case "$txt" in
       *'grep -P'*|*'grep -qP'*|*'grep -oP'*)
         report "$f" "$ln" "grep -P is GNU-only" \