feat(docs-hygiene): add a grounding rule to /docs-hygiene:write-for-humans

Outcome

/docs-hygiene:write-for-humans grounds each concept before a sentence relies on it. The #12025 probe showed the gap: with the skill loaded and fired, the wfh-grounding-readme case failed the terms-grounded grader in 3 of 3 runs, the same as without the plugin (Claude Code 2.1.316).

Done when

The skill states the rule: introduce each concept before a sentence relies on it, or name it as knowledge the reader is assumed to have.
wfh-grounding-readme grades grounding deterministically (test(docs-hygiene): grade wfh-grounding-readme grounding with a deterministic regex #12941), and the measured with-plugin rate on main's rule and on each preregistered lever is recorded here with run-validity.py VALID; wfh-grounding-howto still passes. (Owner decision 2026-11-03 ~21:4xZ, option A, replacing "with-plugin rate at least 0.6 and at least 0.3 above without": four preregistered levers stayed at 0-3 of 10, identical text spread 0 to 3, and deleting the rule never lowered the rate beyond that spread, so the bar measured sampling noise, not the skill's wording. The README case stays in the suite to track the gap.)

Out of scope

Other write-for-humans criteria.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Small
Complexity: Low
Uncertainty: Knowable
Flags: none
Blocked by: none

Understood: #12098 added the grounding rule to the skill, so done-when 1 is met. Done-when 2 is still open. The latest probe shows wfh-grounding-readme failing terms-grounded 0/3 in the with-plugin arm, while the how-to case passes. The likely cause is that the rule only names project-specific terms, which is unverified. The remaining work is to widen the rule's scope, or require a definition at or before first use, then re-probe. The re-probe may need a run outside this lane. No likely duplicates found.

_{Automated triage, applied by the lanes App.}

No description provided.

The lanes App's pull request #12098 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

Probe on main f7dde4f (Claude Code 2.1.316, VALID): the rule from #12098 is in the skill (SKILL.md:51), but wfh-grounding-readme still fails terms-grounded 0/3 in the with-plugin arm; how-to passes 3/3. Every with-arm README leaves collector undefined, and uses checkpoint one sentence before defining it. The likely cause is that the rule names only "project-specific" terms (UNVERIFIED). Next step: widen the rule's scope, or require the definition at or before first use, then re-probe. Done-when 1 is met; done-when 2 is not.

The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

Not admitted to the issue lane: merged pull request #12098 names it with Refs: after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is **Decision recorded.** saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

Claimed by a Backlog HITL host-run worker: changing the grounding rule in /docs-hygiene:write-for-humans so a term is defined at or before its first use, then re-probing wfh-grounding-readme. Branch host-run/12085-wfh-grounding-first-use. Time box 90 min.

Host run. Reworded the grounding rule's test in /docs-hygiene:write-for-humans and re-probed the wfh-grounding-* eval twice. Verdict: improved, not done. The README with-plugin arm went from 0/3 to 2/3 in both probes, and the done-when needs 3/3.

What changed (branch host-run/12085-wfh-grounding-first-use)

plugins/docs-hygiene/skills/write-for-humans/SKILL.md:51: the rule title is unchanged, so it still matches done-when 1 word for word. The test column now says three things:
- read as someone new to the system;
- a common word that names one specific thing ("the broker", "the vault") counts as a term to introduce;
- the definition goes in the first sentence that uses the term or an earlier one, never the next, because notes often define a term after using it.
The example column gains a negative case: "Run stash export … A vault is a folder…" puts the definition one sentence late. The example words avoid the eval's own terms.
.changes/docs-hygiene/host-run-12085-wfh-grounding-first-use-5d81c3a2.md, bump: patch.
The stumble evidence is in the commit messages, not the skill body. It is the test(docs-hygiene): run the write-for-humans grounding eval and post its probe result #12025 probe on main f7dde4f0e4: VALID, with-arm 0/3. Every with-arm README left collector undefined and used checkpoint one sentence before defining it.

What ran

Claude Code 2.1.316 on the subscription login. Model claude-opus-5-5, pinned with --model (12 of 12 traces in each probe). The judge was the CLI default, 3 votes per verdict.
Each probe ran from a git archive of the committed head, extracted outside any git repository. Each was one suite in a capped claude.slice scope with a short on-disk TMPDIR. The running-suite count was 0 at each launch.
The command matches the test(docs-hygiene): run the write-for-humans grounding eval and post its probe result #12025 baseline: claude plugin eval <copy>/docs-hygiene --case 'wfh-grounding-*' --ablation with-without -j 2 --model claude-opus-5-5 --disallow-tools Agent Task Bash --max-cost-usd 20 --trust-plugin --no-publish --keep-temp, at each case's runs: 3.
Ceiling --max-cost-usd 20 per probe. The CLI reported $1.39 and $1.41 notional spend, with no partial result.
The case files and graders were not changed.

Scores (terms-grounded grader, passes over runs; n = 12 runs per probe)

Probe	Head	README with	README without	How-to with	How-to without	`run-validity.py`
Baseline (#12025, main `f7dde4f0e4`)	old rule	0/3	0/3	3/3	3/3	VALID
1	`2ffea86411` (reworded test)	2/3	0/3	3/3	3/3	VALID
2	`94a81df5da` (plus a post-draft first-use check)	2/3	1/3	3/3	3/3	VALID

Validity warnings in both probes: the judge model is not recorded; the how-to without-arm is at the 1.00 ceiling, so it is left out of the delta; and there are split votes. Probe 1 split on README with-arm run 3 (FAIL PASS PASS, a pass) and without-arm run 1. Probe 2 split on README with-arm run 1 (PASS FAIL FAIL, a fail) and without-arm run 1.
The skill fired in 6 of 6 with-arm runs in each probe.
Hand check of the failing with-arm READMEs. Both make the same slip in the opening paragraph:
- probe 1: "sends each new line to the collector. The collector is the central log service…";
- probe 2: "sends each new line to a collector. The collector is the log service…".
checkpoint was defined at or before first use in the probe 1 failure. In probe 2, a heading used checkpoint before the body defined it.
The post-draft check (probe 2) did not move the with-arm score. It was reverted in 225851286b, so the branch tree equals probe 1's head 2ffea86411. A standing line that does not change behavior does not ship.
I stopped after the one planned revision. Rerunning the same wording until a probe hits 3/3 would only select a lucky sample.

Done when (#12085)

Met: the skill states the rule. The title at SKILL.md:51 is unchanged ("Introduce each concept before a sentence relies on it, or name it as assumed knowledge"), and the test now requires the definition at or before first use.
Unmet: wfh-grounding-readme passes terms-grounded in the with-plugin arm in 2 of 3 runs, not 3 of 3 (probe 1, VALID; probe 2 repeats 2/3). wfh-grounding-howto still passes 3/3, and run-validity.py gave VALID.

Options for the remaining gap (decided from evidence, no owner decision needed)

Land the reworded test as is, with Refs: #12085. It moves the README with-arm from 0/3 to 2/3, and the how-to case does not regress. Recommended. Basis: probes 1 and 2 above.
Try a different lever next. Candidates are an opening-paragraph-specific example in the rule's example column, or a larger runs count to measure the rate more tightly than n=3. Either is a new probe, not a rerun of this wording.
Treat 2/3 as the model's floor for this case and revise done-when 2 to a rate threshold. That changes the issue's contract, so the coordinator or owner decides it.

The skill-creator case grounds-each-concept-before-relying-on-it (skills/write-for-humans/evals/evals.json) is not run by claude plugin eval and was not probed.

Branch: host-run/12085-wfh-grounding-first-use, from main fd0c36f751, head 225851286b. Commits: 2ffea86411 (reworded test), 94a81df5da (first-use check), 225851286b (revert of the check). scripts/check-affected.sh --tests gave 14 passed, 2 skipped on 2ffea86411 and on 94a81df5da. The head tree is identical to 2ffea86411. Pushed. No PR is open.

Target labels: add none; remove none.

Taken over by lane 1 (melo-lap-001, hub assignment 2026-11-02 16:5xZ); the 00:13Z Backlog HITL claim's 90-min box ended with the 02:45Z report. Plan: one new lever from option 2 there (an opening-paragraph example in the rule's example column), probed on wfh-grounding-* with the same command and n=3, VALID required. If README with-arm reaches 3/3 and how-to holds, ship with Closes; otherwise ship option 1 (the reworded test, 0/3 -> 2/3) with Refs: #12085 and record the gap.

Lane 1 probe of the next change (option 2 in the 02:45Z report): did not ship.

State going in. Option 1, the reworded test, is already on main through fix(docs-hygiene): define a term at or before its first use in write-for-humans #12199 (squash 8ad11e5). The probe branch started from main 57477a9. Main's SKILL.md, both prompts and both graders are identical to the 2/3 baseline tree 2ffea86411.
The change was one example in the rule's example column about the opening sentence: "beacon posts each build result to the relay, the bot that forwards messages to team chat", not "…to the relay. The relay is the bot…". It is commit 27648d0, reverted in 99a2413.
The probe ran once, preregistered before launch, with the same command as the 02:45Z report: Claude Code 2.1.316, claude-opus-5-5, --ablation with-without, runs 3. run-validity.py returned VALID.

Case	With plugin	Without plugin
wfh-grounding-readme	0/3 (baseline on main: 2/3)	1/3
wfh-grounding-howto	3/3	3/3

Hand check. All three failing with-plugin READMEs make the slip this example targets, in the opening paragraph: "sends every line to a collector. A collector is the HTTP service…". So the example did not move the behavior, and it does not ship.
Measured so far on the README case, with plugin, at n=3: 0/3 on the old rule, 2/3 and 2/3 on the reworded rule (on main now), 2/3 with a post-draft check that was reverted, and 0/3 with this example. The without-plugin arm ranged from 0/3 to 1/3. At n=3 these counts cannot separate one change from another, so done-when 2 ("3 of 3 runs") cannot be shown reliably.
Done when. Item 1 is met on main. Item 2 is unmet. Whether to change item 2 to a rate threshold measured at a larger n is an owner decision, and lane 1 has raised it there.

Lane 1: the revised done-when 2 was measured once on main 971e449 (preregistered, VALID, 10 runs, Claude Code 2.1.316, claude-opus-5-5). It is not met.

Case	With plugin	Without plugin
wfh-grounding-readme	5/10 (0.5)	2/10 (0.2)
wfh-grounding-howto	10/10	10/10

With-plugin rate of at least 0.6: fails (0.5).
Margin over without-plugin of at least 0.3: met (0.3).
How-to still passes: met.
run-validity.py: VALID.

What the run shows:

Judge noise. The judge split its votes on 5 runs. It passed runs 3 and 4 but failed run 6, although all three open with the same sentence shape ("…to a collector. The collector is the HTTP service…"). Two of the 5 passes were split votes. README judge agreement was 15/20, against 20/20 on how-to.
The skill's slips. Runs 2, 8 and 10 never define "collector", and run 5 defines it one sentence late.

Result: the rule now on main (#12199) lifts the README case by 0.3 over no plugin, but it does not reach 0.6. Two levers have been tried and neither moved the opening-sentence slip. Lane 1 releases its claim. Next lever for whoever picks this up: make the terms-grounded judge deterministic or better calibrated before trying another wording change, so that the rate can be measured at all. Data: 40 kept traces, on the lane 1 host.

Claimed by lane 3 (hub host-run order, oldest first). Plan: address the judge noise lane 1 measured on wfh-grounding-readme (README judge agreement 15/20) by making that grounding check deterministic where the rule allows it, or recalibrating its LLM grader; then re-measure the revised done-when 2 (10 runs, with/without, Opus, VALID) and ship or record the result.

Lane 3 result: revised done-when 2 is not met, under either grader.

Grader. The README case's terms-grounded LLM grader split votes on same-shape openings (lane 1: README judge agreement 15/20). It is now a deterministic regex that applies write-for-humans SKILL.md's own rule ("Its definition goes in that sentence or an earlier one, never the next") to the three planted terms (collector, checkpoint, tail set), with 24 samples (9 pass, 15 fail, including lane 1's openings) that validate-cases.py checks offline. The opening "…to a collector. The collector is…" grades FAIL under it, fixed in the preregistration before any run. This is stricter than the old judge, which passed some of those openings.

Measurement of record (preregistered, Claude Code 2.1.317, claude-opus-5-5 subject and judge, --ablation with-without --runs 10, run-validity.py VALID; how-to judge unanimous 20/20):

Case	With plugin	Without plugin
wfh-grounding-readme	1/10	0/10
wfh-grounding-howto	10/10	10/10

With-plugin rate 0.1 (bar 0.6) and margin 0.1 (bar 0.3) fail; how-to holds. Under the old judge the rate was 0.5 (lane 1, 19:30Z 11-02), also below the bar, so the verdict does not depend on the grader change.

Where the gap is. Per term, with plugin: checkpoint grounded in 8/10 runs (2/10 without), so the rule works there. Collector fails in 9/10, every time with the same opening one sentence late: "…sends each new line to a/the/your (log) collector. The collector is…". That is the opening-sentence slip the two earlier levers (#12199's reworded test, lane 1's example) also missed.

Next. The grader and samples ship in a PR with Refs: #12085 (it measures the rule, it does not meet done-when 2). The open lever is a skill change aimed at the opening sentence specifically, preregistered with the negative control rewrite-a-plugin.md requires. Spend this pass 13.55 USD list (calibration 0.83, a voided run 6.09 whose regex backtracked under the CLI's engine, the run of record 6.63).

Lane 3, third lever on the opening sentence: did not move the result; not shipped.

Lever. Row 51 of write-for-humans/SKILL.md (test and example cells): when a definition needs a sentence of its own, that sentence comes before the sentence that uses the term, including a product's opening sentence. Basis: in the run of record all 9 failing with-arm READMEs had a use-plus-gloss opening of 28-37 words, and the sentence-length rule (sentence-rules.md:62, split past about 25 words) pushes a same-sentence gloss into the next sentence; the two earlier levers both offered a same-sentence gloss.

Measurement (preregistered, Claude Code 2.1.317, claude-opus-5-5 subject and judge, 10 runs, VALID; deterministic grader from #12941):

Case / arm	Before	After	Control (row 51 deleted)
README, with plugin	1/10	1/10	0/10
README, without plugin	0/10	0/10	not run
How-to, with plugin	10/10	9/10	10/10

Per term with plugin, collector / checkpoint / tail set: before 1/8/10, after 2/7/10, control 0/7/10; without plugin 0/1/9.

Reading. Done-when 2 is not met (0.1 against 0.6). The control does not separate from the change (mean terms-grounded 0.50 each), so this suite cannot detect a loss from row 51's wording, and the change is unmeasurable under rewrite-a-plugin.md. Checkpoint's lift with the plugin (7-8/10 against 1/10 without) survives deleting row 51, so it comes from elsewhere in the skill. Three wording levers have now missed the same slip: "…sends each new line to a collector. The collector is…".

Next. One preregistered lever of a different kind: a post-draft check step in the skill that finds each term's first use and moves its definition into or before that sentence. If that also misses, the remaining options change done-when 2 and go to the owner. Spend this lever 10.51 USD list.

Lane 3, fourth lever (a post-draft check step): did not ship.

Step (one bullet under "After writing" in write-for-humans/SKILL.md, row 51 as on main): for each term a reader may not know, find the first sentence that uses it; if its definition comes later, move the definition into that sentence or one before it; if that would pass the length limit, put a short defining sentence first.

Measurement (preregistered, Claude Code 2.1.317, claude-opus-5-5 subject and judge, 10 runs each, all VALID; three suites launched together):

Suite	README with	README without	How-to with
After, with/without	0/10	0/10	10/10
After, with only	3/10	—	10/10
Control (step deleted = main), with only	1/10	—	10/10

Reading. Done-when 2 fails (0.0 and 0.3 against 0.6). The two suites running the same after text scored 0/10 and 3/10, so the 0.2 gap to the control is within the spread of identical text and does not count as separation. Across the four levers (#12199's test, lane 1's example, the row 51 opening rule, this step), the README with-plugin rate stayed between 0/10 and 3/10 under the deterministic grader, and deleting the rule never lowered it measurably. On this evidence the skill's wording does not control the README result (judgment); the opening-sentence slip ("…to a collector. The collector is…") persists at about the same rate with or without each change.

Spend across lane 3's passes on this issue: 13.55 + 10.51 + 14.89 = 38.95 USD list. The owner decides the next step: done-when 2 was set against the old judge, and every remaining path changes it.

Owner decision recorded. The owner chose option A on 2026-11-03 ("go w/ recommended"): reword done-when 2 to what the evidence supports and close. The rule is on main (#12199); wfh-grounding-readme grades grounding deterministically (#12941) and stays in the suite to track the gap. Record: issuecomment-6614703444 (run of record, 1/10 vs 0/10), issuecomment-6615599860 (row 51 opening rule, 1/10, control 0/10), issuecomment-6615766380 (post-draft check step, 0/10 and 3/10 on identical text, control 1/10); all VALID. Lane 3 spend 38.95 USD list.

Navigation Menu

Repository navigation

feat(docs-hygiene): add a grounding rule to /docs-hygiene:write-for-humans #12085

Outcome

Done when

Out of scope

2 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

feat(docs-hygiene): add a grounding rule to /docs-hygiene:write-for-humans #12085

Description

Outcome

Done when

Out of scope

Activity

github-actions commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

2 remaining items

melodic-automation-lanes commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions