diff --git a/.changes/harness-config/fix-10456-harness-config-conformance-a7d3e15c.md b/.changes/harness-config/fix-10456-harness-config-conformance-a7d3e15c.md new file mode 100644 index 00000000000..e2df9cc88a7 --- /dev/null +++ b/.changes/harness-config/fix-10456-harness-config-conformance-a7d3e15c.md @@ -0,0 +1,29 @@ +--- +bump: patch +--- + +### Fixed + +- `/harness-config:audit-permission-state`: the reader takes the project file from the primary + working directory rather than the repository root. Below the repository root, where it cannot tell a + session started there from one that ran a Bash `cd`, it also reads the root's file as + `project toplevel`, so a root-started session's deny rules survive a `cd`; run it from the session's + start directory, since after a `cd` from another subdirectory that directory's file is not read. The + output names both directories, + the lint checks each file's values, and `--merge-only` keeps the note that one may be over-reported. + Its cloud-session note also covers self-hosted runner images and project cloud threads. +- `/harness-config:audit`: the baseline drops the `Bash(git push)` ask row, which `Bash(git push *)` + already covers, so a project with the wider rule is no longer flagged for lacking the bare one. An + absent `effort:` no longer reads as the session level, the family alias record no longer rests on a + stale probe. The `fallbackModel` chain cap is read from model-config's Fallback model chains section, + where the page now states it, so the check no longer skips every chain. The effort pin check finds the defaults item by what it says, so a renumbered list no + longer hides a changed default, and its baseline matches the current page, with the listed levels + unchanged. +- `/harness-config:audit-instructions`: harness thinking-disable forms no longer fire the rejected + pairing row on models where they have no effect, exit-2 stderr is no longer stated as always + reaching Claude, and the first-run hold and a wrong settings pointer are gone. +- `/harness-config:audit-permission-grants` and the plugin README: the user settings file is no + longer named as where a prompt approval lands. +- `/harness-config:audit-pass`: the handoff no longer compares a `/doctor` version floor its pointer + has stopped stating, names `skillOverrides` as a documented suppression channel, and the report + location record points at the plugin data rules instead of quoting them. diff --git a/.claude/audit-pass.md b/.claude/audit-pass.md index fb95bf1e7e1..d7f20752217 100644 --- a/.claude/audit-pass.md +++ b/.claude/audit-pass.md @@ -15,12 +15,4 @@ suppressions: anchor/v1: "e:b40d4d51d2c2:6e340b9c" reason: "Cloud sessions on this repository push their working branch as the harness instructs; an ask-gate on git push would prompt on every run and is never answered. The GitHub proxy does not limit which branch a push updates (https://code.claude.com/docs/en/cloud-environments#github-proxy, Push restrictions, as of 2026-10-11; recheck when that bullet changes), so the default branch is held by the repository ruleset's pull-request rule, not by this gate. Local operators keep whatever ask rule they want in their user settings." date: 2026-09-08 - 52bccf5cafa7fd59: - check: harness-config/audit/B/baseline-ask-rules - claim: "missing-pattern:Bash(git push)" - sites: - - surface: .claude/settings.json - anchor/v1: "e:b40d4d51d2c2:6e340b9c" - reason: "Same posture as the argument-bearing form: the harness mandates the push in cloud sessions." - date: 2026-09-08 ``` diff --git a/docs/conventions/harness-config.yaml b/docs/conventions/harness-config.yaml index 343b5333154..c927a56946a 100644 --- a/docs/conventions/harness-config.yaml +++ b/docs/conventions/harness-config.yaml @@ -13,11 +13,3 @@ suppressions: "anchor/v1": "e:b40d4d51d2c2:6e340b9c" reason: "Cloud sessions on this repository push their working branch as the harness instructs; an ask-gate on git push would prompt on every run and is never answered. The GitHub proxy does not limit which branch a push updates (https://code.claude.com/docs/en/cloud-environments#github-proxy, Push restrictions, as of 2026-10-11; recheck when that bullet changes), so the default branch is held by the repository ruleset's pull-request rule, not by this gate. Local operators keep whatever ask rule they want in their user settings." date: "2026-09-08" - 52bccf5cafa7fd59: - check: harness-config/audit/B/baseline-ask-rules - claim: "missing-pattern:Bash(git push)" - sites: - - surface: .claude/settings.json - "anchor/v1": "e:b40d4d51d2c2:6e340b9c" - reason: "Same posture as the argument-bearing form: the harness mandates the push in cloud sessions." - date: "2026-09-08" diff --git a/plugins/harness-config/README.md b/plugins/harness-config/README.md index 85fc11ca49c..7f8b3a213a7 100644 --- a/plugins/harness-config/README.md +++ b/plugins/harness-config/README.md @@ -79,10 +79,11 @@ make a grant silently do nothing: interpreter-wildcard / blanket rules that Clau entering auto mode, hardcoded absolute machine/user paths (Bash rules match literally, no expansion), and inert plugin self-grants. A deterministic detector scans skill/command/agent frontmatter `allowed-tools` and the project, local, and user-global `permissions.allow` arrays, and recommends the -bare-command-on-PATH pattern. The user-global file -(`${CLAUDE_CONFIG_DIR:-$HOME/.claude}/settings.json`) is where Claude Code's own "Always allow" path -writes, so on a long-lived machine expect it to carry most of the findings. Their remediation is -the operator's, since no skill can write that file. The principle and citations live in the marketplace +bare-command-on-PATH pattern. A broad rule in the user-global file +(`${CLAUDE_CONFIG_DIR:-$HOME/.claude}/settings.json`) reaches every project. The remediation of a +finding there is the operator's, since no skill can write that file. Where a prompt approval is +saved is in [permissions: permission system](https://code.claude.com/docs/en/permissions#permission-system) +(as of 2026-11-04; recheck when that section changes the file). The principle and citations live in the marketplace [permission-rule-hygiene convention](../../docs/contracts/permission-rule-hygiene/README.md). Report-only. diff --git a/plugins/harness-config/skills/audit-automation-gaps/context/gotchas.md b/plugins/harness-config/skills/audit-automation-gaps/context/gotchas.md index c198c30ec4f..7500fb1bae8 100644 --- a/plugins/harness-config/skills/audit-automation-gaps/context/gotchas.md +++ b/plugins/harness-config/skills/audit-automation-gaps/context/gotchas.md @@ -84,7 +84,7 @@ current the claim is, not authority. | Claim | Basis | As of | Recheck trigger | |---|---|---|---| -| `PostToolUse` cannot block: the tool has already run, and exit code 2 shows the hook's stderr to Claude as a system reminder | [Claude Code hooks reference](https://code.claude.com/docs/en/hooks), exit-code-2 behavior per event | 2026-09-13 | A re-fetch finds the per-event exit-code table no longer matching this row | +| `PostToolUse` cannot block: the tool has already run, and exit code 2 only shows the hook's stderr to Claude | [Claude Code hooks reference](https://code.claude.com/docs/en/hooks#exit-code-2-behavior-per-event), exit-code-2 behavior per event | 2026-11-04 | A re-fetch finds the per-event exit-code table no longer matching this row | | `PermissionRequest` does not honor exit code 2: "the permission flow proceeds unchanged. Deny through the `decision` object instead" | [Claude Code hooks reference](https://code.claude.com/docs/en/hooks), `PermissionRequest` exit-code behavior | 2026-09-13 | A re-fetch finds `PermissionRequest` honoring exit code 2, or the `decision` object renamed or removed | | The hooks reference documents 33 hook events | [Claude Code hooks reference](https://code.claude.com/docs/en/hooks), the hook-events section headings, counted 2026-09-13 | 2026-09-13 | A re-fetch returns a different number of documented events | | We leave hook enumeration to the native `/hooks` menu, which lists the resolved set with each hook's source; when you need its exact contents, read the section live | [The /hooks menu](https://code.claude.com/docs/en/hooks#the-/hooks-menu) | 2026-10-11 | That section stops showing each hook's source, or the menu stops listing the resolved set of configured hooks | diff --git a/plugins/harness-config/skills/audit-instructions/reference/conflict-criteria.md b/plugins/harness-config/skills/audit-instructions/reference/conflict-criteria.md index 530c12ae6c4..7d8cf887afd 100644 --- a/plugins/harness-config/skills/audit-instructions/reference/conflict-criteria.md +++ b/plugins/harness-config/skills/audit-instructions/reference/conflict-criteria.md @@ -141,9 +141,9 @@ shapes without this gate produces noise, because most surface pairs never co-loa | User `CLAUDE.md` | Every session, unless the file is over the size limit the second pointer names, which skips it | [memory: how CLAUDE.md files load](https://code.claude.com/docs/en/memory#how-claude-md-files-load); [memory: how it works](https://code.claude.com/docs/en/memory#how-it-works) | | Project `CLAUDE.md` / `CLAUDE.local.md` | Every session in that tree, after user scope, under the same size limit | [memory: how CLAUDE.md files load](https://code.claude.com/docs/en/memory#how-claude-md-files-load) | | Project `AGENTS.md` / `.claude/AGENTS.md`, where no `CLAUDE.md` name displaces it, **and, under `claude-md-and-agents-md`, even where one does**, in both cases only where the session reads `AGENTS.md` natively at all | Every session in that tree | [memory: AGENTS.md](https://code.claude.com/docs/en/memory#agents-md); [memory: choose which instruction files load](https://code.claude.com/docs/en/memory#choose-which-instruction-files-load) | -| Nested `CLAUDE.md` in a subdirectory | On demand, when Claude works with a file there | [memory: how CLAUDE.md files load](https://code.claude.com/docs/en/memory#how-claude-md-files-load) | +| Nested `CLAUDE.md` in a subdirectory | On demand, once Claude works with a file there | [memory: how CLAUDE.md files load](https://code.claude.com/docs/en/memory#how-claude-md-files-load) | | `.claude/rules/*` without `paths` | Every session | [memory: set up rules](https://code.claude.com/docs/en/memory#set-up-rules) | -| `.claude/rules/*` with `paths` | Only when a matching file is read | [memory: path-specific rules](https://code.claude.com/docs/en/memory#path-specific-rules) | +| `.claude/rules/*` with `paths` | Only once Claude works with a matching file; read the triggers from the pointer | [memory: path-specific rules](https://code.claude.com/docs/en/memory#path-specific-rules) | | Skill body | Only once invoked, then for the rest of the session | [skills: skill content lifecycle](https://code.claude.com/docs/en/skills#skill-content-lifecycle) | | Auto memory `MEMORY.md` | Every **main** session, only its loaded portion, and **not** in a subagent, except a fork | [memory: how it works](https://code.claude.com/docs/en/memory#how-it-works) | | A subagent's **own** auto memory `MEMORY.md` (its `memory` field) | Every dispatch of **that** subagent, only its loaded portion, never the main session, and never another agent's | [subagents: enable persistent memory](https://code.claude.com/docs/en/sub-agents#enable-persistent-memory) | @@ -162,7 +162,7 @@ preload limit from those sections when it runs. This table copies none of them, moved without an edit here. - **Pointer**: when deciding whether a surface is resident, fetch the section its row names live. -- **As of**: 2026-10-11 +- **As of**: 2026-11-04 - **Recheck trigger**: a row's section is renamed or removed, or it adds or drops a condition on when that surface loads. @@ -306,13 +306,12 @@ Three consequences for residency, and each one bounds a pair rather than admitti [hooks: SessionStart](https://code.claude.com/docs/en/hooks#sessionstart) live (as of 2026-10-11; recheck when its matcher values change). - **Exit-2 stderr is turn-scoped error feedback, not a standing directive, and only some events - have an act to block.** Exit-2 stderr reaches Claude as an error message - ([hooks: exit code 2](https://code.claude.com/docs/en/hooks#exit-code-2), as of 2026-10-11; - recheck when that section stops feeding exit-2 stderr back to Claude), so it is not nothing; - but it is a one-turn message, never a standing rule. Whether it also carries a *gate* is - event-specific, and the sole authority on that is the hooks page's + have an act to block.** Where exit-2 stderr goes varies by event, so read whether Claude, the user + or nobody sees it from the table below; at most it is a one-turn message, never a standing rule. + Whether it also carries a *gate* is event-specific too, and the sole authority on both is the hooks + page's [Exit code 2 behavior per event](https://code.claude.com/docs/en/hooks#exit-code-2-behavior-per-event) - table (as of 2026-10-11; recheck when the table is renamed, removed or restructured, or its + table (as of 2026-11-04; recheck when the table is renamed, removed or restructured, or its `SubagentStop`, `PostToolUse` or `PreToolUse` row changes, the three cited below). Its rows are not reproduced here in either direction: the event set grows, so any list copied into this file becomes a closed partition that silently misgrades the next event added. diff --git a/plugins/harness-config/skills/audit-instructions/reference/criteria.md b/plugins/harness-config/skills/audit-instructions/reference/criteria.md index bb7a9b1454f..904fa8a264c 100644 --- a/plugins/harness-config/skills/audit-instructions/reference/criteria.md +++ b/plugins/harness-config/skills/audit-instructions/reference/criteria.md @@ -255,10 +255,11 @@ each only on the surfaces its row names. and where thinking may not be disabled): - Model configuration (the harness-side thinking-display and thinking-disable surfaces, which effort - levels each surface accepts, the per-model calibration of the effort scale, the first-run - default hold, and the adaptive-reasoning / fixed-thinking-budget partition): + levels each surface accepts, the per-model calibration of the effort scale, and the + adaptive-reasoning / fixed-thinking-budget partition): -- Settings (the `effortLevel` value set): +- Settings reference (the `effortLevel` value set): + - Environment variables (`CLAUDE_CODE_EFFORT_LEVEL`, `MAX_THINKING_TOKENS`, and `CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING` with the models it reaches): ; read it whole per the @@ -1169,15 +1170,15 @@ since each prescribes a request the model refuses. `Alt+T` / `Option+T` session toggle, or API `thinking: {"type": "disabled"}`, together with `xhigh` or `max` effort, on Claude Opus 5 or a later model, or, in the API form only, on Claude Haiku 5.5 (see the `haiku-5-5` widening below). Both operands are configuration - literals. Fire on the API form and on the harness forms alike: in neither does the prescribed - level reach the model, so the finding and its remediation are the same for both. What Claude - Code sends in place of the prescribed level is behind the pointer. - Pointer: for the harness outcome, see + literals. Fire on the API form and, on a model where thinking can be turned off, on the harness + forms alike: the finding and its remediation are the same for both. On a model in I17-a's no-effect + set the harness forms change nothing, so this arm does not fire on them there. + Pointer: for the harness outcome and that set, see [model configuration: extended thinking](https://code.claude.com/docs/en/model-config#extended-thinking) and [errors: effort isn't available with thinking turned off](https://code.claude.com/docs/en/errors#effort-isnt-available-with-thinking-turned-off). - As of: 2026-10-01. Recheck trigger: either section changes what Claude Code sends when thinking - is off and the prescribed level is `xhigh` or `max`. + As of: 2026-11-04. Recheck trigger: either section changes what Claude Code does when thinking + is off and the prescribed level is `xhigh` or `max`, or changes which models accept thinking off. - **Effort literals do not all reach every surface, and the literal set is not the whole set.** Count `max` wherever text sets it through `CLAUDE_CODE_EFFORT_LEVEL`, `--effort`, `/effort`, or skill and subagent `effort` frontmatter, the frontmatter case being a surface this skill already @@ -1203,7 +1204,7 @@ since each prescribes a request the model refuses. **Only the API form belongs to this arm.** On **the models in I17-a's no-effect set** the harness thinking-disable surfaces fail differently, and that failure is I17-a's, not this row's: we treat thinking as unable to be turned off there, and the session toggle, - `alwaysThinkingEnabled` and `MAX_THINKING_TOKENS=0` as silent no-ops rather than errors. **For + `alwaysThinkingEnabled` and `MAX_THINKING_TOKENS=0` as having no effect rather than as errors. **For the Mythos models the harness pages state nothing**, so this row makes no claim about their harness surfaces in either direction; the API reject is the whole of what is stated for them. Pointer: for the per-model accepted values, see [thinking: configuring @@ -1212,7 +1213,7 @@ since each prescribes a request the model refuses. table](https://platform.claude.com/docs/en/build-with-claude/thinking-troubleshooting#supported-models); for the harness no-ops, see [model configuration: extended thinking](https://code.claude.com/docs/en/model-config#extended-thinking). - As of: 2026-10-01. Recheck trigger: a model gains or loses a 400 for `"disabled"` in either + As of: 2026-11-04. Recheck trigger: a model gains or loses a 400 for `"disabled"` in either table, or the harness no-op list changes. - **Third arm, API requests only: `between_tools` at a level or with a change it refuses.** Flag instruction text that prescribes, or a code sample it tells the reader to send, a request to diff --git a/plugins/harness-config/skills/audit-pass/reference/doctor-handoff.md b/plugins/harness-config/skills/audit-pass/reference/doctor-handoff.md index 49a91786afd..66332500c21 100644 --- a/plugins/harness-config/skills/audit-pass/reference/doctor-handoff.md +++ b/plugins/harness-config/skills/audit-pass/reference/doctor-handoff.md @@ -42,16 +42,14 @@ Checking a version number alone is not a presence check. The pass reports **whic failed rather than a bare "unavailable", and it distinguishes what official documentation confirms from what it does not. -1. **Version floor.** The `CLAUDE.md` trim check this pass hands off has a version floor, and - `/doctor`'s shape changed across releases before it. Compare the installed version with the - floor the pointer states; below it, the capability this pass hands off is not the capability - that exists, and a pass that assumes the current shape on an older install is checking for the - wrong thing. +1. **Version floor.** We state no version floor for the `CLAUDE.md` trim check this pass hands off: + the pointer gives none today, so the pass reports the installed version without grading it. If + the pointer states a floor again, compare the installed version with it. - **Pointer**: when checking the floor, fetch [memory: my CLAUDE.md is too large](https://code.claude.com/docs/en/memory#my-claude-md-is-too-large) live. - - **As of**: 2026-10-11 - - **Recheck trigger**: that section moves, raises or drops the floor. + - **As of**: 2026-11-04 + - **Recheck trigger**: that section states a floor, or moves. 2. **`DISABLE_DOCTOR_COMMAND`: documented.** The variable's row is on the environment-variables page again, so it is a documented suppression channel, and its scope is the row's: read which surface it hides, and which it leaves alone, from the row rather than from here. @@ -61,28 +59,27 @@ from what it does not. - **Recheck trigger**: the row leaves the table, as it did between earlier fetches, or changes which surface it hides. -**Suppression channels: one documented, one documented but silent on bundled skills.** -`skillOverrides` is a documented settings key that reaches some skill sources and not others. The -skills page names which, and does not say whether it reaches a bundled skill such as `/doctor`, so -its effect on the handoff is UNVERIFIED; probe rather than assert. +**Suppression channels.** When the handoff is missing, we also check a `skillOverrides` entry for +`/doctor`, and name it as a suspected cause only when the pages below say an override reaches a +bundled skill. -- **Pointer**: when naming `skillOverrides` as a cause, fetch +- **Pointer**: before naming `skillOverrides` as a cause, fetch + [skills: bundled skills](https://code.claude.com/docs/en/skills#bundled-skills) and [skills: override skill visibility from settings](https://code.claude.com/docs/en/skills#override-skill-visibility-from-settings) live. -- **As of**: 2026-10-11 -- **Recheck trigger**: that section states whether an override reaches a bundled skill. +- **As of**: 2026-11-04 +- **Recheck trigger**: either section stops letting an override hide a bundled skill. So the pass **detects absence rather than predicting it**: it checks whether `/doctor` actually -resolves in this environment, and reports the outcome. If it does not resolve while the version floor -is met, the run says so and names these channels as the suspected causes, `DISABLE_DOCTOR_COMMAND` -and a `skillOverrides` entry naming the skill (documented for some skill sources, unverified for -bundled ones), rather than asserting either as the reason. Detecting beats predicting either way: the variable's presence in the list says an operator +resolves in this environment, and reports the outcome. If it does not resolve, the run says so and +names these channels as the suspected causes, `DISABLE_DOCTOR_COMMAND` and a `skillOverrides` entry +naming the skill, rather than asserting either as the reason. Detecting beats predicting either way: the variable's presence in the list says an operator *could* have set it, never that they did. **Recheck trigger:** any Claude Code minor release, or any change to how bundled skills are suppressed. Re-verify against current official documentation before -treating a failed prerequisite as authoritative. A stale floor would report a present capability as -missing, and the entry corrected above shows a stale *absence* claim reads exactly like a fresh one. +treating a failed prerequisite as authoritative. A stale *absence* claim reads exactly like a fresh +one. ## Absence classification: optional capability diff --git a/plugins/harness-config/skills/audit-pass/reference/report-location-and-schema.md b/plugins/harness-config/skills/audit-pass/reference/report-location-and-schema.md index ca85f95a1a2..99e665956c5 100644 --- a/plugins/harness-config/skills/audit-pass/reference/report-location-and-schema.md +++ b/plugins/harness-config/skills/audit-pass/reference/report-location-and-schema.md @@ -37,19 +37,22 @@ while excluding its `findings.json`, which is the same unfalsifiable-idempotence ([plugin manifest reference](https://code.claude.com/docs/en/plugins/manifest-reference#environment-variables), verified 2026-10-07; recheck when that section names a setting that relocates it), and no documented setting relocates it. - **`{id}` is derived, and deriving it wrong loses the report.** Same page, verbatim: `{id}` is *"the - plugin identifier with characters outside `a-z`, `A-Z`, `0-9`, `_`, and `-` replaced by `-`"*, with - the worked example that a plugin installed as `formatter@my-marketplace` lands in - `~/.claude/plugins/data/formatter-my-marketplace/`, because the `@` becomes `-`. A wrong derivation writes - the report where the next run will not look for it, which is also how `--resume` loses a partial. - - **`${CLAUDE_PLUGIN_DATA}` is not in the Bash tool's environment. Do not try to expand it from a - shell.** The same page scopes the export precisely: *"All three are exported as environment variables - to hook processes and to MCP and LSP server subprocesses."* The Bash tool is none of those. The token - does substitute in **skill content**, which is how a resolved path reaches you in this file, but - `echo "$CLAUDE_PLUGIN_DATA"` inside a Bash call yields an empty string. Use the path already - substituted into the text you are reading, or rebuild it from `~/.claude/plugins/data/` plus the - mangled identifier above. + **`{id}` is derived, and deriving it wrong loses the report.** Take the character rule from the + pointer below rather than from memory. A wrong derivation writes the report where the next run + will not look for it, which is also how `--resume` loses a partial. + + **Do not expand `${CLAUDE_PLUGIN_DATA}` from a shell.** We treat it as absent from the Bash tool's + environment and as substituted in **skill content**, which is how a resolved path reaches you in + this file. Use the path already substituted into the text you are reading, or rebuild it from + `~/.claude/plugins/data/` plus the identifier the pointer's rule produces. + + - **Pointer**: when deriving `{id}`, or deciding where the token resolves, fetch + [plugin manifest reference: environment variables](https://code.claude.com/docs/en/plugins/manifest-reference#environment-variables) + and [where each variable resolves](https://code.claude.com/docs/en/plugins/manifest-reference#where-each-variable-resolves) + live. + - **As of**: 2026-11-04 + - **Recheck trigger**: either section changes the character rule, or exports the variable to the + Bash tool. It is therefore **outside** a target below `~` and **inside** any target at or above it. The default path is *usually* outside the scan set and is **not diff --git a/plugins/harness-config/skills/audit-permission-grants/SKILL.md b/plugins/harness-config/skills/audit-permission-grants/SKILL.md index 94cf6a47f3a..65c12b0988a 100644 --- a/plugins/harness-config/skills/audit-permission-grants/SKILL.md +++ b/plugins/harness-config/skills/audit-permission-grants/SKILL.md @@ -108,11 +108,12 @@ numbers into the report: for that version boundary; this skill's silence about them is a boundary, not a result. A user-global finding is reported the same as any other, but its remediation is the operator's: a -skill cannot write that file. When deciding where an approval-prompt grant lands and whether -anything prunes it, fetch +skill cannot write that file. When deciding where an approval-prompt grant lands, fetch +[permissions: permission system](https://code.claude.com/docs/en/permissions#permission-system) +live; when deciding whether anything prunes it, fetch [permissions: manage permissions](https://code.claude.com/docs/en/permissions#manage-permissions) -live (as of 2026-11-03; recheck when that section changes where a prompt approval is saved or how -saved rules are removed). +live (as of 2026-11-04; recheck when the first section changes where a prompt approval is saved or +the second changes how saved rules are removed). If a scope filter was given, run the full detector and present only the matching checks. P1, P2, P2b and P4 are all emitted by the same rule scan, which runs over frontmatter `allowed-tools` and over diff --git a/plugins/harness-config/skills/audit-permission-grants/scripts/permission-rule-check.sh b/plugins/harness-config/skills/audit-permission-grants/scripts/permission-rule-check.sh index fa5b6396634..a5b5a1fad03 100755 --- a/plugins/harness-config/skills/audit-permission-grants/scripts/permission-rule-check.sh +++ b/plugins/harness-config/skills/audit-permission-grants/scripts/permission-rule-check.sh @@ -695,9 +695,12 @@ scan_settings_allow() { scan_settings_allow "$ROOT/.claude/settings.json" ".claude/settings.json permissions.allow" "project" scan_settings_allow "$ROOT/.claude/settings.local.json" ".claude/settings.local.json permissions.allow" "local" -# User-global scope. A project-only scan cannot see it, yet it is where Claude -# Code's own "Always allow" path writes, so it is the scope most likely to -# accumulate the broad rules auto mode drops. CLAUDE_CONFIG_DIR relocates the +# User-global scope. A project-only scan cannot see it, and a broad rule here +# reaches every project, so it is scanned for the rules auto mode drops. For where +# a prompt approval is saved, see +# https://code.claude.com/docs/en/settings#where-claude-code-keeps-the-local-file-in-a-git-repository +# (as of 2026-11-04; recheck when that section changes where approvals are saved). +# CLAUDE_CONFIG_DIR relocates the # whole ~/.claude tree when set (official .claude-directory doc), so it wins over # $HOME; with neither resolvable there is no user scope to scan. The label is the # resolved absolute path — reporting "~/.claude/settings.json" would name the diff --git a/plugins/harness-config/skills/audit-permission-state/SKILL.md b/plugins/harness-config/skills/audit-permission-state/SKILL.md index ef1c744ac84..aef3cf7a0d4 100644 --- a/plugins/harness-config/skills/audit-permission-state/SKILL.md +++ b/plugins/harness-config/skills/audit-permission-state/SKILL.md @@ -129,7 +129,7 @@ NOTE: anything the operator must know | Field | Values | | --- | --- | | `scope` | `managed`, `user`, `project`, `local`, `startdir-local` | -| `surface` | `file`, `dropin-dir`, `dropin-file:`, `registry`, `plist` (managed); `settings` elsewhere | +| `surface` | `file`, `dropin-dir`, `dropin-file:`, `registry`, `plist` (managed); `toplevel` (project, see Scopes); `settings` elsewhere | | `status` | `present`, `absent`, `unreadable`, `invalid-json`, `skipped`, `not-applicable` | | `kind` | `allow`, `ask`, `deny` | @@ -353,7 +353,18 @@ collapse it in the report: Five: `managed`, `user`, `project`, `local` and `startdir-local`. `local` and `startdir-local` are the two easy to get wrong; when deciding where the local file is read from, fetch [settings: where Claude Code keeps the local file in a git repository](https://code.claude.com/docs/en/settings#where-claude-code-keeps-the-local-file-in-a-git-repository) -live (as of 2026-11-03; recheck when that section changes where the local file is kept). `managed` +live (as of 2026-11-04; recheck when that section changes where the local or shared file is kept). +The reader takes its own working directory as the session's primary working directory, the one input +for `project` and `local`. Run it from the directory the session started in (or last moved to with +`/cd`). Below the repository root it reads two shared files, `project settings` from its own +directory and `project toplevel` from the root, and no other. So after an in-session Bash `cd`, the +start directory's file is not read unless it is the root's: a session started in `repo/a` that ran +`cd b` gets `repo/a/b` and the root, never `repo/a`. Both files it reads feed the merge, and the set +can over-report the one not in force. Say which directory the session started in when that decides +the answer. When judging whether a `cd` still carries over, fetch +[tools reference: what persists between commands](https://code.claude.com/docs/en/tools-reference#what-persists-between-commands) +live (as of 2026-11-04; recheck when that section changes what a `cd` does to the working directory). +`managed` is four admin surfaces per OS, not one file, plus a `remote-cache` record for the server-managed cache that is not folded into the effective set. `reference/criteria.md` §Scopes has the full table and the dated record. diff --git a/plugins/harness-config/skills/audit-permission-state/reference/criteria.md b/plugins/harness-config/skills/audit-permission-state/reference/criteria.md index ea5f98627e6..b099b420f1a 100644 --- a/plugins/harness-config/skills/audit-permission-state/reference/criteria.md +++ b/plugins/harness-config/skills/audit-permission-state/reference/criteria.md @@ -36,8 +36,8 @@ permissions page. Recheck when either page's precedence list adds, drops or reor | Scope | Why it is its own member | | --- | --- | | `managed` | Highest precedence. Four surfaces per OS, not one file. See below | -| `user` | `${CLAUDE_CONFIG_DIR:-$HOME/.claude}/settings.json`. Where Claude Code's own "Always allow" path writes, so it accumulates the most rules | -| `project` | `.claude/settings.json` at the repository root | +| `user` | `${CLAUDE_CONFIG_DIR:-$HOME/.claude}/settings.json`. Applies in every project, so a broad rule here reaches furthest. The record below says where a prompt approval lands | +| `project` | `.claude/settings.json` in the session's primary working directory, which the reader takes as its own working directory. A session started in a subdirectory or a worktree reads that directory's file, not the repository root's. Below the root the reader also reports the root's file as the `toplevel` surface, because a Bash `cd` moves the reader and not the session and it cannot tell the two apart: one of the two may be over-reported, a start directory that is neither is not read, and the lint names the root file `project:toplevel` | | `local` | `.claude/settings.local.json`, resolved **through worktrees to the main checkout**, so anchoring on the worktree root looks where the file is not. Four documented conditions keep it beside `settings.json` instead, and the reader resolves all four: no enclosing git repository, a repository root equal to `$HOME`, a Windows host, and a repository root, `.git` or `.claude` entry owned by another user | | `startdir-local` | A pre-v2.1.211 copy left in the session's start directory. Not a fallback: when both exist the repository root wins on a shared key, **but permission rules from both stay in effect**, so both are live | @@ -53,6 +53,19 @@ It treats the Agent SDK's `resolveSettings()` helper as reading the start-direct - **Recheck trigger**: that section names a different version boundary or drops the start-directory case, or a release note names where `settings.local.json` is read from. +The reader treats the primary working directory as the one input to both project files: it reads the +shared file there and resolves the local file's repository root from there. It cannot see a session's +`/cd` or a Bash `cd`, so it names the directory it used in a note, and an audit run from somewhere +else reports that directory's file. + +- **Pointer**: when deciding where the shared file is read from or where a prompt approval lands, + fetch the same section and + [permissions: permission system](https://code.claude.com/docs/en/permissions#permission-system) + live. +- **As of**: 2026-11-04 +- **Recheck trigger**: that section moves the shared file away from the primary working directory, + or the permissions section changes which file a prompt approval is saved to. + ### The four start-directory conditions, and the one that is not detectable The four conditions in the Scopes table's `local` row are the ones the same section documents. All @@ -73,17 +86,19 @@ than guessing at it. ### Cloud sessions read a different scope set The reader treats user settings and project local settings (`~/.claude/settings.json` and -`.claude/settings.local.json`) as not read in a cloud session. Of the managed sources, the reader -counts only server-managed settings there; device files and MDM profiles stay on the operator's -machine. So a user-scope record in a cloud session describes the container's file, never the -operator's, and reporting it without that framing invites the wrong conclusion. +`.claude/settings.local.json`) as not read in a cloud session, and device files and MDM profiles as +staying on the operator's machine. So a user-scope record in a cloud session describes the +container's file, never the operator's, and reporting it without that framing invites the wrong +conclusion. The reader cannot tell which kind of cloud session it is in, so its note also sends the +operator to the pointer for two cases: the managed file a self-hosted runner image can carry, and +the project cloud threads that take no permission rules from a repository's shared file. The reader detects a cloud session only by `CLAUDE_CODE_REMOTE` set to `true`, the only entrypoint variable it branches on. `CLAUDE_CODE_ENTRYPOINT` is not documented and is never read. - **Pointer**: [Settings in cloud sessions](https://code.claude.com/docs/en/settings#settings-in-cloud-sessions) and the `CLAUDE_CODE_REMOTE` row of [Environment variables](https://code.claude.com/docs/en/env-vars). -- **As of**: 2026-09-12 +- **As of**: 2026-11-04 - **Recheck trigger**: the cloud-session scope set changes or the variable is documented differently. @@ -344,7 +359,7 @@ fixed all three. | `C6-contentField` | Fires on a `Tool(param:value)` rule naming the tool's primary content field. The fields the lint checks, by tool: Bash and PowerShell `command`; Read, Edit, Write `file_path`; Grep, Glob `path`; NotebookEdit `notebook_path`; WebFetch `url`. We treat such a rule as ignored with a startup warning. Pointer: [Match by input parameter](https://code.claude.com/docs/en/permissions#match-by-input-parameter) | | `C6-allowParam` | Fires on an **allow** rule in `Tool(param:value)` form, since we treat parameter matching as a deny-and-ask feature only; allow rules keep each tool's own specifier syntax. Pointer: [Match by input parameter](https://code.claude.com/docs/en/permissions#match-by-input-parameter). An operator writing one believes they narrowed a grant and has not. Fires only on parameters the page names for tools whose own syntax is a path or a command. `WebFetch(domain:host)` is the documented WebFetch form and `Bash(npm:*)` is a command prefix, so neither is distinguishable from a parameter by shape and neither fires | | `C6-uncoveredPath` | Fires on a path rule whose tool is `Write`, `NotebookEdit`, `Glob`, or the retired `MultiEdit`, since we treat file access as decided by `Read(path)` and `Edit(path)` rules alone; such a rule is accepted, never consulted, and warned about at startup (v2.1.210+; a `Glob` rule passed in `--allowedTools` is the stated exception). Pointer: [Read and Edit](https://code.claude.com/docs/en/permissions#read-and-edit). **As of** 2026-10-13. **Recheck when** that section stops naming `MultiEdit` or changes which tools path rules cover | -| `C6-colonStar` | Fires on a mid-pattern `:*`, as in `Bash(git:* push)`, since we treat `:*` as prefix syntax only at the end of a pattern: mid-pattern, the colon is a literal and the `*` a wildcard, so `Bash(git:* push)` matches `git: push` and none of the git commands its author meant. We saw no startup warning for such a rule in a `-p` run, so the lint reports it. **Pointer**: when the mid-pattern colon matters, fetch [Wildcard patterns](https://code.claude.com/docs/en/permissions#wildcard-patterns) live. **As of** 2026-10-10. **Recheck when** that section changes how a mid-pattern `:*` matches, or a release makes it prefix syntax. **Probe**: probed on Claude Code 2.1.296, 2026-10-10; evidence in the pull request that carries this change. The mechanic is about **command-prefix** patterns, so a documented parameter form is exempt: WebFetch's `domain:` prefix takes `*` wildcards ([WebFetch](https://code.claude.com/docs/en/permissions#webfetch)), and firing on `WebFetch(domain:*.example.com)` called a documented, working rule broken. A body that starts with `*` and holds no space, as in `PowerShell(*env:*lefthook*)`, is exempt too: it has no command prefix, so the literal colon is what it means to match. A space restores the prefix-then-words shape, so `Bash(* git:* push)` and `Bash(*foo:* rm)` still fire. **Known gap:** in a deny or ask rule a mid-pattern `:*` with NO space after it, as in `Bash(git:*push)`, is not reported. It is structurally identical to the parameter form `Agent(model:*-haiku)`, so once the space is gone nothing in the rule text distinguishes them; the space was the only signal. The documented example is the space form, and the pages show no no-space mid-pattern rule anywhere. The exemption is by **grammar**, where in a deny or ask rule an `identifier:value` body is the parameter form, not by a list of parameter names: we treat parameter matching as working for any scalar parameter on any tool ([Match by input parameter](https://code.claude.com/docs/en/permissions#match-by-input-parameter)), so an allowlist could only ever chase it and would flag documented forms such as `Agent(model:*-haiku)` | +| `C6-colonStar` | Fires on a mid-pattern `:*`, as in `Bash(git:* push)`, since we treat `:*` as prefix syntax only at the end of a pattern: mid-pattern, the colon is a literal and the `*` a wildcard, so `Bash(git:* push)` matches `git: push` and none of the git commands its author meant. We saw no startup warning for such a rule in a `-p` run, so the lint reports it. **Pointer**: when the mid-pattern colon matters, fetch [Wildcard patterns](https://code.claude.com/docs/en/permissions#wildcard-patterns) live. **As of** 2026-10-10. **Recheck when** that section changes how a mid-pattern `:*` matches, or a release makes it prefix syntax. **Probe**: probed on Claude Code 2.1.296, 2026-10-10; evidence in the pull request that carries this change. The mechanic is about **command-prefix** patterns, so a documented parameter form is exempt: WebFetch's `domain:` prefix takes `*` wildcards ([WebFetch](https://code.claude.com/docs/en/permissions#webfetch)), and firing on `WebFetch(domain:*.example.com)` called a documented, working rule broken. A body that starts with `*` and holds no space, as in `PowerShell(*env:*lefthook*)`, is exempt too: it has no command prefix, so the literal colon is what it means to match. A space restores the prefix-then-words shape, so `Bash(* git:* push)` and `Bash(*foo:* rm)` still fire. **Known gap:** in a deny or ask rule a mid-pattern `:*` with NO space after it, as in `Bash(git:*push)`, is not reported. It is structurally identical to the parameter form `Agent(model:*-haiku)`, so once the space is gone nothing in the rule text distinguishes them; the space was the only signal. The documented example is the space form, and the pages show no no-space mid-pattern rule anywhere. The exemption is by **grammar**, where in a deny or ask rule an `identifier:value` body is the parameter form, not by a list of parameter names: we treat parameter matching as working for any scalar parameter on any built-in tool, and an `mcp__` rule with parentheses in a settings file as a separate case this row does not judge ([Match by input parameter](https://code.claude.com/docs/en/permissions#match-by-input-parameter), as of 2026-11-04; recheck when that section changes which tools it covers or how settings files treat such a rule), so an allowlist could only ever chase it and would flag documented forms such as `Agent(model:*-haiku)` | | `C6-malformed` | Fires on text after a specifier's closing parenthesis, as in `Bash(ls) x`, and on an unclosed specifier. A parenthesis inside the specifier does not fire, so `Edit(./Finance (2024)/**)` is one rule. The lint used to keep only the token and treat `Bash(ls) x` as `Bash(ls)`. **Pointer**: when deciding which rule shapes are accepted and how trailing text is treated, fetch [permissions: permission rule syntax](https://code.claude.com/docs/en/permissions#permission-rule-syntax) live. **As of** 2026-11-03. **Recheck when** that section changes how parentheses in a specifier or text after one are read | | `C6-literalPath` | A Read or Edit path that is not a usable gitignore pattern, such as an unclosed `[`. As a deny or ask rule it protects only that literal path; as an allow rule it grants nothing. One such deny used to fail every file edit; that is not the current behavior | diff --git a/plugins/harness-config/skills/audit-permission-state/scripts/permission-merge.sh b/plugins/harness-config/skills/audit-permission-state/scripts/permission-merge.sh index 89bd2d32d89..5c64f08cdb4 100755 --- a/plugins/harness-config/skills/audit-permission-state/scripts/permission-merge.sh +++ b/plugins/harness-config/skills/audit-permission-state/scripts/permission-merge.sh @@ -62,7 +62,8 @@ Records: "effective scopes= precedence_basis= ", "carveout source=: [bare] ", and "CAVEAT: ". With --merge-only the reader's own NOTE records are dropped, including the one -stating where the server-managed settings cache lives. Read both sections when +stating where the server-managed settings cache lives, except the one saying the +set can over-report a project file. Read both sections when the question is what the machine's permission state actually is. Reads only. Exits 2 when the input carries no scope records at all. @@ -127,7 +128,10 @@ function emit_inert(ikind, itext, tag) { # diff reads autoMode.classifyAllShell from them, and without it a narrow shell # rule that auto mode actually suspends is reported as kept. Dropping them made # a documented flag combination silently invert the answer, with no warning. -{ if (passthrough || $1 == "conf") print } +# +# So does the reader note that it read two project files: the merged set can +# over-report one of them, and scopes=project does not say which. +{ if (passthrough || $1 == "conf" || ($1 == "NOTE:" && /can over-report/)) print } $1 == "rule" { kind = $4 diff --git a/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.sh b/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.sh index ce1dd288518..6c1dd34326d 100755 --- a/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.sh +++ b/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.sh @@ -16,6 +16,8 @@ # finding [] # lint summary findings= checks_run= # +# scope a reader scope, or project:toplevel for the root's shared file +# # severity error | warning (review plugin's severity vocabulary) # # Checks, each citing the mechanic it follows from — see reference/criteria.md: @@ -141,11 +143,16 @@ function specifier_body(text, i, n, depth, ch, start) { return substr(text, start + 1) } +# The name the file of a record goes by in findings. The reader may report two project +# files, and neither value replaces the other, so the root file is its own entry. +# A managed drop-in does replace an earlier value, so every other scope stays +# last-wins. +function scope_of(s, surface) { return (s == "project" && surface == "toplevel") ? "project:toplevel" : s } $1 == "rule" { - rules[++n_rules] = $2 SUBSEP $4 SUBSEP text_of(5) + rules[++n_rules] = scope_of($2, $3) SUBSEP $4 SUBSEP text_of(5) next } -$1 == "conf" { conf[$2 SUBSEP $4] = text_of(5); next } +$1 == "conf" { conf[scope_of($2, $3) SUBSEP $4] = text_of(5); next } $1 == "NOTE:" { next } NF >= 3 { n_surfaces++ @@ -179,11 +186,11 @@ END { # .claude/settings.json or .claude/settings.local.json." Before v2.1.207 it # also read local settings, so a local-scope section was live on an older # harness -- the finding says so rather than implying it never worked. - split("project local startdir-local", dead_automode, " ") + split("project project:toplevel local startdir-local", dead_automode, " ") for (i in dead_automode) { s = dead_automode[i] if ((s SUBSEP "autoModePresent") in conf) { - extra = (s == "project") ? "" : " (read from local settings before v2.1.207, so this may have been live on an older harness)" + extra = (s ~ /^project/) ? "" : " (read from local settings before v2.1.207, so this may have been live on an older harness)" finding("error", "C2-autoMode", s, "an autoMode section here is never read: the classifier reads autoMode from user settings, managed settings, and inline --settings/SDK JSON only" extra) } } @@ -207,8 +214,11 @@ END { # "Not read from shared project settings." That names .claude/settings.json # specifically, so a local-settings occurrence is NOT claimed dead here -- # claiming it would be asserting a restriction the page does not state. - if (("project" SUBSEP "useAutoModeDuringPlan") in conf) - finding("error", "C2-planMode", "project", "useAutoModeDuringPlan is not read from shared project settings — move it to user settings, where it takes effect") + split("project project:toplevel", shared_files, " ") + for (i in shared_files) { + if ((shared_files[i] SUBSEP "useAutoModeDuringPlan") in conf) + finding("error", "C2-planMode", shared_files[i], "useAutoModeDuringPlan is not read from shared project settings — move it to user settings, where it takes effect") + } # --- C5: disableAutoMode typed as a boolean -------------------------------- # @@ -217,7 +227,7 @@ END { # JSON and does nothing, which is the worst possible outcome for a lock-out # switch: the operator believes auto mode is disabled and it is not. Checked # at BOTH documented key paths, in EVERY scope: it is not managed-only. - split("managed user project local startdir-local", all_scopes, " ") + split("managed user project project:toplevel local startdir-local", all_scopes, " ") split("disableAutoMode permissions.disableAutoMode", disable_keys, " ") for (i in all_scopes) { for (j in disable_keys) { diff --git a/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.test.sh b/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.test.sh index f81731e3675..147b6af409b 100755 --- a/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.test.sh +++ b/plugins/harness-config/skills/audit-permission-state/scripts/permission-plane-lint.test.sh @@ -156,6 +156,44 @@ C5_MANAGED=$( OUT=$(lint "$C5_MANAGED") assert_contains "managed scope is checked like any other" "$OUT" "[C5-disableType] managed" +# The reader's two project files are two files: below the repository root it +# reports both, and a value in one must not hide the other's, in either order. +TWO_FILES=$( + printf '%s\n' "$SURFACES" + cat <<'EOF' +conf project settings disableAutoMode true +conf project toplevel disableAutoMode "disable" +conf project settings defaultMode "acceptEdits" +conf project toplevel defaultMode "auto" +conf project toplevel autoModePresent true +conf project toplevel useAutoModeDuringPlan false +EOF +) +OUT=$(lint "$TWO_FILES") +assert_contains "two project files: this directory's mistyped switch survives the root's" "$OUT" "[C5-disableType] project disableAutoMode is true" +assert_eq "two project files: the correctly typed switch does not fire" 1 "$(count_matching "$OUT" '\[C5-disableType\]')" +assert_contains "two project files: the root's ignored mode survives this directory's" "$OUT" "[C2-defaultMode] project:toplevel defaultMode:\"auto\"" +assert_contains "two project files: the root's dead autoMode is named as the root's" "$OUT" "[C2-autoMode] project:toplevel an autoMode section here is never read" +assert_not_contains "two project files: it carries no local-settings history" "$(grep -F '[C2-autoMode]' <<<"$OUT")" "before v2.1.207" +assert_contains "two project files: the root's plan key is named as the root's" "$OUT" "[C2-planMode] project:toplevel" +# A rule finding names the root file the same way. +ROOT_RULE=$( + printf '%s\n' "$SURFACES" + printf 'rule project toplevel allow Bash(git:* push)\n' + printf 'rule project settings allow Bash(npm:* test)\n' +) +OUT=$(lint "$ROOT_RULE") +assert_contains "two project files: a root rule finding is named as the root's" "$OUT" "[C6-colonStar] project:toplevel" +assert_contains "two project files: this directory's rule finding stays project" "$OUT" "[C6-colonStar] project " +assert_eq "two project files: one finding each" 2 "$(count_matching "$OUT" '\[C6-colonStar\]')" +# A later managed drop-in replaces an earlier value, so that scope stays last-wins. +LAST_WINS=$( + printf '%s\n' "$SURFACES" + printf 'conf managed file disableAutoMode true\n' + printf 'conf managed dropin-file:z.json disableAutoMode "disable"\n' +) +assert_not_contains "managed: a later drop-in's value is the one in force" "$(lint "$LAST_WINS")" "[C5-disableType]" + # --- C6: rules that cannot match --------------------------------------------- C6=$( printf '%s\n' "$SURFACES" diff --git a/plugins/harness-config/skills/audit-permission-state/scripts/permission-state-primary-dir.test.sh b/plugins/harness-config/skills/audit-permission-state/scripts/permission-state-primary-dir.test.sh new file mode 100755 index 00000000000..abc77a3720c --- /dev/null +++ b/plugins/harness-config/skills/audit-permission-state/scripts/permission-state-primary-dir.test.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# Where permission-state.sh looks for the two project files, run over real git +# trees with no fixture-root seam. Each expected location is the one the settings +# page gives: the shared file in the session's primary working directory +# (settings#where-claude-code-keeps-the-local-file-in-a-git-repository), the local +# file at the repository root, resolved through a worktree to the main checkout. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$SCRIPT_DIR/permission-state.sh" + +TEST_TMPDIR="$(mktemp -d)" +trap 'rm -rf "$TEST_TMPDIR"' EXIT + +# shellcheck source=test-helpers.sh +source "$SCRIPT_DIR/test-helpers.sh" + +# The local file stays beside the shared file on Windows; these cases are about +# the repository-root anchor, so they pin a non-Windows platform through the seam. +export PERMISSION_STATE_OSTYPE=linux-gnu + +if ! command -v jq >/dev/null 2>&1 || ! command -v git >/dev/null 2>&1; then + echo "SKIP: jq and git are required" >&2 + exit 0 +fi + +# A fixture never inherits the caller's git environment. +unset GIT_DIR GIT_WORK_TREE GIT_CONFIG GIT_INDEX_FILE + +BASE="$(cd "$TEST_TMPDIR" && pwd -P)" +ROOT="$BASE/repo" +mkdir -p "$ROOT/.claude" "$ROOT/sub/.claude" "$ROOT/bare-sub" "$BASE/home" "$BASE/outside/.claude" "$BASE/elsewhere/.claude" +git -C "$ROOT" init -q +git -C "$ROOT" -c user.name=fixture -c user.email=fixture@example.com -c commit.gpgsign=false commit -q --allow-empty -m fixture +git -C "$ROOT" worktree add -q "$BASE/wt" -b fixture-wt +mkdir -p "$BASE/wt/.claude" "$BASE/wt/sub" + +jq -n '{permissions:{allow:["Bash(root-shared)"],deny:["Bash(root-deny)"]}}' >"$ROOT/.claude/settings.json" +jq -n '{permissions:{allow:["Bash(root-local)"]}}' >"$ROOT/.claude/settings.local.json" +jq -n '{permissions:{allow:["Bash(sub-shared)"],deny:["Bash(sub-deny)"]}}' >"$ROOT/sub/.claude/settings.json" +jq -n '{permissions:{allow:["Bash(wt-shared)"]}}' >"$BASE/wt/.claude/settings.json" +jq -n '{permissions:{allow:["Bash(outside-shared)"]}}' >"$BASE/outside/.claude/settings.json" +jq -n '{permissions:{allow:["Bash(elsewhere-shared)"]}}' >"$BASE/elsewhere/.claude/settings.json" + +# run_in [env assignments]: the reader started from . +run_in() { + local dir="$1" + shift + ( + cd "$dir" || exit 1 + env -u CLAUDE_CONFIG_DIR -u CLAUDE_CODE_REMOTE -u CLAUDE_PROJECT_DIR -u PERMISSION_STATE_FIXTURE_DIR -u PERMISSION_STATE_STARTDIR \ + HOME="$BASE/home" \ + PERMISSION_STATE_MANAGED_PATH="$BASE/policy/managed-settings.json" \ + PERMISSION_STATE_REGISTRY_KEYS="" PERMISSION_STATE_PLIST_DOMAIN="" \ + "$@" bash "$SCRIPT" + ) +} + +# --- A session at the repository root reads the root's shared file ----------- +OUT="$(run_in "$ROOT")" +assert_contains "root session: shared file at the root" "$OUT" "project settings present $ROOT/.claude/settings.json" +assert_contains "root session: its rule is read" "$OUT" "rule project settings allow Bash(root-shared)" +assert_contains "root session: no second file to read" "$OUT" "project toplevel not-applicable -" +assert_eq "root session: the root's rule is reported once" 1 "$(count_matching "$OUT" 'Bash\(root-shared\)')" + +# --- Below the root: a session started here and a root session after a Bash cd -- +# look the same to the reader (tools-reference, what persists between commands), +# and each reads a different shared file (settings, the shared-file paragraph), so +# both files are reported and neither's deny rules go missing. +OUT="$(run_in "$ROOT/sub")" +assert_contains "below the root: shared file in this directory" "$OUT" "project settings present $ROOT/sub/.claude/settings.json" +assert_contains "below the root: its rule is read" "$OUT" "rule project settings allow Bash(sub-shared)" +assert_contains "below the root: its deny rule is read" "$OUT" "rule project settings deny Bash(sub-deny)" +# The root is matched by its tail: git spells it differently from the shell on Windows. +assert_eq "below the root: the root's shared file is read too" 1 "$(count_matching "$OUT" '^project toplevel present .*/repo/\.claude/settings\.json$')" +assert_contains "below the root: the root's deny rule is not omitted" "$OUT" "rule project toplevel deny Bash(root-deny)" +assert_not_contains "below the root: the root's rule is not filed under this directory" "$OUT" "rule project settings allow Bash(root-shared)" +assert_eq "below the root: a note names this directory and says both were read" 1 "$(count_matching "$OUT" "^NOTE: .*$ROOT/sub.*below the repository root.*reads both")" +assert_eq "subdirectory session: the local file stays at the root" 1 "$(count_matching "$OUT" '^local settings present .*/repo/\.claude/settings\.local\.json$')" +assert_contains "subdirectory session: the local rule is read" "$OUT" "rule local settings allow Bash(root-local)" +MERGED="$(printf '%s\n' "$OUT" | bash "$SCRIPT_DIR/permission-merge.sh" --merge-only)" +assert_contains "below the root: the merge keeps the root's deny" "$MERGED" "precedence_basis=uncontested Bash(root-deny)" +assert_contains "below the root: the merge keeps this directory's deny" "$MERGED" "precedence_basis=uncontested Bash(sub-deny)" +assert_eq "below the root: --merge-only keeps the over-report note" 1 "$(count_matching "$MERGED" '^NOTE: .*can over-report')" +assert_eq "below the root: --merge-only still drops the reader's other notes" 0 "$(count_matching "$MERGED" '^NOTE: local scope')" + +# --- With no shared file here, the root's is still labeled as the root's -------- +OUT="$(run_in "$ROOT/bare-sub")" +assert_contains "no shared file here: reported absent at this directory" "$OUT" "project settings absent $ROOT/bare-sub/.claude/settings.json" +assert_contains "no shared file here: the root's deny rule is not omitted" "$OUT" "rule project toplevel deny Bash(root-deny)" +assert_not_contains "no shared file here: the root's rule is not filed under this directory" "$OUT" "rule project settings allow Bash(root-shared)" + +# --- A root file that cannot be read makes the merge incomplete, not clean ------- +BROKEN="$BASE/broken" +mkdir -p "$BROKEN/.claude" "$BROKEN/sub" +git -C "$BROKEN" init -q +printf '{not json' >"$BROKEN/.claude/settings.json" +OUT="$(run_in "$BROKEN/sub")" +assert_eq "broken root file: reported invalid" 1 "$(count_matching "$OUT" '^project toplevel invalid-json ')" +assert_contains "broken root file: the merge says it is incomplete" "$(printf '%s\n' "$OUT" | bash "$SCRIPT_DIR/permission-merge.sh" --merge-only)" "status=incomplete" + +# --- A repository rooted at the home directory: the root file is the user file -- +OUT="$(run_in "$ROOT/sub" HOME="$ROOT")" +assert_eq "root is home: the root file is not read as project" 1 "$(count_matching "$OUT" '^project toplevel not-applicable .*/repo/\.claude/settings\.json$')" +assert_contains "root is home: it is read as user" "$OUT" "rule user settings deny Bash(root-deny)" +assert_not_contains "root is home: it is not read twice" "$OUT" "rule project toplevel" + +# --- A worktree session: shared file in the worktree, local at the main checkout +OUT="$(run_in "$BASE/wt")" +assert_contains "worktree session: shared file in the worktree" "$OUT" "project settings present $BASE/wt/.claude/settings.json" +assert_contains "worktree session: its rule is read" "$OUT" "rule project settings allow Bash(wt-shared)" +assert_contains "worktree session: no second file to read" "$OUT" "project toplevel not-applicable -" +assert_eq "worktree session: local file at the main checkout" 1 "$(count_matching "$OUT" '^local settings present .*/repo/\.claude/settings\.local\.json$')" +assert_contains "worktree session: the basis line says so" "$OUT" "main checkout, resolved through this worktree" + +# --- Below a worktree's root: the worktree's shared file, not the main checkout's +OUT="$(run_in "$BASE/wt/sub")" +assert_contains "below a worktree root: the worktree's rule is read" "$OUT" "rule project toplevel allow Bash(wt-shared)" +assert_not_contains "below a worktree root: the main checkout's shared rule is not" "$OUT" "Bash(root-shared)" + +# --- Outside a repository: the working directory, whatever the environment says +OUT="$(run_in "$BASE/outside" CLAUDE_PROJECT_DIR="$BASE/elsewhere")" +assert_contains "outside a repository: shared file here" "$OUT" "project settings present $BASE/outside/.claude/settings.json" +assert_not_contains "outside a repository: CLAUDE_PROJECT_DIR is not a second input" "$OUT" "Bash(elsewhere-shared)" +assert_contains "outside a repository: the local file sits beside it" "$OUT" "local settings absent $BASE/outside/.claude/settings.local.json" +assert_contains "outside a repository: no second file to read" "$OUT" "project toplevel not-applicable -" + +report_and_exit diff --git a/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.sh b/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.sh index 21834504516..7975b608984 100755 --- a/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.sh +++ b/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.sh @@ -21,6 +21,8 @@ # # scope managed | user | project | local | startdir-local # surface file | dropin-dir | dropin-file: | registry | plist (managed); +# toplevel (project: the repository root's shared file, read when the +# working directory is below the root; see Root resolution); # settings (everything else) # status present | absent | unreadable | invalid-json | skipped | not-applicable # kind allow | ask | deny @@ -46,7 +48,7 @@ # # Test seams (the real locations are absolute system paths and a real user home, # which no test may touch): -# PERMISSION_STATE_FIXTURE_DIR project root +# PERMISSION_STATE_FIXTURE_DIR primary working directory, also its repository root # PERMISSION_STATE_STARTDIR session start directory # PERMISSION_STATE_MANAGED_PATH managed-settings.json (drop-in dir follows it) # PERMISSION_STATE_REGISTRY_KEYS newline-separated registry keys to query @@ -157,13 +159,36 @@ reg_cmd() { MSYS2_ARG_CONV_EXCL='*' reg "$@"; } # before v2.1.211 and still reads what an earlier version left behind. START_DIR="${PERMISSION_STATE_STARTDIR:-$PWD}" +# The session's primary working directory, the one input to both project-file +# lookups: the shared file is read from it, and the repository root that anchors +# the local file is resolved from it. This script cannot see the session itself, +# so it takes its own working directory as that directory and says so in a note. if [[ -n "${PERMISSION_STATE_FIXTURE_DIR:-}" ]]; then - PROJECT_ROOT="$PERMISSION_STATE_FIXTURE_DIR" + PRIMARY_DIR="$PERMISSION_STATE_FIXTURE_DIR" + REPO_ROOT="$PERMISSION_STATE_FIXTURE_DIR" else - PROJECT_ROOT="$(git rev-parse --show-toplevel 2>/dev/null | tr -d '\r')" - [[ -n "$PROJECT_ROOT" ]] || PROJECT_ROOT="${CLAUDE_PROJECT_DIR:-$PWD}" + PRIMARY_DIR="$PWD" + # The prefix, not a path comparison: it is empty at the root however the shell + # and git spell that path. + { + read -r REPO_ROOT + read -r REPO_PREFIX + } < <(git rev-parse --show-toplevel --show-prefix 2>/dev/null | tr -d '\r') fi +# A Bash `cd` inside the project changes this script's working directory and not +# the session's primary one, and nothing here can tell that from a session started +# in this directory. So below the repository root both shared files are candidates. +# Taking either one alone could drop the deny rules of the file in force: this +# directory's when the session started here, the root's after a `cd` from the root. +# Both are read, which can over-report one of them. A session that started in +# another subdirectory and then ran `cd` has its own file read by neither, which is +# why the skill says to run this from the start directory. The root's is +# `toplevel`, a surface of the project scope, so the merge and lint stages treat it +# as project settings. +TOPLEVEL_SHARED="" +[[ -z "${REPO_PREFIX:-}" || -z "${REPO_ROOT:-}" ]] || TOPLEVEL_SHARED="$REPO_ROOT/.claude/settings.json" + # settings.local.json lives "at the root of the git repository, resolved through # worktrees to the main checkout, so one file covers sessions started in any # subdirectory or worktree" — anchoring on --show-toplevel would look for it in @@ -181,13 +206,13 @@ fi # directory. That is a helper's behavior, not a property of the running session, # so nothing here can observe it. No documented environment variable identifies an # Agent SDK or headless run either, so it is stated rather than guessed at. -LOCAL_ROOT="$PROJECT_ROOT" +LOCAL_ROOT="$REPO_ROOT" local_basis="repository root" if [[ -z "${PERMISSION_STATE_FIXTURE_DIR:-}" ]]; then common_dir="$(git rev-parse --git-common-dir 2>/dev/null | tr -d '\r')" if [[ -n "$common_dir" ]]; then main_root="$(cd "$common_dir/.." 2>/dev/null && pwd)" - if [[ -n "$main_root" && "$main_root" != "$PROJECT_ROOT" ]]; then + if [[ -n "$main_root" && "$main_root" != "$REPO_ROOT" ]]; then LOCAL_ROOT="$main_root" local_basis="main checkout, resolved through this worktree" fi @@ -548,17 +573,24 @@ fi same_file() { [[ "$1" -ef "$2" || "$1" == "$2" ]]; } emit_file_scope() { - # emit_file_scope - local scope="$1" path="$2" + # emit_file_scope [surface] + local scope="$1" path="$2" surface="${3:-settings}" if [[ -n "$USER_CONFIG_ROOT" ]] && same_file "$path" "$USER_CONFIG_ROOT/settings.json"; then - emit "$scope" settings not-applicable "$path" + emit "$scope" "$surface" not-applicable "$path" note "$scope scope skipped: $path is the user settings file (as when a session starts in the home directory, outside a repository), so it is reported once, as user, and no $scope-scope checks run on it." return 0 fi - emit_json_scope "$scope" settings "$path" + emit_json_scope "$scope" "$surface" "$path" } -emit_file_scope project "$PROJECT_ROOT/.claude/settings.json" +emit_file_scope project "$PRIMARY_DIR/.claude/settings.json" +if [[ -n "$TOPLEVEL_SHARED" ]]; then + emit_file_scope project "$TOPLEVEL_SHARED" toplevel + note "$PRIMARY_DIR is below the repository root $REPO_ROOT. A Bash cd inside the project moves this reader without moving the session, so this directory may or may not be the session's primary working directory, and the reader reads both shared files: 'project settings' from here and 'project toplevel' from the root. Only the file in the directory the session started in, or last moved to with /cd, is in force, so the merged set can over-report one of these two files. If the session started in neither directory, its file is not read here: run this reader from the session's start directory. https://code.claude.com/docs/en/settings#where-claude-code-keeps-the-local-file-in-a-git-repository" +else + emit project toplevel not-applicable "-" + note "project scope read from $PRIMARY_DIR, the reader's working directory, taken as the session's primary working directory. A repository root's .claude/settings.json does not apply to a session whose primary working directory is elsewhere." +fi emit_file_scope local "$LOCAL_ROOT/.claude/settings.local.json" note "local scope anchored on the $local_basis. The Agent SDK's resolveSettings() helper always reads the file from the starting directory; that is the helper's behavior, not a session property, and no documented environment variable identifies an Agent SDK or headless run, so this reader does not detect either." @@ -569,7 +601,7 @@ note "local scope anchored on the $local_basis. The Agent SDK's resolveSettings( # when Claude Code is running as a cloud session. Read this from a hook or setup # script to detect whether you are in a cloud session." if [[ "${CLAUDE_CODE_REMOTE:-}" == "true" ]]; then - note "CLAUDE_CODE_REMOTE=true: this is a cloud session, which reads a DIFFERENT scope set than a local one. Your own machine's user settings and .claude/settings.local.json are not read at all, and only server-managed settings reach the session — a managed-settings.json file or MDM profile on your device does not. Any user scope reported above is this container's file, not yours." + note "CLAUDE_CODE_REMOTE=true: this is a cloud session, which reads a DIFFERENT scope set than a local one. Your own machine's user settings and .claude/settings.local.json are not read at all, and a managed-settings.json file or MDM profile on your device does not reach it. A self-hosted environment's runner image can carry its own managed file, and a cloud thread in a project takes no permission rules from a repository's .claude/settings.json; this reader detects neither case, so read https://code.claude.com/docs/en/settings#settings-in-cloud-sessions before trusting the managed or project records above. Any user scope reported above is this container's file, not yours." fi # The pre-v2.1.211 copy is a DISTINCT scope member, not a fallback: when both diff --git a/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.test.sh b/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.test.sh index 3bd09391279..41c1baf0a05 100755 --- a/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.test.sh +++ b/plugins/harness-config/skills/audit-permission-state/scripts/permission-state.test.sh @@ -84,7 +84,8 @@ rc=0 OUT=$(run) || rc=$? assert_exit "full run exits 0" 0 "$rc" assert_eq "user scope appears exactly once" "1" "$(count_matching "$OUT" '^user ')" -assert_eq "project scope appears exactly once" "1" "$(count_matching "$OUT" '^project ')" +assert_eq "project settings surface appears exactly once" "1" "$(count_matching "$OUT" '^project settings ')" +assert_eq "project toplevel surface appears exactly once" "1" "$(count_matching "$OUT" '^project toplevel ')" assert_eq "local scope appears exactly once" "1" "$(count_matching "$OUT" '^local ')" assert_eq "start-directory scope appears exactly once" "1" "$(count_matching "$OUT" '^startdir-local ')" diff --git a/plugins/harness-config/skills/audit/SKILL.md b/plugins/harness-config/skills/audit/SKILL.md index d97895c7907..e107b9105d3 100644 --- a/plugins/harness-config/skills/audit/SKILL.md +++ b/plugins/harness-config/skills/audit/SKILL.md @@ -64,7 +64,7 @@ the document and does only what needs judgment: | E: marketplace membership, every `false` key as an inventory row (a finding only when an enabled plugin depends on it), ORPHAN / RENAME / REMOVED drift, catalog plugins with no entry in any scope as one inventory row per marketplace, keys the drift check did not diff, `strict` versus `plugin.json` | E: the fix for a disabled dependency, orphan-`true` review, rename confirmation | | F: token-shaped values, documentation status against the fetched `env-vars` page | F: whether an undocumented custom variable is justified | | G: the measurement, read from an existing debug log; `skillOverrides` keys that name a known plugin (inert, `warning`), colon keys whose prefix names no plugin (`skip`), and entries in the user dir's `settings.local.json` (`info`) | G: the levers, scoped to the roster's composition, and what an undecided colon key names | -| H and I: every value check; the accepted `effortLevel` and `disableDeepLinkRegistration` values, the version `enforceAvailableModels` requires, the `fallbackModel` cap and the values of every other string key whose Type bullet lists them come from the fetched `settings-reference`; a `deniedModels` or `availableModelsMatch` key outside managed settings is flagged from a rule kept in our words | H and I: nothing, once the Phase 3 fetch confirms the behavior the row rests on | +| H and I: every value check; the accepted `effortLevel` and `disableDeepLinkRegistration` values, the version `enforceAvailableModels` requires and the values of every other string key whose Type bullet lists them come from the fetched `settings-reference`, and the `fallbackModel` cap from the fetched `model-config` "Fallback model chains" section; a `deniedModels` or `availableModelsMatch` key outside managed settings is flagged from a rule kept in our words | H and I: nothing, once the Phase 3 fetch confirms the behavior the row rests on | | J: each `Fixed in vX.Y.Z` a `reference/known-issues.md` row records, compared with the installed Claude Code version (the form is that file's "Recording a fix version"). K: the age of each suppression entry against `exceptions.max_days` (default 90; the team YAML may set another), a `warning` past it that names the entry, its reason and its age, never a removal | J: the live status of each issue (Phase 3.2), and whether a workaround an `info` row flags is still needed. K: nothing; report each aged entry for the operator to re-date or remove | A row the engine marks `skip` or `not-inspectable` is exactly that in the report: never clean. diff --git a/plugins/harness-config/skills/audit/reference/audit-checklist.md b/plugins/harness-config/skills/audit/reference/audit-checklist.md index 6bc8da0bab0..a7f71d58727 100644 --- a/plugins/harness-config/skills/audit/reference/audit-checklist.md +++ b/plugins/harness-config/skills/audit/reference/audit-checklist.md @@ -194,21 +194,22 @@ section F resolves environment variables against their own page. | Check | Severity | How to verify | | --- | --- | --- | | `effortLevel` is a value its **Type** bullet documents | warning | `jq '.effortLevel'`. The accepted values are the **Type** bullet of the [`effortLevel` section](https://code.claude.com/docs/en/settings-reference#effortlevel) on settings-reference; the engine reads that bullet each run and reports any other value. Report it as a value the page does not document; the page does not state what runs instead, so do not assert a level | -| `fallbackModel` keeps no more distinct models than the page says the chain keeps | warning | The cap is the number the [`fallbackModel` section](https://code.claude.com/docs/en/settings-reference#fallbackmodel) states for distinct allowed models; the engine reads it each run and applies no other. A section that states none gives one `skip` row. Two tests can disagree: the declared schema caps RAW array length, while the page caps the chain after duplicate removal, so with a cap of 3 `["sonnet","haiku","sonnet","opus"]` has 4 raw entries and exactly 3 distinct. Only the distinct count is a finding; a raw count above the cap is a `skip` row because the schema is not read. Dedupe in place with `jq '.fallbackModel \| reduce .[] as $m ([]; if index($m) then . else . + [$m] end)'`, since `unique` would sort away the order the chain is tried in, and name any entry past the cap as at risk of being ignored. Not "dead": allowlist-excluded entries are also dropped when the chain is read, and the page does not state whether that dropping happens before or after the cap. Pointer: for the cap and the allowlist drop, see the [`fallbackModel` section](https://code.claude.com/docs/en/settings-reference#fallbackmodel) and [model configuration: fallback model chains](https://code.claude.com/docs/en/model-config#fallback-model-chains); for the raw-length limit, the SchemaStore document's `properties.fallbackModel`. As of: 2026-10-01 (both pages state a cap and neither orders the allowlist drop against it). Recheck trigger: either page drops or changes its cap, or states whether the allowlist drop runs before the cap | +| `fallbackModel` keeps no more distinct models than the page says the chain keeps | warning | The cap is the number [model configuration: fallback model chains](https://code.claude.com/docs/en/model-config#fallback-model-chains) states for a chain after duplicate removal; the engine reads that section each run, only when a file sets `fallbackModel`, and applies no other. A section that states none gives one `skip` row; an unread page, or one without the section, gives one `not-inspectable` row. Two tests can disagree: the declared schema caps RAW array length, while the page caps the chain after duplicate removal, so with a cap of 3 `["sonnet","haiku","sonnet","opus"]` has 4 raw entries and exactly 3 distinct. Only the distinct count is a finding; a raw count above the cap is a `skip` row because the schema is not read. Dedupe in place with `jq '.fallbackModel \| reduce .[] as $m ([]; if index($m) then . else . + [$m] end)'`, since `unique` would sort away the order the chain is tried in, and name any entry past the cap as at risk of being ignored. Not "dead": allowlist-excluded entries are also dropped when the chain is read, and the page does not state whether that dropping happens before or after the cap. Pointer: for the cap and the allowlist drop, see [model configuration: fallback model chains](https://code.claude.com/docs/en/model-config#fallback-model-chains); for the raw-length limit, the SchemaStore document's `properties.fallbackModel`. As of: 2026-11-04 (that section states the cap, settings-reference's `fallbackModel` section no longer does, and neither orders the allowlist drop against it). Recheck trigger: that section drops, moves or changes its cap, or states whether the allowlist drop runs before the cap | | Any other string-valued key whose **Type** bullet lists its values holds one of them | warning | `jq 'to_entries[] \| select(.value \| type == "string")'` over `settings.json` and `~/.claude/settings.json`; `settings.local.json` is not read by value. The engine takes the values from the `string, one of:` bullets on settings-reference, treats an entry such as `custom:` as matching any text in its placeholder, and gives a key with no such list no row. A list holding a bullet that is not a literal value, such as the strftime pattern in the `timeFormat` list, is open: its literals are not the whole set, so the key has no row. `effortLevel` and `disableDeepLinkRegistration` keep their own rows. A key whose Type bullet states its values inline in a sentence is not parsed, so it has no row | -| `availableModels` does not mix a family wildcard with a specific entry of that family | warning | The engine fixes the families as `opus`, `sonnet`, `haiku` and `fable`; the record below the table says why. Flag a list that holds a family's wildcard and also an entry naming a specific model of that family; we read the pair as narrower than its author meant. A Mantle ID, or an `ANTHROPIC_CUSTOM_MODEL_OPTION` value embedding a family name, counts as such an entry. This row is not silent at run time (the pointer says how Claude Code tells the user), so the finding is the narrowing itself. For what the list then permits, follow the pointer. Report the models they most likely still expect to be selectable. Pointer: [model configuration: merge behavior](https://code.claude.com/docs/en/model-config#merge-behavior). As of: 2026-10-01. Recheck trigger: that section changes how a specific entry interacts with its family's wildcard. This skill does not read managed settings, so say the finding assumes no managed `availableModelsMatch` or `deniedModels` applies. Pointer: when a managed match mode or deny list may apply, fetch [settings-reference: `availableModelsMatch`](https://code.claude.com/docs/en/settings-reference#availablemodelsmatch) and [settings-reference: `deniedModels`](https://code.claude.com/docs/en/settings-reference#deniedmodels) live. As of: 2026-10-10. Recheck trigger: either section changes what its key does to the models `availableModels` permits | +| `availableModels` does not mix a family wildcard with a specific entry of that family | warning | The engine fixes the families as `opus`, `sonnet`, `haiku` and `fable`; the record below the table says why. Flag a list that holds a family's wildcard and also an entry naming a specific model of that family; we read the pair as narrower than its author meant. A Mantle ID, or an `ANTHROPIC_CUSTOM_MODEL_OPTION` value embedding a family name, counts as such an entry. The finding is the narrowing itself. For what the list then permits, follow the pointer. Report the models they most likely still expect to be selectable. Pointer: [model configuration: merge behavior](https://code.claude.com/docs/en/model-config#merge-behavior). As of: 2026-11-04. Recheck trigger: that section changes how a specific entry interacts with its family's wildcard. This skill does not read managed settings, so say the finding assumes no managed `availableModelsMatch` or `deniedModels` applies. Pointer: when a managed match mode or deny list may apply, fetch [settings-reference: `availableModelsMatch`](https://code.claude.com/docs/en/settings-reference#availablemodelsmatch) and [settings-reference: `deniedModels`](https://code.claude.com/docs/en/settings-reference#deniedmodels) live. As of: 2026-10-10. Recheck trigger: either section changes what its key does to the models `availableModels` permits | | `enforceAvailableModels: true` is paired with a non-empty `availableModels` | error | `jq 'select(.enforceAvailableModels == true) \| .availableModels'`. The finding requires the flag to be `true` AND the list unset or empty. An explicit `false` is someone turning enforcement off on purpose and is never a finding, so gate on the value rather than the key's presence. When it does fire: we treat the key as having no effect without a non-empty `availableModels`, so the constraint its author set is not in force. That is an enforcement bypass, which this skill's severity guide rates `error`. Where both keys must live is a managed-settings placement question this skill cannot decide from the files it reads, so report the pairing, not the placement. Pointer: [settings-reference: `enforceAvailableModels`](https://code.claude.com/docs/en/settings-reference#enforceavailablemodels). As of: 2026-10-01. Recheck trigger: that section changes what the key does with an empty or unset list | | `deniedModels` and `availableModelsMatch` appear only in managed settings | warning | The engine reports it for the project, local and user files; by hand, `jq 'has("deniedModels"), has("availableModelsMatch")'` over `settings.json` and `~/.claude/settings.json` only. The engine fires on a rule kept in our words, reading no page; confirm it with the settings-reference fetch before reporting. We treat both keys as managed-only. Every file this skill reads is a user, project or local file, so either key there is a model policy its author believes is in force and is not. Report the key and the file, and name managed settings as where the policy belongs. Pointer: when you need which sources read these keys and how managed sources combine them, fetch [settings-reference: `availableModelsMatch`](https://code.claude.com/docs/en/settings-reference#availablemodelsmatch), [settings-reference: `deniedModels`](https://code.claude.com/docs/en/settings-reference#deniedmodels) and [managed settings: compose every managed source](https://code.claude.com/docs/en/managed-settings#compose-every-managed-source) live. As of: 2026-10-10. Recheck trigger: any of those sections reads these keys outside managed settings, or changes how managed sources combine them | -The engine keeps the family list fixed because no page it reads gives a list to parse: our probe of -the model-config page found no table or list that marks which aliases name a family (no artifact -stored). +The engine keeps the family list fixed and reads no page for it, because a change here is a deliberate +edit to one check. Compare it by hand with the page's own naming of the families when this row is +rechecked. -- **Pointer**: for the aliases, see - [model configuration: model aliases](https://code.claude.com/docs/en/model-config#model-aliases). -- **As of**: 2026-09-30, Claude Code 2.1.285 -- **Recheck trigger**: that table gains a marker for family aliases, the page states the family - aliases as a list, or the engine starts reading the page. +- **Pointer**: when comparing the list, fetch + [model configuration: restrict model selection](https://code.claude.com/docs/en/model-config#restrict-model-selection) + live. +- **As of**: 2026-11-04 +- **Recheck trigger**: that section names a family the engine's list lacks or drops one it holds, or + the engine starts reading the page. Model IDs in `modelOverrides` are not validated here: deciding whether a key is a real Anthropic model ID means resolving it against @@ -252,20 +253,24 @@ pointing at the other is the shape a hand-off takes when nobody closes it. | Check | Severity | How to verify | | --- | --- | --- | | A component's `effort:` pin names the model it was calibrated against, or an event that re-opens it | info | Read the frontmatter of every `skills/*/SKILL.md` and `agents/*.md` in scope. We treat a pin as calibrated to the model it was set on, so a pin that names neither that model nor an event that re-opens it is flagged. **Do not flag a pin at the resolved model's own default level.** That pin encodes no measurement that could go stale. Read the resolved model's default from the first pointer when the audit runs; this row keeps no copy of the per-model defaults, because the list has gained models without an edit here. When the page cannot be read, report the pin's default as unresolved rather than grading it. Report the missing re-derivation, never the level itself; whether the level is high enough is the next row's question, not this one's. Pointer: when deciding a model's default effort, fetch [model configuration: adjust effort level](https://code.claude.com/docs/en/model-config#adjust-effort-level) live; for per-model calibration, see [choose an effort level](https://code.claude.com/docs/en/model-config#choose-an-effort-level). As of: 2026-10-11. Recheck trigger: that section stops stating a per-model default, or moves it to another section | -| A code-changing or verifying component's `effort:` is not pinned below `medium` | warning | Read the same frontmatter. Flag an agent or skill whose `effort:` is pinned below `medium` (today that is `low`) when its body has the model change code (edit, implement, refactor, fix) or verify a change (run the build, the tests, a review, or a check that a change works). The floor is the marketplace's [Effort floor](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/plugin-philosophy.md#effort-floor): `medium` or above for that work on every model that supports effort. Classify by what the body has the model do, not by its name or tool grants. **Do not flag:** a component with no `effort:` pin, which inherits the session level and is not a pin; a read-only, chat-like, or mechanical component (lookups, formatting, listing) pinned `low`; or a model the pin runs on that has no effort support, where the key has no effect. Propose `medium` as the floor and leave any higher level to the author. Pointer: the floor's own record in that section, which points at the upstream effort pages. As of: 2026-10-01. Recheck trigger: that section changes the floor level or the kinds of work it covers | +| A code-changing or verifying component's `effort:` is not pinned below `medium` | warning | Read the same frontmatter. Flag an agent or skill whose `effort:` is pinned below `medium` (today that is `low`) when its body has the model change code (edit, implement, refactor, fix) or verify a change (run the build, the tests, a review, or a check that a change works). The floor is the marketplace's [Effort floor](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/plugin-philosophy.md#effort-floor): `medium` or above for that work on every model that supports effort. Classify by what the body has the model do, not by its name or tool grants. **Do not flag:** a component with no `effort:` pin, which is not a pin whatever level it then runs at; a read-only, chat-like, or mechanical component (lookups, formatting, listing) pinned `low`; or a model the pin runs on that has no effort support, where the key has no effect. Propose `medium` as the floor and leave any higher level to the author. Pointer: the floor's own record in that section, which points at the upstream effort pages. As of: 2026-10-01. Recheck trigger: that section changes the floor level or the kinds of work it covers | | No effort pin rests on a model-config effort table or default that changed since the baseline | warning | Run `bash "/scripts/check-effort-pins.sh" [--root ] [--baseline ] [--docs-dir ]` and report its lines verbatim; `--print-baseline` prints a new baseline line and writes nothing. Exit 0: the page matches [`effort-table.baseline`](effort-table.baseline) and every pin names a listed level. Exit 1: the page changed (`reason=table-changed` on every pin) or a pin names an unlisted level (`reason=level-not-in-table`); a person re-decides each flagged pin. Exit 3: page unread or reshaped, no claim about any pin. Exit 2: fatal. The script never edits a pin or the baseline; the rebaseline step is in [SKILL.md](../SKILL.md) | | A component's `effort:` and `model:` are consistent with each other | info | We read a definition's `model:` and `effort:` as the pair the author chose, not as all that a spawn runs on: other sources can override either one. Flag only the combination the author is unlikely to have intended: a cheap `model:` tier paired with a top effort level, or the reverse, with no stated reason. Report the mismatch, never a preferred pairing. Pointer: when deciding what a spawn actually runs on, fetch [choose a model](https://code.claude.com/docs/en/sub-agents#choose-a-model) and [choose an effort level](https://code.claude.com/docs/en/sub-agents#choose-an-effort-level) live. As of: 2026-11-01. Recheck trigger: either section's resolution order gains or loses a source | -These rows read a set `effort:` as the level the component runs at and an absent one as the -session level. A level the frontmatter field accepts is not evidence that the `effortLevel` -settings key accepts it; check each against its own pointer. +These rows read a set `effort:` as the level the component runs at and an absent one as no pin, +never as a claim about the level it runs at: that depends on the model it runs on. A level the +frontmatter field accepts is not evidence that the `effortLevel` settings key accepts it; check each +against its own pointer. - **Pointer**: for the field's precedence and accepted levels, see the [subagents frontmatter fields](https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields); + for a subagent with no level, see + [when a subagent has no `effort` field or parameter](https://code.claude.com/docs/en/sub-agents#when-a-subagent-has-no-effort-field-or-parameter); for the settings key, see [settings-reference: `effortLevel`](https://code.claude.com/docs/en/settings-reference#effortlevel). -- **As of**: 2026-10-01 (both pages read as raw markdown) -- **Recheck trigger**: the field's precedence or either accepted-level list changing. +- **As of**: 2026-11-04 +- **Recheck trigger**: the field's precedence or either accepted-level list changing, or the second + section changing which model an absent level follows. ## I. Deep-link registration diff --git a/plugins/harness-config/skills/audit/reference/effort-table.baseline b/plugins/harness-config/skills/audit/reference/effort-table.baseline index c6513f92023..7482206d15e 100644 --- a/plugins/harness-config/skills/audit/reference/effort-table.baseline +++ b/plugins/harness-config/skills/audit/reference/effort-table.baseline @@ -1 +1 @@ -sha256=8e796135748dcec14baab663d704cf635687852d16e395d89a9d2b0c84c420fb levels=low,medium,high,xhigh,max as_of=2026-10-10 source=https://code.claude.com/docs/en/model-config +sha256=07672ad77278551d31f01effe810d07764761a8e62714bfb5790d27767aaf22d levels=low,medium,high,xhigh,max as_of=2026-11-04 source=https://code.claude.com/docs/en/model-config diff --git a/plugins/harness-config/skills/audit/reference/required-permissions.md b/plugins/harness-config/skills/audit/reference/required-permissions.md index 17b0810454f..82c85796ab3 100644 --- a/plugins/harness-config/skills/audit/reference/required-permissions.md +++ b/plugins/harness-config/skills/audit/reference/required-permissions.md @@ -216,8 +216,17 @@ because pushes carry intent the agent should not infer. | Pattern | Purpose | | --- | --- | -| `Bash(git push *)` | Require confirmation before pushing with args | -| `Bash(git push)` | Require confirmation before pushing without args | +| `Bash(git push *)` | Require confirmation before pushing, with or without args | + +We keep one row because we read a trailing space and `*` as also covering the bare command, so a project +with this rule is not flagged for lacking a bare `Bash(git push)` rule. + +- **Pointer**: when deciding whether a rule covers the bare command, fetch + [permissions: wildcard patterns](https://code.claude.com/docs/en/permissions#wildcard-patterns) + live. +- **As of**: 2026-11-04 +- **Recheck trigger**: that section stops treating a trailing space and `*` as covering the bare + command. ## Relation to the standards permission floor @@ -263,11 +272,11 @@ This baseline is the smaller set the audit flags when absent. Where the two diff flagged for every extra. A project that adopts the floor's extras declares them in its own rules files, and Category B checks them there. - **`git push` is an ask rule here and an allow rule there.** The floor allows `Bash(git push *)` and - `Bash(git push)` and has no `ask` list. The baseline keeps both as ask rules, because a push carries - intent the agent should not infer. Carrying both does not cancel the ask rule: Claude Code evaluates - deny, then ask, then allow, and a matching ask rule prompts even when an allow rule matches too. A - project that takes the floor's allow without the ask rules gets the `ask-rules` warning. Narrowing 1 - below covers a project that pushes unattended. + `Bash(git push)` and has no `ask` list. The baseline keeps `Bash(git push *)` as an ask rule, because + a push carries intent the agent should not infer. Carrying both does not cancel the ask rule: Claude + Code evaluates deny, then ask, then allow, and a matching ask rule prompts even when an allow rule + matches too. A project that takes the floor's allow without the ask rule gets the `ask-rules` + warning. Narrowing 1 below covers a project that pushes unattended. The record for this comparison: diff --git a/plugins/harness-config/skills/audit/scripts/audit-engine.sh b/plugins/harness-config/skills/audit/scripts/audit-engine.sh index 467e3218e4a..41f5206a572 100755 --- a/plugins/harness-config/skills/audit/scripts/audit-engine.sh +++ b/plugins/harness-config/skills/audit/scripts/audit-engine.sh @@ -924,6 +924,14 @@ fi # reads another page adds its slug here. A page settings-reference does not link # is not requested. One more fetcher call reads those --docs-dir does not hold. LINKED_PAGES=(hooks) +# The fallbackModel chain cap is stated on model-config, which settings-reference's +# fallbackModel section links; the page is requested only when a file sets the key. +for f in "$SETTINGS:$PROJECT_OK" "$LOCAL:$LOCAL_OK" "$USER_SETTINGS:$USER_OK"; do + if [[ "${f##*:}" == 1 && "$(jqf "${f%:*}" -r 'has("fallbackModel")')" == "true" ]]; then + LINKED_PAGES+=(model-config) + break + fi +done linked=() if [[ -n "$SR" ]]; then for slug in "${LINKED_PAGES[@]}"; do @@ -2256,14 +2264,34 @@ documented_value() { fi } -# fallback_cap: how many distinct models the fallbackModel section says the -# chain keeps ("at most three distinct allowed models"), as digits, from a digit -# string or a number word up to ten. Nothing when the section states none. +# The "Fallback model chains" section of model-config, fenced code left out, or +# nothing when the page was not read or has no such heading. +MC="${PAGE_FILE[model-config]:-}" +MC_CHAINS="" +[[ -n "$MC" ]] && MC_CHAINS="$(awk ' + /^[[:space:]]*```/ { fence = !fence; next } + fence { next } + /^#+ / { if (on && length($1) <= lvl) exit; if ($0 ~ /^#+ Fallback model chains$/) { on = 1; lvl = length($1) } next } + on { print } +' "$MC")" +if [[ -z "$MC" ]]; then + MC_UNREAD_WHY="model-config was not read this run" + [[ -z "$SR" ]] && MC_UNREAD_WHY="$SR_UNREAD_WHY, so model-config was not requested" +elif [[ -z "$MC_CHAINS" ]]; then + MC_UNREAD_WHY="model-config was read but has no Fallback model chains section" +else + MC_UNREAD_WHY="" +fi + +# fallback_cap: how many distinct models model-config's Fallback model chains +# section says a chain keeps ("caps chains at three models after duplicate +# removal"), as digits, from a digit string or a number word up to ten. Nothing +# when the section states none. fallback_cap() { local t - t="$(sr_section fallbackModel | tr '\n' ' ')" + t="$(tr '\n' ' ' <<<"$MC_CHAINS")" t="${t,,}" - [[ "$t" =~ (at\ most|up\ to|no\ more\ than)\ ([0-9]+|one|two|three|four|five|six|seven|eight|nine|ten)(\ [a-z]+){0,2}\ models ]] || return 0 + [[ "$t" =~ (at\ most|up\ to|no\ more\ than|caps\ chains\ at)\ ([0-9]+|one|two|three|four|five|six|seven|eight|nine|ten)(\ [a-z]+){0,2}\ models ]] || return 0 case "${BASH_REMATCH[2]}" in one) echo 1 ;; two) echo 2 ;; three) echo 3 ;; four) echo 4 ;; five) echo 5 ;; six) echo 6 ;; seven) echo 7 ;; eight) echo 8 ;; nine) echo 9 ;; ten) echo 10 ;; @@ -2295,16 +2323,16 @@ check_h() { raw="$(jqf "$file" -r '.fallbackModel | if type=="array" then length else 1 end')" dedup="$(jqf "$file" -r '.fallbackModel | if type=="array" then (reduce .[] as $m ([]; if index($m) then . else . + [$m] end) | length) else 1 end')" cap="$(fallback_cap)" - if [[ -z "$SR" ]]; then - row H fallback-chain not-inspectable none "$surface" "fallbackModel-cap" "$SR_UNREAD_WHY; how many fallback models the chain keeps is not known" - + if [[ -n "$MC_UNREAD_WHY" ]]; then + row H fallback-chain not-inspectable none "$surface" "fallbackModel-cap" "$MC_UNREAD_WHY; how many fallback models the chain keeps is not known" - elif [[ -z "$cap" ]]; then - row H fallback-chain skip none "$surface" "fallbackModel-cap" "the fallbackModel section on settings-reference states no chain length; not decided" - + row H fallback-chain skip none "$surface" "fallbackModel-cap" "the Fallback model chains section on model-config states no chain length; not decided" - else if [[ "$dedup" -gt "$cap" ]]; then - row H fallback-chain finding warning "$surface" "fallbackModel-dedup-length:$dedup" "fallbackModel keeps $dedup distinct entries; settings-reference keeps at most $cap distinct models, so later entries may be ignored" /fallbackModel + row H fallback-chain finding warning "$surface" "fallbackModel-dedup-length:$dedup" "fallbackModel keeps $dedup distinct entries; model-config caps a chain at most $cap distinct models, so later entries may be ignored" /fallbackModel fi if [[ "$raw" -gt "$cap" ]]; then - row H fallback-chain skip none "$surface" "fallbackModel-raw-length:$raw" "fallbackModel has $raw entries against the $cap settings-reference keeps; a limit on the raw array belongs to the declared schema, which this run does not read; not decided" - + row H fallback-chain skip none "$surface" "fallbackModel-raw-length:$raw" "fallbackModel has $raw entries against the $cap model-config keeps; a limit on the raw array belongs to the declared schema, which this run does not read; not decided" - fi fi fi diff --git a/plugins/harness-config/skills/audit/scripts/audit-engine.test.sh b/plugins/harness-config/skills/audit/scripts/audit-engine.test.sh index 923437acaa1..710d75526ba 100755 --- a/plugins/harness-config/skills/audit/scripts/audit-engine.test.sh +++ b/plugins/harness-config/skills/audit/scripts/audit-engine.test.sh @@ -113,6 +113,28 @@ cat >"$DOCS/llms.txt" <<'EOF' - [All settings](https://code.claude.com/docs/en/settings-reference.md): Complete reference for every Claude Code settings.json key. - [Environment variables](https://code.claude.com/docs/en/env-vars.md): Reference for environment variables that control Claude Code behavior. - [Mods overview](https://code.claude.com/docs/en/plugins/mods/overview.md): Add panes, commands, and tool call rules to Claude Code with a mod. +- [Model configuration](https://code.claude.com/docs/en/model-config.md): Configure which model Claude Code uses. +EOF +# The model-config section the fallbackModel cap is read from, in the published +# page's shape (https://code.claude.com/docs/en/model-config.md#fallback-model-chains, +# read 2026-11-04), then a later section carrying a decoy cap. +cat >"$DOCS/model-config.md" <<'EOF' +# Model configuration + +## Fallback + +### Fallback model chains + +Configure one or more fallback models and Claude Code tries them in order. +Claude Code caps chains at three models after duplicate removal and ignores extra entries. + +```text +Claude Code caps chains at one model inside a fence, which is not prose. +``` + +### Automatic model fallback + +A decoy: Claude Code caps chains at one model in this section. EOF # The mods overview, holding the two sentences the mod rows quote, copied from # the published page (https://code.claude.com/docs/en/plugins/mods/overview.md, @@ -151,7 +173,7 @@ section() { printf '### `enforceAvailableModels`\n\nThis key closes that gap. Requires Claude Code v2.1.175 or later.\n\n' printf '* **Scope**: [`Any file`](#scopes)\n* **Type**: Boolean\n * `true`: Default resolves to the first available model\n* **Default**: `false`\n\n' printf 'This key has no effect when `availableModels` is unset or empty. Requires Claude Code v2.1.999 or later.\n\n' - section fallbackModel '* **Type**: array of strings' 'Claude Code keeps at most three distinct allowed models from the list and ignores the rest.' + section fallbackModel '* **Type**: array of strings' 'Name [backup models](https://code.claude.com/docs/en/model-config.md#fallback-model-chains) for when the primary is overloaded.' printf '### `editorMode`\n\nSet the prompt input key bindings.\n\n* **Scope**: [`Any file`](#scopes)\n* **Type**: string, one of:\n' printf ' * `"normal"`: standard key bindings\n * `"vim"`: vim-style editing\n* **Default**: `"normal"`\n\n' printf '### `theme`\n\nPick the color theme.\n\n* **Scope**: [`Any file`](#scopes)\n* **Type**: string, one of:\n' @@ -1797,31 +1819,62 @@ out=$(run "$m" --json --docs-dir "$m/nolink" 2>&1) || true assert_eq "case 51: a page settings-reference does not link is not acquired" "settings-reference env-vars plugins/mods/overview" "$(pages "$out")" # --- Case 52: the fallbackModel cap and enum values come from the page ---------- +# The cap is read from model-config's "Fallback model chains" section, which +# settings-reference's fallbackModel section links; settings-reference states none. m="$(make_machine caps)" printf '%s\n' "$CLEAN_SETTINGS" | jq '. + {fallbackModel:["a","b","c"]}' >"$m/project/.claude/settings.json" fb_rows() { jq -r --arg c "$2" '[.rows[] | select(.claim | startswith($c)) | "\(.status) \(.claim)"] | join(",")' <<<"$1"; } +# mc_dir : a docs directory whose model-config is the fixture's run through . +mc_dir() { + mkdir -p "$1" + cp "$DOCS/llms.txt" "$DOCS/env-vars.md" "$DOCS/settings-reference.md" "$1/" + sed -e "$2" "$DOCS/model-config.md" >"$1/model-config.md" + cmp -s "$DOCS/model-config.md" "$1/model-config.md" && fail "case 52: $1 edits the page" "the sed script changed nothing" +} # A page stating a cap of two, as a word and as a digit: three distinct is above it. -mkdir -p "$m/two" "$m/nine" -cp "$DOCS/llms.txt" "$DOCS/env-vars.md" "$m/two/" -cp "$DOCS/llms.txt" "$DOCS/env-vars.md" "$m/nine/" -sed 's/at most three distinct/at most two distinct/' "$DOCS/settings-reference.md" >"$m/two/settings-reference.md" -sed 's/at most three distinct/up to 9 distinct/' "$DOCS/settings-reference.md" >"$m/nine/settings-reference.md" +mc_dir "$m/two" 's/caps chains at three models/caps chains at two models/' +mc_dir "$m/nine" 's/caps chains at three models/caps chains at 9 models/' out=$(run "$m" --json --docs-dir "$m/two" 2>&1) || true assert_eq "case 52: three distinct entries above a stated cap of two are flagged" "finding fallbackModel-dedup-length:3" "$(fb_rows "$out" fallbackModel-dedup)" assert_contains "case 52: the finding names the page's cap" "$(jq -r '.findings[] | select(.identity.claim=="fallbackModel-dedup-length:3") | .detail' <<<"$out")" "at most 2 distinct" +assert_contains "case 52: the finding names model-config as the source" "$(jq -r '.findings[] | select(.identity.claim=="fallbackModel-dedup-length:3") | .detail' <<<"$out")" "model-config" out=$(run "$m" --json --docs-dir "$m/nine" 2>&1) || true assert_eq "case 52: a digit cap of nine leaves three distinct entries clean" "" "$(fb_rows "$out" fallbackModel)" # A page stating three: at the cap, not above it. out=$(run "$m" --json 2>&1) || true assert_eq "case 52: three distinct entries at a stated cap of three are clean" "" "$(fb_rows "$out" fallbackModel)" +assert_eq "case 52: model-config is read" "read" "$(jq -r '.docs.pages[] | select(.slug=="model-config") | .state' <<<"$out")" +# A cap of one in another section of the same page is not the chain's cap. +mc_dir "$m/decoy" 's/caps chains at three models/keeps a chain/' +out=$(run "$m" --json --docs-dir "$m/decoy" 2>&1) || true +assert_eq "case 52: a cap outside the Fallback model chains section is not read" "skip fallbackModel-cap" "$(fb_rows "$out" fallbackModel)" +# A cap of one on settings-reference is no longer the source. +mkdir -p "$m/srcap" +cp "$DOCS/llms.txt" "$DOCS/env-vars.md" "$DOCS/model-config.md" "$m/srcap/" +sed 's/^What fallbackModel does\.$/Claude Code keeps at most one distinct allowed model from the list./' "$DOCS/settings-reference.md" >"$m/srcap/settings-reference.md" +cmp -s "$DOCS/settings-reference.md" "$m/srcap/settings-reference.md" && fail "case 52: srcap edits settings-reference" "the sed script changed nothing" +out=$(run "$m" --json --docs-dir "$m/srcap" 2>&1) || true +assert_eq "case 52: a cap stated on settings-reference is not read" "" "$(fb_rows "$out" fallbackModel)" # A page that states no cap: one skip row, and no finding is invented from a number. -mkdir -p "$m/nocap" -cp "$DOCS/llms.txt" "$DOCS/env-vars.md" "$m/nocap/" -grep -v 'keeps at most three distinct' "$DOCS/settings-reference.md" >"$m/nocap/settings-reference.md" +mc_dir "$m/nocap" '/caps chains at three models/d' printf '%s\n' "$CLEAN_SETTINGS" | jq '. + {fallbackModel:["a","b","c","d","e","f"]}' >"$m/project/.claude/settings.json" out=$(run "$m" --json --docs-dir "$m/nocap" 2>&1) || true assert_eq "case 52: a page with no cap gives one skip row" "skip fallbackModel-cap" "$(fb_rows "$out" fallbackModel)" assert_eq "case 52: and no fallbackModel finding" "0" "$(jq '[.findings[] | select(.identity.claim | startswith("fallbackModel"))] | length' <<<"$out")" +# A page without the section, or not read at all: not inspectable, never clean. +mc_dir "$m/nosection" 's/^### Fallback model chains$/### Backup models/' +out=$(run "$m" --json --docs-dir "$m/nosection" 2>&1) || true +assert_eq "case 52: a page without the section is not inspectable" "not-inspectable fallbackModel-cap" "$(fb_rows "$out" fallbackModel)" +mkdir -p "$m/nomc" +cp "$DOCS/env-vars.md" "$DOCS/settings-reference.md" "$m/nomc/" +grep -v 'model-config' "$DOCS/llms.txt" >"$m/nomc/llms.txt" +out=$(DOCS_FIXTURE="$m/nomc" run "$m" --json --docs-dir "$m/nomc" 2>&1) || true +assert_eq "case 52: an unread model-config is not inspectable" "not-inspectable fallbackModel-cap" "$(fb_rows "$out" fallbackModel)" +# A file with no fallbackModel requests no model-config page. +printf '%s\n' "$CLEAN_SETTINGS" >"$m/project/.claude/settings.json" +out=$(run "$m" --json 2>&1) || true +assert_eq "case 52: no fallbackModel, no model-config request" "0" "$(jq '[.docs.pages[] | select(.slug=="model-config")] | length' <<<"$out")" +printf '%s\n' "$CLEAN_SETTINGS" | jq '. + {fallbackModel:["a","b","c","d","e","f"]}' >"$m/project/.claude/settings.json" out=$(run "$m" --json 2>&1) || true assert_eq "case 52: six distinct entries above the cap are flagged" "finding fallbackModel-dedup-length:6" "$(fb_rows "$out" fallbackModel-dedup)" assert_eq "case 52: six raw entries above the cap are not decided from the page" "skip fallbackModel-raw-length:6" "$(fb_rows "$out" fallbackModel-raw)" diff --git a/plugins/harness-config/skills/audit/scripts/check-effort-pins.sh b/plugins/harness-config/skills/audit/scripts/check-effort-pins.sh index 6e22cdf74ba..8ab218bb552 100755 --- a/plugins/harness-config/skills/audit/scripts/check-effort-pins.sh +++ b/plugins/harness-config/skills/audit/scripts/check-effort-pins.sh @@ -8,7 +8,7 @@ # "Adjust effort level" section (with its child sections) from the cache's # section map, hashes the three parts a pin rests on (the Levels column table # under "Adjust effort level", the level rows of the "Choose an effort level" -# table, and item 3 of the resolution list, which states each model's default), +# table, and the item of the resolution list that states each model's default), # and compares the hash with a committed baseline. It then lists every pin and # flags the ones a human must re-decide. It never edits a pin and never # rewrites the baseline. @@ -152,7 +152,10 @@ adjust_section() { # parse_page : print Llevel (level set, first-seen order), Arow # (Levels table rows), Crow (level rows of the Choose table) and -# Dline (resolution item 3), or one Ereason when a part is missing. +# Dline (the defaults item), or one Ereason when a part is missing. +# The defaults item is found by content: every numbered item of the section that +# says "default", so the list renumbering cannot point the hash at another item, +# and a second such item is hashed rather than skipped. # Headings inside fenced code are ignored; trailing whitespace is trimmed. parse_page() { awk ' @@ -186,7 +189,7 @@ parse_page() { } if (tbl == 1) tbl = 2 if (line ~ /^[0-9]+\.[ \t]/) { - if (!seen_item3 && line ~ /^3\.[ \t]/) { item = 1; seen_item3 = 1; nD++; D[nD] = line; next } + if (tolower(line) ~ /default/) { item = 1; nD++; D[nD] = line; next } item = 0; next } if (item && line ~ /^[ \t]+[^ \t]/) { nD++; D[nD] = line; next } diff --git a/plugins/harness-config/skills/audit/scripts/check-effort-pins.test.sh b/plugins/harness-config/skills/audit/scripts/check-effort-pins.test.sh index dd9b443f060..5a91e69507e 100755 --- a/plugins/harness-config/skills/audit/scripts/check-effort-pins.test.sh +++ b/plugins/harness-config/skills/audit/scripts/check-effort-pins.test.sh @@ -117,7 +117,7 @@ assert_not_contains "print-baseline: no row text in the baseline" "$(cat "$BASE" # --- slice path: the hash the whole-page scan gave on the committed fixture --------- # The hashed lines, copied by hand from the fixture: the level rows of the Choose -# table, the Levels table rows, then item 3 of the list. +# table, the Levels table rows, then the list item that states the defaults. want="$(printf '%s\n' \ '| `low` | Sorting socks by color |' \ '| `medium` | Baking bread from a known recipe |' \ @@ -198,7 +198,7 @@ run "$(variant levels 's/^| Model Gamma | `low`, `medium`, `high`, `max` |$/| Mo assert_eq "changed-table, Levels row: exits 1" 1 "$RC" assert_contains "changed-table, Levels row: changed" "$OUT" "table status=changed" -# --- changed-defaults: item 3 of the resolution list changed ----------------------- +# --- changed-defaults: the resolution-list item that states the defaults changed ---- run "$(variant defaults 's/Model Gamma at `low`/Model Gamma at `high`/')" --baseline "$BASE" --root "$ROOT" assert_eq "changed-defaults: exits 1" 1 "$RC" assert_contains "changed-defaults: table line is changed" "$OUT" "table status=changed" @@ -206,6 +206,35 @@ assert_eq "changed-defaults: every pin flagged table-changed" 8 "$(count "$OUT" run "$(variant item4 's/a coin toss/a dice roll/')" --baseline "$BASE" --root "$ROOT" assert_contains "changed-defaults: another list item is not hashed" "$OUT" "table status=same" +# The defaults item is found by what it says, wherever the list numbers it: new +# sources above it push it to item 5, and item 3 becomes a line with no defaults. +MOVE_DEFAULTS='/^3\. Pretend default source/{ +s/^3\. /5. / +i\ +3. Pretend flag source: a switch at launch +}' +run "$(variant moved "$MOVE_DEFAULTS")" --print-baseline --root "$EMPTY" +assert_eq "defaults moved: still parses" 0 "$RC" +want="$(printf '%s\n' \ + '| `low` | Sorting socks by color |' \ + '| `medium` | Baking bread from a known recipe |' \ + '| `high` | Tuning a piano with a friend listening |' \ + '| `xhigh` | Charting a coastline by moonlight |' \ + '| `max` | Solving a locked-room puzzle alone |' \ + '| Model Alpha and Model Beta | `low`, `medium`, `high`, `xhigh`, `max` |' \ + '| Model Gamma | `low`, `medium`, `high`, `max` |' \ + '5. Pretend default source: Model Alpha starts at `medium`, Model Beta at `high`, Model Gamma at `low`' | + sha256sum | cut -d' ' -f1)" +assert_contains "defaults moved: the hash covers the defaults line, not item 3" "$OUT" "sha256=$want" +MOVED_BASE="$T/moved.baseline" +printf '%s\n' "$OUT" >"$MOVED_BASE" +run "$(variant moved-edited "$MOVE_DEFAULTS;"'s/Model Gamma at `low`/Model Gamma at `high`/')" --baseline "$MOVED_BASE" --root "$ROOT" +assert_contains "defaults moved: an edit to the defaults line still changes the hash" "$OUT" "table status=changed" + +# A second item that mentions a default is hashed too, never skipped for the first. +run "$(variant two-defaults 's/a coin toss/a default coin toss/')" --baseline "$BASE" --root "$ROOT" +assert_contains "two default items: a change to either one is seen" "$OUT" "table status=changed" + # --- failed-fetch: the page cannot be read --------------------------------------- mkdir -p "$T/v-noindex" printf '%s\n' '# Synthetic docs index' '- [Other](https://code.claude.com/docs/en/other.md): other' >"$T/v-noindex/llms.txt" @@ -231,9 +260,13 @@ assert_eq "reshaped, Choose heading renamed: exits 3" 3 "$RC" assert_contains "reshaped, Choose heading renamed: unparsed" "$OUT" "table status=unparsed" assert_contains "reshaped, Choose heading renamed: reason" "$OUT" "reason=no-choose-heading" assert_eq "reshaped, Choose heading renamed: no pin line" 0 "$(count "$OUT" '^pin ')" -run "$(variant noitem3 '/^3\. /d')" --baseline "$BASE" --root "$ROOT" -assert_eq "reshaped, no item 3: exits 3" 3 "$RC" -assert_contains "reshaped, no item 3: reason" "$OUT" "reason=no-defaults-item" +run "$(variant nodefaults '/^3\. /d')" --baseline "$BASE" --root "$ROOT" +assert_eq "reshaped, no defaults item: exits 3" 3 "$RC" +assert_contains "reshaped, no defaults item: reason" "$OUT" "reason=no-defaults-item" +# Item 3 is still there but no longer states the defaults: no claim, not a stale hash. +run "$(variant reworded 's/^3\. Pretend default source.*/3. Pretend flag source: a switch at launch/')" --baseline "$BASE" --root "$ROOT" +assert_eq "reshaped, item 3 states no default: exits 3" 3 "$RC" +assert_contains "reshaped, item 3 states no default: reason" "$OUT" "reason=no-defaults-item" run "$(variant nocol 's/^| Model | Levels |$/| Model | Options |/')" --baseline "$BASE" --root "$ROOT" assert_eq "reshaped, Levels column renamed: exits 3" 3 "$RC" assert_contains "reshaped, Levels column renamed: reason" "$OUT" "reason=no-levels-column"