diff --git a/.agents/skills/backlog-burndown/SKILL.md b/.agents/skills/backlog-burndown/SKILL.md index ef2073f14..1e5dede7c 100644 --- a/.agents/skills/backlog-burndown/SKILL.md +++ b/.agents/skills/backlog-burndown/SKILL.md @@ -326,12 +326,13 @@ A worker still live, a promotion fix included, keeps its worktree until the next ### Choosing the Worker's Model Tier -`AGENTS.md` "Delegation" owns the model-tier rule and names the tiers. Read it there. This is only -what it leaves to judgment here. The tier is chosen per group rather than defaulted, since a +`AGENTS.md` "Model Tiers" owns the model-tier rule and names the tiers, carried under "Model Tiers" +below. This is only what it leaves to judgment here. The tier is chosen per group rather than defaulted, since a stronger tier produces better work and takes fewer review rounds to land it. That often costs less than a cheaper worker looping. The tier is `frontier` or `standard`, never `bulk`, since every worker judges its pull request's review findings. It is `highest` only where the maintainer -named it for this run. Groups of these kinds are never tiered +named it for this run and launched the session with it allowed, per "Model Tiers" below. Where a +dispatch at it is refused, ask the maintainer rather than dispatching lower. Groups of these kinds are never tiered below `frontier`: - One touching **carried canonical content**, as `GOVERNANCE.md` "Verification Discipline" bounds @@ -536,6 +537,31 @@ Report at every round boundary and at each ending: what merged to develop, what request carries, what was newly filed, what is stopped and on which question, what was held back and on which blocker, and what the next round would pick. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Mechanics Live Elsewhere - The review loop, the Merge Gate, the finding outcomes, and when a finding is a question: diff --git a/.agents/skills/local-strict-review/SKILL.md b/.agents/skills/local-strict-review/SKILL.md index fac8dfd41..144f2d89c 100644 --- a/.agents/skills/local-strict-review/SKILL.md +++ b/.agents/skills/local-strict-review/SKILL.md @@ -72,7 +72,7 @@ Bounds: read-only. No edit, no stage, no commit, no push, no PR-hosted write of Before dispatching, grep the tree for other statements of each rule the diff adds or changes, and add each file holding one to the `Paths:` floor, so a statement the diff has put in disagreement is read rather than missed. -**Model tier:** `frontier`, the tier `AGENTS.md` "Delegation" names for adversarial review, which this pass is. Where `frontier` is not reachable, run the pass on the strongest tier below it that is, even the one that authored the change. A second, adversarially-prompted look still catches what the authoring pass's own "looks ready" judgment did not. +**Model tier:** `frontier`, the tier "Model Tiers" below names for adversarial review, which this pass is. Where `frontier` is not reachable, run the pass on the strongest tier below it that is, even the one that authored the change. A second, adversarially-prompted look still catches what the authoring pass's own "looks ready" judgment did not. Reachable means the tier this session can name when it dispatches the reviewer, rather than the tier this session is itself running on. A session deliberately tiered down for execution work, a worker dispatched by an orchestrator being the ordinary case, names `frontier` for the reviewer where its harness lets it, since tiering down the author is the reason the reviewer must not follow it down. What a given harness and account actually permit varies, so treat this as the tier to ask for rather than one to assume. Where a dispatch exposes no way to name a tier, take what it gives and run the pass, for the reason the fallback above states. A seat that cannot dispatch a subagent hands the pass to whoever dispatched it. The seat commits, runs `python3 /scripts/local_review.py status --target ` in its worktree, and hands back its worktree path, target, and `contentDigest`. The dispatcher fills the brief's `` slot and runs the reviewer. It then runs the record command from "Recording the Pass" in that worktree, with `--reviewer agent-skill --target --expect-digest `. It returns the findings. Each fix commit repeats this cycle, and the seat pushes only when a record matches its content. The brief dispatching the seat states whether the dispatcher can continue that seat. Where the dispatcher cannot, or under `unattended-handoff`, the seat stops, reports, and does not push. Where nobody dispatched the seat, it reports to the maintainer that it could not run the pass, and does not push. Either way the push waits rather than the pass being quietly skipped. The headless `run --backend` route under "Recording the Pass" is not the substitute: it runs a vendor CLI against its own review, which never carries the brief above, so it satisfies the rule this section states only where that separate route is what a capture point asked for. @@ -176,6 +176,31 @@ In the hub, `.husky/pre-push` checks the receipt at the push itself, so the mome This table is the fleet's one enumeration of these, and every other surface states the principle and routes here rather than listing the shapes. That is deliberate: every review round that added a shape also left a restatement of it somewhere else, and keeping one table is what stops the next round doing the same. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Mechanics Live Elsewhere - Review criteria: `fleet-code-review`. diff --git a/.agents/skills/unattended-handoff/SKILL.md b/.agents/skills/unattended-handoff/SKILL.md index 42aea3bb3..218c4d766 100644 --- a/.agents/skills/unattended-handoff/SKILL.md +++ b/.agents/skills/unattended-handoff/SKILL.md @@ -90,22 +90,25 @@ maintainer starts a second run on a repository only once the first has ended. The orchestrator first resolves `/` from the checkout's `origin`, or takes it from the invocation. Apart from the timer "A Dispatch That Fails" names, that is the one command it runs. -From then on it holds only the scope and any `highest` the invocation named, the round count, and +From then on it holds only the scope and any allowed `highest` the invocation named, the round count, and the dispatched handoff numbers. It also holds each round's one-line outcome and the waits spent on a dispatch being retried. Each round runs: -1. **Dispatch the picker** at `frontier` with the brief below, or at `highest` where the - maintainer's invocation named that tier or its model. Its ranking, decision, and worker-tier +1. **Dispatch the picker** at `frontier` with the brief below. Use `highest` only where the + invocation named that tier or its model and the session was launched with it allowed, per + "Model Tiers" below. A refusal there ends the run the way step 3's does. Its ranking, decision, and worker-tier calls are the ones `backlog-burndown` keeps in its orchestrator, and here no maintainer reviews them. Wait on the dispatch mechanism's own completion signal rather than polling. 2. **Read its one line.** `NONE` or `STOP` ends the run, and so does any reply that is not a picker line below. A handoff number already dispatched in this run ends it too, since a handoff a worker neither closed nor parked means the worker failed in a way this seat must not investigate. -3. **Dispatch the worker** on that handoff and track, at the model `AGENTS.md` "Delegation" names +3. **Dispatch the worker** on that handoff and track, at the model "Model Tiers" below names for the tier in the picker's line. Wait the same way. Dispatch at `highest` instead only where - the maintainer's invocation named that tier or its model. + the invocation named that tier or its model and the session was launched with it allowed, per + "Model Tiers" below. Without that allowance every such dispatch is refused. The run then ends + and reports the refusal rather than continuing at a tier nobody chose. 4. **Record its one line** and start the next round. `STOP` ends the run, and so does a reply that is not one of the lines below, rather than being read further. @@ -180,6 +183,31 @@ Under `main` or `release` every unblocked round promotes, so each promotion ordi fix. A change another session merged to develop meanwhile rides along with it. Naming the scope accepts that. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Auto-Resolvable An issue needs no maintainer decision when every one of these holds. Where any is unclear, it diff --git a/.claude-plugin/fleet-skills/.source-digests/backlog-burndown b/.claude-plugin/fleet-skills/.source-digests/backlog-burndown index 437896f66..2d14be623 100644 --- a/.claude-plugin/fleet-skills/.source-digests/backlog-burndown +++ b/.claude-plugin/fleet-skills/.source-digests/backlog-burndown @@ -1 +1 @@ -1bee0d96e7d55aac +e1af7dadfb2070a2 diff --git a/.claude-plugin/fleet-skills/.source-digests/local-strict-review b/.claude-plugin/fleet-skills/.source-digests/local-strict-review index 32f161ade..b2cd43dd9 100644 --- a/.claude-plugin/fleet-skills/.source-digests/local-strict-review +++ b/.claude-plugin/fleet-skills/.source-digests/local-strict-review @@ -1 +1 @@ -6da3e35f478872a6 +9dc270f21b0302cd diff --git a/.claude-plugin/fleet-skills/.source-digests/unattended-handoff b/.claude-plugin/fleet-skills/.source-digests/unattended-handoff index 752540e52..106cab165 100644 --- a/.claude-plugin/fleet-skills/.source-digests/unattended-handoff +++ b/.claude-plugin/fleet-skills/.source-digests/unattended-handoff @@ -1 +1 @@ -151cf349b4115615 +eddb00a5222e6b31 diff --git a/.claude-plugin/fleet-skills/skills/backlog-burndown/SKILL.md b/.claude-plugin/fleet-skills/skills/backlog-burndown/SKILL.md index ef2073f14..1e5dede7c 100644 --- a/.claude-plugin/fleet-skills/skills/backlog-burndown/SKILL.md +++ b/.claude-plugin/fleet-skills/skills/backlog-burndown/SKILL.md @@ -326,12 +326,13 @@ A worker still live, a promotion fix included, keeps its worktree until the next ### Choosing the Worker's Model Tier -`AGENTS.md` "Delegation" owns the model-tier rule and names the tiers. Read it there. This is only -what it leaves to judgment here. The tier is chosen per group rather than defaulted, since a +`AGENTS.md` "Model Tiers" owns the model-tier rule and names the tiers, carried under "Model Tiers" +below. This is only what it leaves to judgment here. The tier is chosen per group rather than defaulted, since a stronger tier produces better work and takes fewer review rounds to land it. That often costs less than a cheaper worker looping. The tier is `frontier` or `standard`, never `bulk`, since every worker judges its pull request's review findings. It is `highest` only where the maintainer -named it for this run. Groups of these kinds are never tiered +named it for this run and launched the session with it allowed, per "Model Tiers" below. Where a +dispatch at it is refused, ask the maintainer rather than dispatching lower. Groups of these kinds are never tiered below `frontier`: - One touching **carried canonical content**, as `GOVERNANCE.md` "Verification Discipline" bounds @@ -536,6 +537,31 @@ Report at every round boundary and at each ending: what merged to develop, what request carries, what was newly filed, what is stopped and on which question, what was held back and on which blocker, and what the next round would pick. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Mechanics Live Elsewhere - The review loop, the Merge Gate, the finding outcomes, and when a finding is a question: diff --git a/.claude-plugin/fleet-skills/skills/local-strict-review/SKILL.md b/.claude-plugin/fleet-skills/skills/local-strict-review/SKILL.md index fac8dfd41..144f2d89c 100644 --- a/.claude-plugin/fleet-skills/skills/local-strict-review/SKILL.md +++ b/.claude-plugin/fleet-skills/skills/local-strict-review/SKILL.md @@ -72,7 +72,7 @@ Bounds: read-only. No edit, no stage, no commit, no push, no PR-hosted write of Before dispatching, grep the tree for other statements of each rule the diff adds or changes, and add each file holding one to the `Paths:` floor, so a statement the diff has put in disagreement is read rather than missed. -**Model tier:** `frontier`, the tier `AGENTS.md` "Delegation" names for adversarial review, which this pass is. Where `frontier` is not reachable, run the pass on the strongest tier below it that is, even the one that authored the change. A second, adversarially-prompted look still catches what the authoring pass's own "looks ready" judgment did not. +**Model tier:** `frontier`, the tier "Model Tiers" below names for adversarial review, which this pass is. Where `frontier` is not reachable, run the pass on the strongest tier below it that is, even the one that authored the change. A second, adversarially-prompted look still catches what the authoring pass's own "looks ready" judgment did not. Reachable means the tier this session can name when it dispatches the reviewer, rather than the tier this session is itself running on. A session deliberately tiered down for execution work, a worker dispatched by an orchestrator being the ordinary case, names `frontier` for the reviewer where its harness lets it, since tiering down the author is the reason the reviewer must not follow it down. What a given harness and account actually permit varies, so treat this as the tier to ask for rather than one to assume. Where a dispatch exposes no way to name a tier, take what it gives and run the pass, for the reason the fallback above states. A seat that cannot dispatch a subagent hands the pass to whoever dispatched it. The seat commits, runs `python3 /scripts/local_review.py status --target ` in its worktree, and hands back its worktree path, target, and `contentDigest`. The dispatcher fills the brief's `` slot and runs the reviewer. It then runs the record command from "Recording the Pass" in that worktree, with `--reviewer agent-skill --target --expect-digest `. It returns the findings. Each fix commit repeats this cycle, and the seat pushes only when a record matches its content. The brief dispatching the seat states whether the dispatcher can continue that seat. Where the dispatcher cannot, or under `unattended-handoff`, the seat stops, reports, and does not push. Where nobody dispatched the seat, it reports to the maintainer that it could not run the pass, and does not push. Either way the push waits rather than the pass being quietly skipped. The headless `run --backend` route under "Recording the Pass" is not the substitute: it runs a vendor CLI against its own review, which never carries the brief above, so it satisfies the rule this section states only where that separate route is what a capture point asked for. @@ -176,6 +176,31 @@ In the hub, `.husky/pre-push` checks the receipt at the push itself, so the mome This table is the fleet's one enumeration of these, and every other surface states the principle and routes here rather than listing the shapes. That is deliberate: every review round that added a shape also left a restatement of it somewhere else, and keeping one table is what stops the next round doing the same. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Mechanics Live Elsewhere - Review criteria: `fleet-code-review`. diff --git a/.claude-plugin/fleet-skills/skills/unattended-handoff/SKILL.md b/.claude-plugin/fleet-skills/skills/unattended-handoff/SKILL.md index 42aea3bb3..218c4d766 100644 --- a/.claude-plugin/fleet-skills/skills/unattended-handoff/SKILL.md +++ b/.claude-plugin/fleet-skills/skills/unattended-handoff/SKILL.md @@ -90,22 +90,25 @@ maintainer starts a second run on a repository only once the first has ended. The orchestrator first resolves `/` from the checkout's `origin`, or takes it from the invocation. Apart from the timer "A Dispatch That Fails" names, that is the one command it runs. -From then on it holds only the scope and any `highest` the invocation named, the round count, and +From then on it holds only the scope and any allowed `highest` the invocation named, the round count, and the dispatched handoff numbers. It also holds each round's one-line outcome and the waits spent on a dispatch being retried. Each round runs: -1. **Dispatch the picker** at `frontier` with the brief below, or at `highest` where the - maintainer's invocation named that tier or its model. Its ranking, decision, and worker-tier +1. **Dispatch the picker** at `frontier` with the brief below. Use `highest` only where the + invocation named that tier or its model and the session was launched with it allowed, per + "Model Tiers" below. A refusal there ends the run the way step 3's does. Its ranking, decision, and worker-tier calls are the ones `backlog-burndown` keeps in its orchestrator, and here no maintainer reviews them. Wait on the dispatch mechanism's own completion signal rather than polling. 2. **Read its one line.** `NONE` or `STOP` ends the run, and so does any reply that is not a picker line below. A handoff number already dispatched in this run ends it too, since a handoff a worker neither closed nor parked means the worker failed in a way this seat must not investigate. -3. **Dispatch the worker** on that handoff and track, at the model `AGENTS.md` "Delegation" names +3. **Dispatch the worker** on that handoff and track, at the model "Model Tiers" below names for the tier in the picker's line. Wait the same way. Dispatch at `highest` instead only where - the maintainer's invocation named that tier or its model. + the invocation named that tier or its model and the session was launched with it allowed, per + "Model Tiers" below. Without that allowance every such dispatch is refused. The run then ends + and reports the refusal rather than continuing at a tier nobody chose. 4. **Record its one line** and start the next round. `STOP` ends the run, and so does a reply that is not one of the lines below, rather than being read further. @@ -180,6 +183,31 @@ Under `main` or `release` every unblocked round promotes, so each promotion ordi fix. A change another session merged to develop meanwhile rides along with it. Naming the scope accepts that. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Auto-Resolvable An issue needs no maintainer decision when every one of these holds. Where any is unclear, it diff --git a/.github/actions/validate/action.yml b/.github/actions/validate/action.yml index 7b8edd8c6..ba9f169c3 100644 --- a/.github/actions/validate/action.yml +++ b/.github/actions/validate/action.yml @@ -34,6 +34,8 @@ runs: uvx --with-requirements tests/requirements.txt coverage@latest run --source=scripts,spec,host-setup --append spec/workflow_reuse.py --selftest uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/gh-write-guard.py --selftest uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/test_install.py + uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/model-tier-guard.py --selftest + uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/test_model_tier_guard.py uvx coverage@latest report # Read-only: fails if a generated skill distribution differs from .agents/skills/. diff --git a/.github/skills/backlog-burndown/SKILL.md b/.github/skills/backlog-burndown/SKILL.md index ef2073f14..1e5dede7c 100644 --- a/.github/skills/backlog-burndown/SKILL.md +++ b/.github/skills/backlog-burndown/SKILL.md @@ -326,12 +326,13 @@ A worker still live, a promotion fix included, keeps its worktree until the next ### Choosing the Worker's Model Tier -`AGENTS.md` "Delegation" owns the model-tier rule and names the tiers. Read it there. This is only -what it leaves to judgment here. The tier is chosen per group rather than defaulted, since a +`AGENTS.md` "Model Tiers" owns the model-tier rule and names the tiers, carried under "Model Tiers" +below. This is only what it leaves to judgment here. The tier is chosen per group rather than defaulted, since a stronger tier produces better work and takes fewer review rounds to land it. That often costs less than a cheaper worker looping. The tier is `frontier` or `standard`, never `bulk`, since every worker judges its pull request's review findings. It is `highest` only where the maintainer -named it for this run. Groups of these kinds are never tiered +named it for this run and launched the session with it allowed, per "Model Tiers" below. Where a +dispatch at it is refused, ask the maintainer rather than dispatching lower. Groups of these kinds are never tiered below `frontier`: - One touching **carried canonical content**, as `GOVERNANCE.md` "Verification Discipline" bounds @@ -536,6 +537,31 @@ Report at every round boundary and at each ending: what merged to develop, what request carries, what was newly filed, what is stopped and on which question, what was held back and on which blocker, and what the next round would pick. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Mechanics Live Elsewhere - The review loop, the Merge Gate, the finding outcomes, and when a finding is a question: diff --git a/.github/skills/local-strict-review/SKILL.md b/.github/skills/local-strict-review/SKILL.md index fac8dfd41..144f2d89c 100644 --- a/.github/skills/local-strict-review/SKILL.md +++ b/.github/skills/local-strict-review/SKILL.md @@ -72,7 +72,7 @@ Bounds: read-only. No edit, no stage, no commit, no push, no PR-hosted write of Before dispatching, grep the tree for other statements of each rule the diff adds or changes, and add each file holding one to the `Paths:` floor, so a statement the diff has put in disagreement is read rather than missed. -**Model tier:** `frontier`, the tier `AGENTS.md` "Delegation" names for adversarial review, which this pass is. Where `frontier` is not reachable, run the pass on the strongest tier below it that is, even the one that authored the change. A second, adversarially-prompted look still catches what the authoring pass's own "looks ready" judgment did not. +**Model tier:** `frontier`, the tier "Model Tiers" below names for adversarial review, which this pass is. Where `frontier` is not reachable, run the pass on the strongest tier below it that is, even the one that authored the change. A second, adversarially-prompted look still catches what the authoring pass's own "looks ready" judgment did not. Reachable means the tier this session can name when it dispatches the reviewer, rather than the tier this session is itself running on. A session deliberately tiered down for execution work, a worker dispatched by an orchestrator being the ordinary case, names `frontier` for the reviewer where its harness lets it, since tiering down the author is the reason the reviewer must not follow it down. What a given harness and account actually permit varies, so treat this as the tier to ask for rather than one to assume. Where a dispatch exposes no way to name a tier, take what it gives and run the pass, for the reason the fallback above states. A seat that cannot dispatch a subagent hands the pass to whoever dispatched it. The seat commits, runs `python3 /scripts/local_review.py status --target ` in its worktree, and hands back its worktree path, target, and `contentDigest`. The dispatcher fills the brief's `` slot and runs the reviewer. It then runs the record command from "Recording the Pass" in that worktree, with `--reviewer agent-skill --target --expect-digest `. It returns the findings. Each fix commit repeats this cycle, and the seat pushes only when a record matches its content. The brief dispatching the seat states whether the dispatcher can continue that seat. Where the dispatcher cannot, or under `unattended-handoff`, the seat stops, reports, and does not push. Where nobody dispatched the seat, it reports to the maintainer that it could not run the pass, and does not push. Either way the push waits rather than the pass being quietly skipped. The headless `run --backend` route under "Recording the Pass" is not the substitute: it runs a vendor CLI against its own review, which never carries the brief above, so it satisfies the rule this section states only where that separate route is what a capture point asked for. @@ -176,6 +176,31 @@ In the hub, `.husky/pre-push` checks the receipt at the push itself, so the mome This table is the fleet's one enumeration of these, and every other surface states the principle and routes here rather than listing the shapes. That is deliberate: every review round that added a shape also left a restatement of it somewhere else, and keeping one table is what stops the next round doing the same. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Mechanics Live Elsewhere - Review criteria: `fleet-code-review`. diff --git a/.github/skills/unattended-handoff/SKILL.md b/.github/skills/unattended-handoff/SKILL.md index 42aea3bb3..218c4d766 100644 --- a/.github/skills/unattended-handoff/SKILL.md +++ b/.github/skills/unattended-handoff/SKILL.md @@ -90,22 +90,25 @@ maintainer starts a second run on a repository only once the first has ended. The orchestrator first resolves `/` from the checkout's `origin`, or takes it from the invocation. Apart from the timer "A Dispatch That Fails" names, that is the one command it runs. -From then on it holds only the scope and any `highest` the invocation named, the round count, and +From then on it holds only the scope and any allowed `highest` the invocation named, the round count, and the dispatched handoff numbers. It also holds each round's one-line outcome and the waits spent on a dispatch being retried. Each round runs: -1. **Dispatch the picker** at `frontier` with the brief below, or at `highest` where the - maintainer's invocation named that tier or its model. Its ranking, decision, and worker-tier +1. **Dispatch the picker** at `frontier` with the brief below. Use `highest` only where the + invocation named that tier or its model and the session was launched with it allowed, per + "Model Tiers" below. A refusal there ends the run the way step 3's does. Its ranking, decision, and worker-tier calls are the ones `backlog-burndown` keeps in its orchestrator, and here no maintainer reviews them. Wait on the dispatch mechanism's own completion signal rather than polling. 2. **Read its one line.** `NONE` or `STOP` ends the run, and so does any reply that is not a picker line below. A handoff number already dispatched in this run ends it too, since a handoff a worker neither closed nor parked means the worker failed in a way this seat must not investigate. -3. **Dispatch the worker** on that handoff and track, at the model `AGENTS.md` "Delegation" names +3. **Dispatch the worker** on that handoff and track, at the model "Model Tiers" below names for the tier in the picker's line. Wait the same way. Dispatch at `highest` instead only where - the maintainer's invocation named that tier or its model. + the invocation named that tier or its model and the session was launched with it allowed, per + "Model Tiers" below. Without that allowance every such dispatch is refused. The run then ends + and reports the refusal rather than continuing at a tier nobody chose. 4. **Record its one line** and start the next round. `STOP` ends the run, and so does a reply that is not one of the lines below, rather than being read further. @@ -180,6 +183,31 @@ Under `main` or `release` every unblocked round promotes, so each promotion ordi fix. A change another session merged to develop meanwhile rides along with it. Naming the scope accepts that. +## Model Tiers + +`AGENTS.md` "Model Tiers" keeps the tier rule, carried here whole as a generated include. A tier +this skill names then resolves to its model even where a carried `AGENTS.md` predates the table. + + + +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. + +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. +- **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. +- **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. + +| Tool | `highest` | `frontier` | `standard` | `bulk` | +| --- | --- | --- | --- | --- | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. + +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + + + ## Auto-Resolvable An issue needs no maintainer decision when every one of these holds. Where any is unclear, it diff --git a/AGENTS.md b/AGENTS.md index e9cdfe57f..4138e0455 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,20 +54,25 @@ An agent session is billed on the context it carries, not the work it does. Ever - **Report an execution boundary separately from a check finding.** A denied path, network request, or Docker socket says the check did not run. Preserve that failure, then use the executor's approval mechanism for the required rerun. Request the narrowest reusable command prefix the executor supports. Report the rerun's result as the verification evidence. - **Bound every probe, and prove it ended.** A throwaway probe or test harness runs under `timeout`, and one that can recurse or fan out also runs under a task and memory cap. Start it in a process group of its own, as `setsid --wait timeout ` on Linux, and end it by that group with `kill -- -`. Without `--wait`, `setsid` returns at once when its caller leads a process group, and the probe runs on detached. A `pkill -f` pattern that matches nothing exits 1 and kills nothing. Then confirm with `pgrep` that nothing from any earlier run survives, since a later clean run proves nothing about an earlier one. Never suppress a cleanup command's output or exit status. A command the harness moved to the background, where no background job was intended, is stopped and confirmed gone before work continues. -### Delegation +### Model Tiers -A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below alone. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. +A rule, a Skill, or a brief names a model tier, never a model. A model release is then an edit to the table below and to the hub's `spec/model-tiers.json`. That file states the same mapping as data, and the hub's spec gate holds the two equal. Reasoning effort is a separate axis stated beside the tier, such as `bulk` at low effort. -- **`frontier`** is the strongest reasoning the fleet uses by default, for rule text, design, spec logic, and adversarial review. +- **`frontier`** is the fleet's default reasoning tier, for orchestrating, rule text, design, spec logic, and adversarial review. It is not the most capable model available, which is `highest`. - **`standard`** is ordinary implementation and driving a pull request. Judging a review finding is never mechanical, however often it repeats, so it runs at `standard` or above. A `standard` driver's evidence-backed decline of a finding raised on its pull request stands, since that pull request's reviewers and its merge gate check it. - **`bulk`** is the cheapest model that does the work correctly. It is for high fan-out mechanical work, such as an extraction or a status check. -- **`highest`** is the most capable model available. Only the maintainer selects it, naming the tier or the model in the session or the invocation. +- **`highest`** is the most capable model available, and only the maintainer selects it. The maintainer chooses it for the session itself, or allows it for the session's subagents at launch. The host's agent-safety kit reads that launch setting, `FLEET_ALLOW_HIGHEST=1` on Claude Code. Naming the tier in a prompt does not set it. A maintainer naming it partway through a session restarts the session with it allowed, and an agent never sets it. Without that setting, the kit refuses a dispatch that names it. A dispatch naming no model runs at its agent definition's model where one is set, and otherwise at the session's own. A fork always runs at the session's model, whatever it names. So a session launched on `highest` names a tier on every dispatch meant to run below it, and uses no fork for such work. A session never reaches for it because a task looks hard. | Tool | `highest` | `frontier` | `standard` | `bulk` | | --- | --- | --- | --- | --- | -| Claude Code | Fable | Opus | Sonnet | Haiku | +| Claude Code | `fable` | `opus` | `sonnet` | `haiku` | + +Each cell is the name that tool's subagent dispatch takes, which follows the latest model of its family. A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. -A tool with no row here, or one that cannot name a model per subagent, runs every tier at the session's own model. +- **A tier resolves only through this table.** A word such as "frontier", "strongest", or "most capable" never resolves to `highest`. The industry's "frontier model" is the opposite of what this tier means here. A session that cannot see this table names no model rather than guessing one, so the subagent runs at the session's own model. +- **An orchestrating seat runs at `frontier` and names a tier on every dispatch.** It defaults a subagent to `frontier`, and picks `standard` or `bulk` deliberately for execution or mechanical work. A dispatch that names no model inherits whatever the session runs on, which is a choice nobody made. + +### Delegation - **Delegate exploration, keep judgment.** A subagent starts from an empty context and returns only its conclusion, so a wide search, a multi-file audit, or a "which of these is affected" question costs a fraction of the same work inline. Delegate when the finding compresses to a short answer, and stay inline when the intermediate detail drives the next edit. - **Match the model tier to the judgment, not to the diff size.** Mechanical work (a known-shape edit repeated across files, an extraction, a status check, a lint fix) runs at `bulk`, at the lowest effort that holds. State the tier in the delegation itself rather than accepting the default. A change to a gate, a ruleset, a release condition, or a carried governance section is a design change however small it looks. diff --git a/OPERATIONS.md b/OPERATIONS.md index 75e028de0..9048d82e7 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -26,6 +26,8 @@ uvx --with-requirements tests/requirements.txt coverage@latest run --source=scri uvx --with-requirements tests/requirements.txt coverage@latest run --source=scripts,spec,host-setup --append spec/workflow_reuse.py --selftest uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/gh-write-guard.py --selftest uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/test_install.py +uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/model-tier-guard.py --selftest +uvx coverage@latest run --source=scripts,spec,host-setup --append host-setup/agent-safety/claude/test_model_tier_guard.py uvx coverage@latest report uv run --no-project --script scripts/build_dist.py --check uv run --no-project --script scripts/canonical_review.py status > /dev/null diff --git a/host-setup/agent-safety/README.md b/host-setup/agent-safety/README.md index a7198e28f..fe6781482 100644 --- a/host-setup/agent-safety/README.md +++ b/host-setup/agent-safety/README.md @@ -3,7 +3,7 @@ What any coding agent must be stopped from doing when it runs unattended on this host, stated once, independent of which agent implements it. Most of it guards the maintainer's `gh` credentials, which is where the harm started. Requirements 7 to 9 guard the machine itself, which is the other thing -an unattended agent takes down. This file is the source +an unattended agent takes down. Requirement 10 guards the maintainer's model choice and quota. This file is the source of truth: an implementation is built from the requirements below, and an implementation is audited by checking its decisions against them, not by reading its source as the implicit spec. @@ -32,7 +32,7 @@ Requirements 1 to 7 judge a command's text, read the way a simple lexer reads it 1. **A GitHub write with its output discarded or forced to success is denied.** The write is a mutating `gh` call or a `git push`. Its output goes to `/dev/null` (`>/dev/null`, `2>/dev/null`, `&>/dev/null`), or `|| true`, `|| :`, or `|| echo` follows it. Either hides the one signal that tells a client-reported failure apart from a server-side success. Deny it, so the write runs again with its result visible. A tail on a wrapper shell or an `eval` counts for the write inside it. A tail the reading does not attach to the write, such as one on an enclosing group, is allowed. 2. **A GraphQL mutation carrying a literal GitHub node id is denied.** Node ids resolve globally, so a fabricated, stale, or hand-typed id can land on a real object in a different repository. A literal id is an uppercase-letter prefix such as `PR_`, `PRRT_`, `IC_`, or `BOT_` followed by a long body, or the legacy `MD`-prefixed base64 form. It is denied as a `-f`/`-F` field value of a `gh api graphql` mutation, and as a quoted string in the mutation's query text. A `-F name="$VAR"` value is allowed. A static hook cannot see where a variable's value came from, and this rule's job is the literal-id mistake. The fleet's prose rule (`GOVERNANCE.md` "Repository Boundaries and Write Safety") requires that the value came from a live query. A query read from a file, as in `query=@file.graphql`, is not read and is allowed. -3. **A GitHub write with an explicit target outside the checkout's own owner is denied, unless the maintainer granted it.** Each target the write names is compared against the owner of the checkout's `origin`. A sibling repository under the same owner is allowed with no grant, since the harm this guards is reaching a stranger's repository, not working across one maintainer's own fleet. A different owner is allowed only when named in a grant read from the environment the session was launched with. That is a channel the agent cannot set for itself, so an inline `VAR=x cmd` prefix or an `export` in the same call never satisfies it. A write is a known-mutating `gh` subcommand, a `gh api` call whose method is not GET, or a GraphQL mutation. It names a target by any of these: +3. **A GitHub write with an explicit target outside the checkout's own owner is denied, unless the maintainer granted it.** Each target the write names is compared against the owner of the checkout's `origin`. A sibling repository under the same owner is allowed with no grant, since the harm this guards is reaching a stranger's repository, not working across one maintainer's own fleet. A different owner is allowed only when named in a grant read from the environment the session was launched with. A tool call cannot change that environment for its own session. So an inline `VAR=x cmd` prefix or an `export` in the same call never satisfies it. A write is a known-mutating `gh` subcommand, a `gh api` call whose method is not GET, or a GraphQL mutation. It names a target by any of these: - an `-R`/`--repo` value in any of gh's spellings, read by the owner in `[HOST/]OWNER/REPO` - a `repos//` endpoint on a `gh api` write - a `GH_REPO` value set before a write carrying no repo flag, in its prefix, earlier in the text, or in the session's environment @@ -106,6 +106,25 @@ Requirements 1 to 7 judge a command's text, read the way a simple lexer reads it them. They bound an accident rather than an adversary, since a command can still raise its own group's limits or start a group outside it, and the incident this answers was a mistake. +10. **A subagent dispatch at the maintainer-only model tier is denied, unless the maintainer allowed + it when the session started.** A skill named the `frontier` tier in a repository whose + instruction set carried no tier table. It was taken as "the frontier model", and the work ran on + the most capable and most expensive model nobody had asked for. Read the tier mapping from the + hub's `spec/model-tiers.json`, deployed beside the hook. Refuse a dispatch naming the + maintainer-only tier's model unless the session was launched with that tier allowed. Setting the + allowance is the maintainer's act. An agent never does it, for its own session or for one it + launches, and that is conduct the hook does not enforce. A mapping the hook cannot read refuses + every dispatch naming a model, since a guard that cannot tell what to block must not go quiet. A + session launched with the allowance blocks no model, so it needs no mapping. + + This one guards the maintainer's choice of model and his quota rather than the credentials or + the machine. It sits in the host layer all the same, because a dispatch happens inside the + harness, where no hook a repository carries can see it. It refuses only a dispatch that names + the model. A saved workflow, a child workflow a script starts, a computed model, and an agent + definition pinning one all pass. So do a dispatch naming no model and a fork naming another + model, which run at a model the hook cannot read. So does a nested session an agent launches + with the allowance set, which is circumvention rather than the mistake this guards. + ## Decision Flow ```mermaid @@ -149,8 +168,8 @@ flowchart LR ``` The first diagram is this spec's actual decision flow, generalized from `claude/gh-write-guard.py`'s -`classify()`. Requirements 8 and 9 appear nowhere in it, deliberately: neither judges a command's -text. Requirement 9 wraps every command the flow allows, and requirement 8 runs once at the end of a +`classify()`. Requirements 8, 9, and 10 appear nowhere in it, deliberately. None judges a command's +text, and requirement 10 judges a subagent dispatch instead. Requirement 9 wraps every command the flow allows, and requirement 8 runs once at the end of a session on whatever the denials and the ceilings did not prevent. The second is why a failure lands in one layer and not another. A rule that never reached the session at all is a loading bug, fixed the way PR #1081 fixed `local-strict-review`'s missed trigger, by wiring `CLAUDE.md` to import `AGENTS.md`. A rule that reached the session and @@ -185,7 +204,7 @@ every agent. | Agent | Status | Implementation | | --- | --- | --- | -| Claude Code | All 9 requirements, via a `PreToolUse` hook, a shell prefix, and a `SessionEnd` sweep | [`claude/README.md`][claude] | +| Claude Code | Every requirement, via `PreToolUse` hooks, a shell prefix, and a `SessionEnd` sweep | [`claude/README.md`][claude] | | Codex | No hook yet -- tracked at [issue #781][issue-781] | [`codex/README.md`][codex] | | opencode | No hook yet -- tracked at [issue #781][issue-781] | [`opencode/README.md`][opencode] | @@ -196,8 +215,8 @@ local host-safety hazard for this kit to cover. ## Auditing an Implementation Against This Spec Run each of the implementation's own self-tests (`claude/gh-write-guard.py --selftest`, -`claude/stray-process-sweep.py --selftest`, and `claude/tool-containment.py --selftest` for Claude -Code) and compare every case against the +`claude/stray-process-sweep.py --selftest`, `claude/tool-containment.py --selftest`, and +`claude/model-tier-guard.py --selftest` for Claude Code) and compare every case against the requirements list above, one by one, rather than reading the implementation's source as though it were the spec. A case the self-test doesn't cover is a gap in the audit, not evidence the requirement is satisfied. This is the concrete shape of "ask Claude to diff --git a/host-setup/agent-safety/claude/README.md b/host-setup/agent-safety/claude/README.md index 113f8baf3..7bab15ad3 100644 --- a/host-setup/agent-safety/claude/README.md +++ b/host-setup/agent-safety/claude/README.md @@ -12,9 +12,10 @@ installation and operational detail. Codex and opencode have no equivalent hook Into `~/.claude/` (or `%USERPROFILE%\.claude\` on Windows): -- **`hooks/gh-write-guard.py`**: a PreToolUse hook that denies the plain shapes of four classes of dangerous action. First, the GitHub **write** footguns behind the cross-repo comment incident: a GitHub write whose output is discarded or forced to success, a GraphQL mutation passing a **literal** node id instead of a `$variable`, and a `gh` write whose explicit target is under an owner other than the checkout origin's. Sibling repositories under the same owner are allowed, since the harm this guards is reaching a stranger's repository rather than working across one maintainer's own fleet, and a different owner is allowed only when the maintainer names it in `GH_WRITE_GUARD_ALLOW` (an `owner/repo` list, where `owner/*` grants a whole owner). That variable is read from the environment the session was launched with, which is the one channel an agent cannot set for itself: a hook runs as its own process, so an inline `VAR=x cmd` prefix or an `export` inside a Bash call never reaches it. Second, a **git operation that bypasses a repository protection**: `gh pr merge --admin`, a commit or push that skips the hooks by `--no-verify` or by a `core.hooksPath` or include override, and a push the branch's **live** rules forbid. So a code-style `develop` is denied while a config-style `develop` is allowed with no per-repo configuration. A push to a protected-default branch fails closed when its rules cannot be determined. Third, a **mutating git operation run directly against a primary checkout**, the harm behind a second incident where an agent reused the maintainer's own primary checkout instead of a worktree despite having read the prose rule against it. `GH_WRITE_GUARD_ALLOW_PRIMARY_CHECKOUT` grants an exception through the same channel as `GH_WRITE_GUARD_ALLOW`. Fourth, an **unbounded shell wait**, a loop that sleeps with no bound, the harm behind a third incident where seven such loops outlived the subagents that wrote them and the run that dispatched those subagents. The hook reads a command with one simple lexer and denies a shape only where it appears plainly. A shape it does not read is allowed, per the root spec's threat model. It fires even in autonomous / bypass-permissions sessions, which is how the first incident happened. See the root spec's requirements 1 to 7 for the exact rules. +- **`hooks/gh-write-guard.py`**: a PreToolUse hook that denies the plain shapes of four classes of dangerous action. First, the GitHub **write** footguns behind the cross-repo comment incident: a GitHub write whose output is discarded or forced to success, a GraphQL mutation passing a **literal** node id instead of a `$variable`, and a `gh` write whose explicit target is under an owner other than the checkout origin's. Sibling repositories under the same owner are allowed, since the harm this guards is reaching a stranger's repository rather than working across one maintainer's own fleet, and a different owner is allowed only when the maintainer names it in `GH_WRITE_GUARD_ALLOW` (an `owner/repo` list, where `owner/*` grants a whole owner). That variable is read from the environment the session was launched with, which a tool call inside that session cannot change. A hook runs as its own process, so an inline `VAR=x cmd` prefix or an `export` inside a Bash call never reaches it. Second, a **git operation that bypasses a repository protection**: `gh pr merge --admin`, a commit or push that skips the hooks by `--no-verify` or by a `core.hooksPath` or include override, and a push the branch's **live** rules forbid. So a code-style `develop` is denied while a config-style `develop` is allowed with no per-repo configuration. A push to a protected-default branch fails closed when its rules cannot be determined. Third, a **mutating git operation run directly against a primary checkout**, the harm behind a second incident where an agent reused the maintainer's own primary checkout instead of a worktree despite having read the prose rule against it. `GH_WRITE_GUARD_ALLOW_PRIMARY_CHECKOUT` grants an exception through the same channel as `GH_WRITE_GUARD_ALLOW`. Fourth, an **unbounded shell wait**, a loop that sleeps with no bound, the harm behind a third incident where seven such loops outlived the subagents that wrote them and the run that dispatched those subagents. The hook reads a command with one simple lexer and denies a shape only where it appears plainly. A shape it does not read is allowed, per the root spec's threat model. It fires even in autonomous / bypass-permissions sessions, which is how the first incident happened. See the root spec's requirements 1 to 7 for the exact rules. - **`hooks/tool-containment.py`**: a shell prefix, registered as `CLAUDE_CODE_SHELL_PREFIX` in the `env` block of `settings.json`, that runs every Bash tool call inside a transient systemd user scope carrying `TasksMax`, `MemoryMax`, and `MemorySwapMax=0`, the last so a runaway at the memory ceiling is killed rather than spilled into the host's swap. A runaway fan-out then fails at the ceiling with `fork: Resource temporarily unavailable` instead of growing until the host has to be reset, the harm behind the probe-shim incident in the root spec's requirement 9. The guard above reads the command text, and that incident's recursion lived in a script file the command only named. The ceilings default to 8192 tasks and 25% of RAM. A `claude` session started from a tool call is nested in the session that ran it. Every scope of such a chain also joins one slice, `claude-tree-.slice`. That slice carries a total ceiling for the whole tree, 32768 tasks and 50% of RAM. The root's id reaches nested sessions as `AGENT_CONTAINMENT_ROOT`, which every contained command inherits. The tree's ceiling is a runtime drop-in put on when its slice is first created, and the sweep below reverts it. The maintainer raises any ceiling in the same `env` block. An inline `VAR=x` in a command cannot change one, since the prefix reads it before the command runs. The variables are `AGENT_CONTAINMENT_TASKS_MAX` and `AGENT_CONTAINMENT_MEMORY_MAX` for a scope, and `AGENT_CONTAINMENT_TREE_TASKS_MAX` and `AGENT_CONTAINMENT_TREE_MEMORY_MAX` for the tree. A tree ceiling set below a scope's binds first. Each scope is named `claude-tool---`, so the sweep below can stop one session's tool calls and nothing else. Its description records the pid of the agent that ran it. That record is how the sweep tells a scope whose session has ended from one whose session still runs. The installer sets the prefix only where it can contain, per the Linux bullet under "Scope and Limits", and never over a `CLAUDE_CODE_SHELL_PREFIX` value that names anything else. - **`hooks/stray-process-sweep.py`**: a SessionEnd hook in two phases. First, it stops every `claude-tool--*` scope this session left loaded and reports what each held, since the scope's name proves which session started it. It then stops every scope in this session's tree whose recorded agent is no longer running, and names each one's session. That is how a nested session's scopes end, since stopping the parent's scopes kills the nested `claude` before its own sweep runs. Each stop can end an agent nested deeper, so it judges again, up to four rounds. The root session then stops its tree's slice and reverts the ceiling put on it, once nothing is left in it. Second, it reports, and never ends, the process trees outside those scopes that outlive the session. The guard above stops a leak from being written. This finds the ones already running: the ones started before the guard was installed, and the shapes the guard deliberately does not reach. It reports every descendant of the agent process running outside the agent's own session. It names each subtree root's command and age, and hands over the `kill` line. This phase kills nothing itself. A long build backgrounded on purpose reads identically to a leaked wait from there. What is still wanted on the machine is the maintainer's call. A SessionEnd hook cannot block, and its JSON output is discarded, so stderr on exit 2 is the one channel that reaches the maintainer. The sweep uses that for a report rather than for a refusal. It registers under a group carrying no matcher, which fires on every exit reason. It also carries an explicit timeout, since the event's own budget is 1.5 seconds and only a declared per-hook timeout raises it. See the root spec's requirement 8. +- **`hooks/model-tier-guard.py`** and **`hooks/model-tiers.json`**: a PreToolUse hook that denies a subagent dispatch at the maintainer-only model tier, today `fable`. It reads the tier mapping from the `model-tiers.json` beside it, a copy of the hub's `spec/model-tiers.json` the installer makes. It checks an `Agent` call's `model` and a `Workflow` script's `model` options, from the inline `script` and the file at `scriptPath`. A saved workflow named by `name` alone, or a child workflow a script starts, cannot be inspected and is allowed. A dispatch naming no model runs at a model the hook cannot read, and so does a fork naming another model, so both are allowed. The maintainer allows the tier by launching the session with `FLEET_ALLOW_HIGHEST=1`, which a tool call inside that session cannot change. Any other value stays denied, and the deny text never suggests setting it. Where the maintainer asked for the tier, the deny text says to stop and report rather than dispatch lower. Otherwise it names the default tier's model. A mapping that is missing or unreadable denies any call naming a model and says to re-run the installer, so the guard never goes quiet. A session launched with the allowance blocks no model, so it needs no mapping. It registers in its own `Agent|Workflow` matcher group rather than the Bash one. - **The whole of `CLAUDE.md`, rendered on every run**, loaded into every session on the machine (including ad-hoc work outside any project). It holds a header, two marker-delimited blocks of host-wide content, and the host-local file. Nothing else survives an install. Content outside the blocks was never written or checked, and it restated fleet rules in wording the fleet had since changed. - The host-local file is `~/.config/agent-fleet/local.md`, under `XDG_CONFIG_HOME` where that is set to an absolute path. The installer appends it under a `host-local` marker and never edits it. It is one file for every agent, so a Codex or opencode global file rendered later appends the same text. - The stamp records a digest of the file each install wrote. A file matching it is an earlier render and is replaced silently. Any other file was edited by hand, or predates whole-file ownership, so it is copied to `CLAUDE.md..bak` first and the install prints that path. `--report` names which of the two it found. @@ -52,6 +53,7 @@ The deployed copy on each machine is a snapshot, so when the guard changes upstr python3 ~/.claude/hooks/gh-write-guard.py --selftest # decision matrix: all cases pass python3 ~/.claude/hooks/stray-process-sweep.py --selftest # reporting matrix: all cases pass python3 ~/.claude/hooks/tool-containment.py --selftest # containment matrix: all cases pass +python3 ~/.claude/hooks/model-tier-guard.py --selftest # tier matrix: all cases pass python3 ~/.claude/hooks/stray-process-sweep.py --orphans # exit 0: no command scope outlived its session grep -c CLAUDE_CODE_SHELL_PREFIX ~/.claude/settings.json # expect 1 where the installer reported the prefix set grep -c 'agent-safety v' ~/.claude/CLAUDE.md # expect 2 (start + end marker) @@ -66,6 +68,7 @@ On Windows PowerShell: py -3 "$env:USERPROFILE\.claude\hooks\gh-write-guard.py" --selftest # all cases pass py -3 "$env:USERPROFILE\.claude\hooks\stray-process-sweep.py" --selftest # all cases pass py -3 "$env:USERPROFILE\.claude\hooks\tool-containment.py" --selftest # all cases pass +py -3 "$env:USERPROFILE\.claude\hooks\model-tier-guard.py" --selftest # all cases pass (Select-String 'agent-safety v' "$env:USERPROFILE\.claude\CLAUDE.md").Count # expect 2 (Select-String 'fleet-bootstrap v' "$env:USERPROFILE\.claude\CLAUDE.md").Count # expect 2 (Select-String -SimpleMatch 'Bash(python3 scripts/pr_review.py:*)' "$env:USERPROFILE\.claude\settings.json").Count # expect 1 @@ -139,7 +142,8 @@ The `uv run --no-project --script` rule beside it matches the spelling the fleet }, "hooks": { "PreToolUse": [ - { "matcher": "Bash", "hooks": [ { "type": "command", "command": "\"python3\" \"/.claude/hooks/gh-write-guard.py\"" } ] } + { "matcher": "Bash", "hooks": [ { "type": "command", "command": "\"python3\" \"/.claude/hooks/gh-write-guard.py\"" } ] }, + { "matcher": "Agent|Workflow", "hooks": [ { "type": "command", "command": "\"python3\" \"/.claude/hooks/model-tier-guard.py\"" } ] } ], "SessionEnd": [ { "hooks": [ { "type": "command", "command": "\"python3\" \"/.claude/hooks/stray-process-sweep.py\"", "timeout": 35 } ] } diff --git a/host-setup/agent-safety/claude/claude-md-safety.md b/host-setup/agent-safety/claude/claude-md-safety.md index f7050ada1..f8cf5bc23 100644 --- a/host-setup/agent-safety/claude/claude-md-safety.md +++ b/host-setup/agent-safety/claude/claude-md-safety.md @@ -27,4 +27,5 @@ A permission the maintainer grants is bounded by what he said, where he said it, - **Never widen a grant by inference.** Permission for one repository does not carry to a sibling, permission for one pull request does not carry to the next, and permission for one task does not become a mode for the session. Similarity is not authorization. Re-ask instead, because re-asking is cheap and an unwanted write is not. - **Never grant to yourself.** Do not record an authorization he did not give, infer one from a tool's capability, or restore one he narrowed or withdrew. Capability is not permission. - **The irreversible step stays his.** Merging, publishing, releasing, force-pushing, deleting, and changing branch protection each stop for an explicit and current go-ahead, however green the work is and whatever a memory says about a past session. +- **The most capable model is his to select.** The `highest` model tier is the most capable model available. It runs only where he chose it for the session, or allowed it for subagents when the session started. A task looking hard is not that choice. Neither is a rule or a skill calling for the `frontier` tier, which is the default reasoning tier below it rather than the industry's "frontier model". A tier resolves through a repository's `AGENTS.md` "Model Tiers" table or a fleet skill carrying it. A session with neither in view names no model rather than guessing one. The `model-tier-guard` hook refuses a dispatch naming that model unless `FLEET_ALLOW_HIGHEST=1` was set before the session started, and an agent never sets it. diff --git a/host-setup/agent-safety/claude/install.py b/host-setup/agent-safety/claude/install.py index d9b3232c5..eff0c2679 100755 --- a/host-setup/agent-safety/claude/install.py +++ b/host-setup/agent-safety/claude/install.py @@ -1,11 +1,12 @@ #!/usr/bin/env python3 """Install the agent host-safety kit for the current user account. Cross-platform, idempotent. -Deploys the PreToolUse hook, the SessionEnd stray-process sweep, and the tool-containment shell prefix, -registers the two hooks in the user settings.json and, on a host with a systemd user manager, the prefix -in its `env`, merges the permission rules this kit owns into the same file, renders the user -CLAUDE.md whole from the kit's blocks and the host-local instruction file, and self-tests each hook -before registering it. +Deploys the PreToolUse write guard, the PreToolUse model-tier guard with its tier mapping, the +SessionEnd stray-process sweep, and the tool-containment shell prefix. It registers the hooks in the +user settings.json. On a host with a systemd user manager it also sets the prefix in that file's +`env`. It merges the permission rules this kit owns into the same file. It renders the user +CLAUDE.md whole from the kit's blocks and the host-local instruction file. Each hook is self-tested +before it is registered. The bash and PowerShell wrappers both call this, so every OS runs one tested code path. Every run records a stamp at ~/.claude/agent-safety-stamp.json naming the machine, what was @@ -45,13 +46,20 @@ SWEEP_NAME = "stray-process-sweep.py" SWEEP_STEM = "stray-process-sweep" CONTAIN_NAME = "tool-containment.py" +TIER_NAME = "model-tier-guard.py" +TIER_CONFIG_NAME = "model-tiers.json" +# The tier guard fires on the tools that dispatch a subagent, so it is its own group and not the Bash one. +TIER_MATCHER = "Agent|Workflow" PREFIX_VAR = "CLAUDE_CODE_SHELL_PREFIX" # The hook files this kit copies into ~/.claude/hooks, in deploy order. # Named here rather than spelled inside `main`, since a test scraping `main` for a literal path goes silent when the copy is refactored. # That silence reads as a pass. -DEPLOYED_HOOKS = (GUARD_NAME, SWEEP_NAME, CONTAIN_NAME) +DEPLOYED_HOOKS = (GUARD_NAME, SWEEP_NAME, CONTAIN_NAME, TIER_NAME) + +# The tier mapping lives in the hub's spec/ so a gate and the hook read one file, and is copied here, never edited here. +DEPLOYED_DATA = {TIER_CONFIG_NAME: HERE.parent.parent.parent / "spec" / TIER_CONFIG_NAME} # A SessionEnd hook's own budget is 1.5 seconds, raised to the highest per-hook timeout the settings declare. # The sweep's own calls are each bounded, two `ps` reads at 5s and four `systemctl` calls at 5s, all inside this. @@ -81,13 +89,20 @@ # Written out, this list and the block list drifted apart silently and the digest stopped covering a file. # The digest is taken over these rather than over the commit, since it is the content that runs. # A clean commit and a dirty checkout install different bytes while reporting the same SHA. -PAYLOAD_FILES = DEPLOYED_HOOKS + tuple(filename for _, filename in CLAUDE_MD_BLOCKS) +PAYLOAD_FILES = ( + DEPLOYED_HOOKS + tuple(DEPLOYED_DATA) + tuple(filename for _, filename in CLAUDE_MD_BLOCKS) +) # Distinguishes an absent key from one holding an explicit null, which `dict.get` reports alike. # The two need different answers, since a gap is filled and a null is a settings error. MISSING = object() +def payload_source(name): + """Where a payload file is read from, the kit directory unless it is a data file kept in spec/.""" + return DEPLOYED_DATA.get(name, HERE / name) + + def hook_command(launcher, path): """The `command` string a registered hook carries, spelled once for writer and reader alike.""" return f'"{launcher}" "{path}"' @@ -157,8 +172,8 @@ def names_hook(command, name): return re.search(quoted, text) is not None or re.search(bare, text) is not None -def matcher_sees_bash(matcher): - """Whether a PreToolUse matcher group receives a Bash call. +def matcher_sees_tool(matcher, tool): + """Whether a PreToolUse matcher group receives a call to `tool`. An absent or empty matcher runs on every tool and `*` is the all-tools spelling, so neither is a defect. A matcher is otherwise a regular expression, which makes `Bash|Task` as good as @@ -169,7 +184,7 @@ def matcher_sees_bash(matcher): if not isinstance(matcher, str): return False try: - return re.fullmatch(matcher, "Bash") is not None + return re.fullmatch(matcher, tool) is not None except re.error: return False @@ -179,13 +194,31 @@ def matcher_covers_every_exit_reason(matcher): A SessionEnd matcher filters by exit reason the way a PreToolUse matcher filters by tool: an absent or empty matcher runs on every reason and `*` is the all-reasons spelling, so none of - the three is a defect. Unlike `matcher_sees_bash`, there is no single reason to test regex + the three is a defect. Unlike `matcher_sees_tool`, there is no single reason to test regex membership against, since the sweep must fire on every exit reason rather than one particular one, so this classifies the shape rather than evaluating a pattern. """ return matcher is None or matcher in ("", "*") +def register_hook(groups, name, entry, selects, matcher=None): + """Leave `entry` as the only registration of hook `name` across an event's matcher `groups`. + + Every entry naming the hook is stripped first, so a re-run leaves no duplicate behind even + where more than one group would select it. `entry` then goes into the first group `selects` + accepts, or into a new group carrying `matcher`, which an absent `matcher` leaves unfiltered. + """ + for g in groups: + hooks_list = g.get("hooks") + if isinstance(hooks_list, list): + hooks_list[:] = [h for h in hooks_list if not names_hook(h.get("command", ""), name)] + group = next((g for g in groups if selects(g)), None) + if group is None: + group = {"hooks": []} if matcher is None else {"matcher": matcher, "hooks": []} + groups.append(group) + group.setdefault("hooks", []).append(entry) + + def owns(entry, prefix): """Whether an allow rule names the script the prefix identifies, rather than a longer path. @@ -401,7 +434,9 @@ def git(*args): file=sys.stderr, ) else: - status = git("status", "--porcelain", "--", *PAYLOAD_FILES) + status = git( + "status", "--porcelain", "--", *(str(payload_source(n)) for n in PAYLOAD_FILES) + ) ref["dirty"] = bool(status) return ref @@ -755,11 +790,12 @@ def payload_digest(): Fixed order because a set of files has none, and a digest that depends on directory listing order reports drift on a machine where nothing changed. The order matches the one - `installed_digest` reads, so the two are directly comparable: each deployed hook, then each block. + `installed_digest` reads, so the two are directly comparable: each deployed hook, then each data + file, then each block. """ h = hashlib.sha256() for name in PAYLOAD_FILES: - raw = normalized((HERE / name).read_bytes()) + raw = normalized(payload_source(name).read_bytes()) # A snippet is embedded stripped, so trailing whitespace is not installed content. # The hook is copied byte for byte, so nothing about it is stripped. if name.endswith(".md"): @@ -832,7 +868,7 @@ def installed_digest(claude_home): """ # Read from `DEPLOYED_HOOKS` rather than by name, since naming them here is the drift that constant closes. # A hook added to the deploy list and not to this one installs and is never covered by the currentness digest. - deployed = [claude_home / "hooks" / name for name in DEPLOYED_HOOKS] + deployed = [claude_home / "hooks" / name for name in (*DEPLOYED_HOOKS, *DEPLOYED_DATA)] claude_md = claude_home / "CLAUDE.md" claude_text = read_claude_md(claude_md) try: @@ -924,11 +960,11 @@ def registration_problems(claude_home): try: raw = read_regular_file(claude_home / "settings.json") except FileNotFoundError: - return ["settings.json is missing, so the hook is not registered"] + return ["settings.json is missing, so the hooks are not registered"] except OSError as e: return [f"settings.json cannot be read ({e})"] if raw is None: - return ["settings.json is not a regular file, so the hook is not registered"] + return ["settings.json is not a regular file, so the hooks are not registered"] try: data = json.loads(raw.decode("utf-8") or "{}") # ValueError rather than JSONDecodeError, since it also covers UnicodeDecodeError. @@ -972,95 +1008,116 @@ def event_groups(event): return None return groups - groups = event_groups("PreToolUse") - # Counted apart from the registrations, since an entry with a problem is still an entry. - # Counting only the sound ones added "the guard never runs" under every problem reported here. - # That line is false, and it sends a reader to the wrong fix. - named = 0 - registered = 0 - guard_path = claude_home / "hooks" / GUARD_NAME - for group in groups or []: - if not isinstance(group, dict): - continue - for hook in group.get("hooks") or []: - if not isinstance(hook, dict) or not names_hook(hook.get("command", ""), GUARD_NAME): - continue - named += 1 - if not runs_hook(hook.get("command"), guard_path) or hook.get("type") != "command": - out.append( - "a PreToolUse entry names the guard but does not run the deployed one " - f"({hook.get('type')!r} running {hook.get('command')!r})" - ) - continue - if not matcher_sees_bash(group.get("matcher")): - report( - f"a PreToolUse group registers the guard under matcher " - f"{group.get('matcher')!r}, which no Bash call matches, so that group never " - "fires" - ) - continue - registered += 1 - if groups is None: - pass # the shape error is already reported, and a count from it would be meaningless - elif named == 0: - out.append( - "the PreToolUse hook is not registered in settings.json, so the guard never runs" - ) - elif registered > 1: - out.append( - f"the PreToolUse hook is registered {registered} times, so it runs more than once" - ) - ends = event_groups("SessionEnd") - sweeps_named = 0 - swept = 0 - for group in ends or []: - if not isinstance(group, dict): - continue - sweep_path = claude_home / "hooks" / SWEEP_NAME - for hook in group.get("hooks") or []: - if not isinstance(hook, dict) or not names_hook(hook.get("command", ""), SWEEP_NAME): - continue - sweeps_named += 1 - if not runs_hook(hook.get("command"), sweep_path) or hook.get("type") != "command": - out.append( - "a SessionEnd entry names the sweep but does not run the deployed one " - f"({hook.get('type')!r} running {hook.get('command')!r})" - ) - continue - # A longer budget than this installer writes is not a defect, so only a shorter one is reported. - # The event's own default is 1.5s, which is what the sweep needs more than. - timeout = hook.get("timeout") - too_short = isinstance(timeout, bool) or not isinstance(timeout, (int, float)) - if not too_short: - too_short = timeout < SWEEP_TIMEOUT_SECONDS - if too_short: - carries = ( - "carries no timeout" - if "timeout" not in hook - else f"carries timeout {timeout!r}" - ) - out.append( - f"the SessionEnd sweep {carries} where this installer writes " - f"{SWEEP_TIMEOUT_SECONDS}, too little for a `ps` on a loaded machine" - ) + def check_hook(event, groups, name, label, subject, absent_effect, judge): + """Report each way the `event` `groups` fail to run hook `name` exactly once. + + `label` names the hook in an entry defect, and `subject` names it in the count lines, with + `absent_effect` saying what a missing entry costs. `judge(group, hook)` reports what `event` + itself requires of an entry that runs the deployed hook, and returns whether it counts as registered. + """ + if groups is None: + return # the shape error is already reported, and a count from it would be meaningless + # Counted apart from the registrations, since an entry with a problem is still an entry. + # Counting only the sound ones added a "not registered" line under every problem reported here. + # That line is false, and it sends a reader to the wrong fix. + named = 0 + registered = 0 + path = claude_home / "hooks" / name + for group in groups: + if not isinstance(group, dict): continue - swept += 1 - # Any matcher other than absent, empty, or `*` may filter some exit reasons out. - # Counting it as registered reports a machine current while the sweep may never fire on an ordinary exit. - if not matcher_covers_every_exit_reason(group.get("matcher")): + for hook in group.get("hooks") or []: + if not isinstance(hook, dict) or not names_hook(hook.get("command", ""), name): + continue + named += 1 + if not runs_hook(hook.get("command"), path) or hook.get("type") != "command": + out.append( + f"a {event} entry names the {label} but does not run the deployed one " + f"({hook.get('type')!r} running {hook.get('command')!r})" + ) + continue + if judge(group, hook): + registered += 1 + if named == 0: + out.append(f"{subject} is not registered in settings.json, so {absent_effect}") + elif registered > 1: + out.append(f"{subject} is registered {registered} times, so it runs more than once") + + def check_pre_tool_hook(groups, name, tools, label, subject, unseen_effect, absent_effect): + """Report each way the PreToolUse `groups` fail to run hook `name` once on every one of `tools`. + + `label` names the hook in an entry or group defect, and `subject` names it in the count + lines. `unseen_effect` and `absent_effect` say what a blind matcher and a missing entry cost. + """ + + def judge(group, hook): + # Every tool is checked, since a matcher seeing only some leaves the others' calls unguarded. + unseen = [t for t in tools if not matcher_sees_tool(group.get("matcher"), t)] + if unseen: report( - f"the SessionEnd sweep is registered under a matcher " - f"({group.get('matcher')!r}) other than absent, empty, or `*`, so it may not run on every exit reason" + f"a PreToolUse group registers the {label} under matcher " + f"{group.get('matcher')!r}, which no {'/'.join(unseen)} call matches, so " + f"{unseen_effect}" ) - if ends is None: - pass # likewise reported as a shape error above - elif sweeps_named == 0: - out.append( - "the SessionEnd sweep is not registered in settings.json, so a surviving shell is " - "never reported" - ) - elif swept > 1: - out.append(f"the SessionEnd sweep is registered {swept} times, so it runs more than once") + return False + return True + + check_hook("PreToolUse", groups, name, label, subject, absent_effect, judge) + + def judge_sweep(group, hook): + """Report a SessionEnd sweep entry's timeout and matcher, counting it unless its timeout is too short.""" + # A longer budget than this installer writes is not a defect, so only a shorter one is reported. + # The event's own default is 1.5s, which is what the sweep needs more than. + timeout = hook.get("timeout") + too_short = isinstance(timeout, bool) or not isinstance(timeout, (int, float)) + if not too_short: + too_short = timeout < SWEEP_TIMEOUT_SECONDS + if too_short: + carries = ( + "carries no timeout" if "timeout" not in hook else f"carries timeout {timeout!r}" + ) + out.append( + f"the SessionEnd sweep {carries} where this installer writes " + f"{SWEEP_TIMEOUT_SECONDS}, too little for a `ps` on a loaded machine" + ) + return False + # Any matcher other than absent, empty, or `*` may filter some exit reasons out. + # Counting it as registered reports a machine current while the sweep may never fire on an ordinary exit. + if not matcher_covers_every_exit_reason(group.get("matcher")): + report( + f"the SessionEnd sweep is registered under a matcher " + f"({group.get('matcher')!r}) other than absent, empty, or `*`, so it may not run on every exit reason" + ) + return True + + groups = event_groups("PreToolUse") + check_pre_tool_hook( + groups, + GUARD_NAME, + ("Bash",), + "write guard", + "the write guard", + "that group never fires", + "the guard never runs", + ) + check_pre_tool_hook( + groups, + TIER_NAME, + ("Agent", "Workflow"), + "model tier guard", + "the model tier guard", + "that dispatch is never checked", + "a maintainer-only model dispatch is never refused", + ) + check_hook( + "SessionEnd", + event_groups("SessionEnd"), + SWEEP_NAME, + "sweep", + "the SessionEnd sweep", + "a surviving shell is never reported", + judge_sweep, + ) capable, reason = containment_capable(claude_home / "hooks" / CONTAIN_NAME) env = data.get("env") held = env.get(PREFIX_VAR) if isinstance(env, dict) else None @@ -1167,7 +1224,7 @@ def report(claude_home): live_installed = installed_digest(claude_home) if live_installed is None: problems.append( - "a deployed hook or CLAUDE.md is missing or unreadable, so the kit is not fully installed" + "a deployed hook, its data file, or CLAUDE.md is missing or unreadable, so the kit is not fully installed" ) elif live_installed != current: problems.append("the installed content differs from what this checkout would write") @@ -1283,6 +1340,7 @@ def main(): hook_dst = hooks_dir / GUARD_NAME sweep_dst = hooks_dir / SWEEP_NAME contain_dst = hooks_dir / CONTAIN_NAME + tier_dst = hooks_dir / TIER_NAME settings = claude_home / "settings.json" claude_md = claude_home / "CLAUDE.md" @@ -1408,6 +1466,20 @@ def main(): ) return 1 print(f" {src_name} self-test: PASS") + # The data the hooks read goes in after every self-test passed and before any hook is replaced. + # A new hook then never runs beside a config it cannot find. + for data_name, data_src in DEPLOYED_DATA.items(): + data_tmp = hooks_dir / f"{data_name}.{os.getpid()}.staged" + try: + shutil.copyfile(data_src, data_tmp) + os.replace(data_tmp, hooks_dir / data_name) + except OSError as e: + data_tmp.unlink(missing_ok=True) + for _n, f in staged: + f.unlink(missing_ok=True) + sys.stderr.write(f"copying {data_name} failed ({e}); no hook was replaced.\n") + return 1 + print(f" data -> {hooks_dir / data_name}") for done_count, (src_name, tmp) in enumerate(staged): dst = hooks_dir / src_name # `os.replace` is atomic on the same filesystem, so a live hook is never a partial file. @@ -1456,7 +1528,7 @@ def main(): if not isinstance(data, dict): sys.stderr.write( f"{settings} is valid JSON but holds {type(data).__name__} at its root where an object " - "is required. Fix or remove it, then re-run. This file is unchanged, so the hook is " + "is required. Fix or remove it, then re-run. This file is unchanged, so the hooks are " "deployed but not registered.\n" ) return 1 @@ -1468,7 +1540,7 @@ def main(): def reject(where, held, want): sys.stderr.write( f"{settings} has `{where}` as {type(held).__name__} where {want.__name__} is required. " - "Fix or remove that key, then re-run. This file is unchanged, so the hook is deployed " + "Fix or remove that key, then re-run. This file is unchanged, so the hooks are deployed " "but not registered.\n" ) @@ -1506,41 +1578,37 @@ def reject(where, held, want): return 1 pre = data.setdefault("hooks", {}).setdefault("PreToolUse", []) - # Strip the hook from every existing group first, so a re-run leaves no duplicate behind. - # That matters when settings.json already carries more than one Bash group. - # Then register it in a single Bash group. - for g in pre: - hooks_list = g.get("hooks") - if isinstance(hooks_list, list): - hooks_list[:] = [ - h for h in hooks_list if not names_hook(h.get("command", ""), GUARD_NAME) - ] - group = next((g for g in pre if g.get("matcher") == "Bash"), None) - if group is None: - group = {"matcher": "Bash", "hooks": []} - pre.append(group) - group.setdefault("hooks", []).append({"type": "command", "command": hook_cmd}) + register_hook( + pre, + GUARD_NAME, + {"type": "command", "command": hook_cmd}, + lambda g: g.get("matcher") == "Bash", + "Bash", + ) done = ["PreToolUse/Bash hook registered"] + # The tier guard gets its own Agent|Workflow group, stripped and re-added the same way so a re-run leaves one. + register_hook( + pre, + TIER_NAME, + {"type": "command", "command": hook_command(launcher, tier_dst)}, + lambda g: g.get("matcher") == TIER_MATCHER, + TIER_MATCHER, + ) + done.append(f"PreToolUse/{TIER_MATCHER} model tier guard registered") + # Step 2b registers the SessionEnd sweep the same strip-then-register way as the guard above. # The chosen or newly created group covers every exit reason, so the sweep fires on all of them. ends = data.setdefault("hooks", {}).setdefault("SessionEnd", []) - for g in ends: - hooks_list = g.get("hooks") - if isinstance(hooks_list, list): - hooks_list[:] = [ - h for h in hooks_list if not names_hook(h.get("command", ""), SWEEP_NAME) - ] - end_group = next((g for g in ends if matcher_covers_every_exit_reason(g.get("matcher"))), None) - if end_group is None: - end_group = {"hooks": []} - ends.append(end_group) - end_group.setdefault("hooks", []).append( + register_hook( + ends, + SWEEP_NAME, { "type": "command", "command": hook_command(launcher, sweep_dst), "timeout": SWEEP_TIMEOUT_SECONDS, - } + }, + lambda g: matcher_covers_every_exit_reason(g.get("matcher")), ) done.append("SessionEnd sweep registered") diff --git a/host-setup/agent-safety/claude/model-tier-guard.py b/host-setup/agent-safety/claude/model-tier-guard.py new file mode 100755 index 000000000..15915d7c8 --- /dev/null +++ b/host-setup/agent-safety/claude/model-tier-guard.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +"""PreToolUse guard: deny a subagent dispatch at the maintainer-only model tier. + +Registered as a Claude Code PreToolUse hook on the Agent and Workflow tools. It reads the +tool-input JSON on stdin. It prints a deny decision when the call names the model of the tier +`model-tiers.json` marks maintainer-only, today `highest`, so `fable`. Everything else passes. + +The mapping is read from `model-tiers.json` beside this file, which the installer copies from the +hub's spec/model-tiers.json. A model release is then an edit to the hub's spec/model-tiers.json and +to its AGENTS.md tier table, never to this hook. The hub's spec gate holds the two equal. + +The maintainer allows the tier for a session by launching it with FLEET_ALLOW_HIGHEST=1 in the +environment. A hook inherits the environment its session was launched with. A tool call inside that +session cannot change it, and the deny text never suggests that it try. The value must be exactly +`1`, so a loose truthy spelling stays denied. A session launched with it blocks no model, so it +needs no mapping either. + +It bounds a good-faith agent's mistake rather than an adversary. A nested `claude` an agent launches +with the variable set is not told apart, since that is circumvention rather than a mistake. A +workflow script is read as text, so a model chosen by a computed expression is not seen. A saved +workflow named by `name` alone cannot be inspected and is allowed. So is a child workflow a script +starts by name or path, since only the top-level script is read. A dispatch naming no model runs at +a model this hook cannot read, and so does a fork naming another model, so both are allowed. + +A config that is missing, unreadable, or lacks the keys cannot say which model is blocked. A call +naming any model is then denied, so the guard never goes quiet without saying so, and a call naming +none is allowed. + +Run `model-tier-guard.py --selftest` to check the decision matrix without Claude Code. +""" + +import json +import os +import re +import sys + +CONFIG_NAME = "model-tiers.json" +ALLOW_ENV = "FLEET_ALLOW_HIGHEST" +TOOL = "claude-code" + + +def load_tiers(path): + """The (blocked model, default tier, default model) triple from the config, or None where it cannot be read.""" + try: + with open(path, encoding="utf-8") as f: + cfg = json.load(f) + models = cfg["tools"][TOOL]["models"] + blocked = models[cfg["maintainerOnly"]] + default_tier = cfg["defaultTier"] + default = models[default_tier] + except (OSError, ValueError, KeyError, TypeError): + return None + if not isinstance(blocked, str) or not isinstance(default, str) or not blocked: + return None + return blocked, default_tier, default + + +def script_text(tool_input, cwd=None): + """`(text, unreadable)` for the script the Workflow tool will run, the file at `scriptPath` and the inline `script`. + + Both are read and joined, since a call passing both runs the file. Reading only the inline + text would let a model named in the file pass unseen. A relative path resolves against the + session's `cwd` rather than this hook's own directory. A file that cannot be read is reported, + so the caller refuses rather than passing a script it never saw. + """ + parts = [] + unreadable = False + path = tool_input.get("scriptPath") + if isinstance(path, str) and path: + path = os.path.expanduser(path) + if not os.path.isabs(path) and isinstance(cwd, str) and cwd: + path = os.path.join(cwd, path) + try: + with open(path, encoding="utf-8") as f: + parts.append(f.read()) + except (OSError, ValueError): + unreadable = True + inline = tool_input.get("script") + if isinstance(inline, str): + parts.append(inline) + return "\n".join(parts), unreadable + + +def names_model(text, model): + """Whether a script text carries a `model: ''` option naming `model`.""" + # A JavaScript key may be bare or quoted, so `{"model": "x"}` is the same option as `{model: "x"}`. + # The value may be a full model id rather than the alias, so the alias is matched as a word inside it. + value = r"[^'\"`]*(? [spec]: ../README.md diff --git a/host-setup/agent-safety/opencode/README.md b/host-setup/agent-safety/opencode/README.md index ae25f7d29..baf0572b3 100644 --- a/host-setup/agent-safety/opencode/README.md +++ b/host-setup/agent-safety/opencode/README.md @@ -31,8 +31,8 @@ on [issue #781][issue-781], not silently worked around. ## Auditing Once an opencode-side implementation exists, audit it the way [`../README.md`][spec] "Auditing an -Implementation Against This Spec" describes: run its own self-test and check every case against the -nine requirements, not against `gh-write-guard.py`'s source. +Implementation Against This Spec" describes. Run its own self-test and check every case against the +requirements, not against `gh-write-guard.py`'s source. [spec]: ../README.md diff --git a/spec/model-tiers.json b/spec/model-tiers.json new file mode 100644 index 000000000..f97dbeafc --- /dev/null +++ b/spec/model-tiers.json @@ -0,0 +1,17 @@ +{ + "note": "The model-tier mapping AGENTS.md \"Model Tiers\" states, as data a gate and a hook can read. A rule, a Skill, or a brief names a tier, never a model, so a model release is an edit to a `models` value here and to the AGENTS.md table, which spec/validate.py holds equal. Each value is the alias that tool's own subagent dispatch accepts, an alias that already tracks the latest model of its family. A tool with no entry, or with no way to name a model per subagent, runs every tier at the session's own model. The model-tier-guard hook in host-setup/agent-safety/claude/ refuses a dispatch at the maintainerOnly tier's model unless the maintainer allowed it at launch.", + "tiers": ["highest", "frontier", "standard", "bulk"], + "maintainerOnly": "highest", + "defaultTier": "frontier", + "tools": { + "claude-code": { + "name": "Claude Code", + "models": { + "highest": "fable", + "frontier": "opus", + "standard": "sonnet", + "bulk": "haiku" + } + } + } +} diff --git a/spec/validate.py b/spec/validate.py index 5b4bc297e..c368e0dca 100755 --- a/spec/validate.py +++ b/spec/validate.py @@ -688,6 +688,93 @@ def version_literal_errors(root): return errors +def model_tier_errors(root): + """Hold spec/model-tiers.json to its own invariants and AGENTS.md's tier table to it. + + The table is prose a session reads, and the JSON is data the model-tier-guard hook reads. A + model release editing only one would leave the hook refusing a model the rule no longer names, + or allowing the one it now names. AGENTS.md is carried, so the table cannot be + a generated include region, and an equality check is what keeps the two in step instead. + """ + try: + config = json.loads((root / "spec/model-tiers.json").read_text(encoding="utf-8")) + tiers = config["tiers"] + reserved = config["maintainerOnly"] + default = config["defaultTier"] + tools = config["tools"] + except (OSError, ValueError, KeyError, TypeError) as exc: + return [ + f"spec/model-tiers.json: cannot read tiers, maintainerOnly, defaultTier, and tools ({exc})" + ] + if not is_str_list(tiers) or not isinstance(tools, dict): + return ["spec/model-tiers.json: tiers must be a list of strings and tools an object"] + errors = [] + if len(set(tiers)) != len(tiers): + errors.append("spec/model-tiers.json: tiers must not repeat a name") + if reserved not in tiers: + errors.append(f"spec/model-tiers.json: maintainerOnly {reserved!r} is not a declared tier") + # The default is what an orchestrator dispatches at without being told otherwise, so it can never be the tier only the maintainer selects. + if default not in tiers or default == reserved: + errors.append( + f"spec/model-tiers.json: defaultTier {default!r} must be a declared tier other than maintainerOnly" + ) + rows = [ + "| Tool | " + " | ".join(f"`{t}`" for t in tiers) + " |", + "| --- |" + " --- |" * len(tiers), + ] + for tool, entry in tools.items(): + if not isinstance(entry, dict): + errors.append(f"spec/model-tiers.json: tool {tool!r} needs a name and a models object") + continue + models = entry.get("models") + if not isinstance(models, dict) or not isinstance(entry.get("name"), str): + errors.append(f"spec/model-tiers.json: tool {tool!r} needs a name and a models object") + continue + # A row with a hole leaves a tier resolving to whatever the session runs on. + # For the default tier, that is the leak this file exists to close. + if set(models) != set(tiers): + errors.append( + f"spec/model-tiers.json: tool {tool!r} must map exactly the declared tiers" + ) + continue + if not all(isinstance(m, str) and m for m in models.values()): + errors.append( + f"spec/model-tiers.json: tool {tool!r} must map each tier to a non-empty model name" + ) + continue + if ( + isinstance(reserved, str) + and reserved in models + and any( + t != reserved and m.lower() == models[reserved].lower() for t, m in models.items() + ) + ): + errors.append( + f"spec/model-tiers.json: tool {tool!r} must give maintainerOnly a model no other tier shares" + ) + continue + rows.append(f"| {entry['name']} | " + " | ".join(f"`{models[t]}`" for t in tiers) + " |") + agents_path = root / "AGENTS.md" + if not agents_path.is_file(): + return errors + agents = agents_path.read_text(encoding="utf-8").splitlines() + try: + start = agents.index(rows[0]) + except ValueError: + return [*errors, f"AGENTS.md: no tier table whose header is {rows[0]!r}"] + table = [] + for line in agents[start:]: + if not line.startswith("|"): + break + table.append(line) + if table != rows: + errors.append( + "AGENTS.md: the Model Tiers table does not match spec/model-tiers.json, expected " + + " / ".join(rows[2:]) + ) + return errors + + def utf8_console() -> None: """Write the console streams as UTF-8, whatever the host code page is. @@ -1541,6 +1628,7 @@ def check_selector(where, applies_to): errors.extend(carried_link_errors(ROOT, baseline)) errors.extend(version_literal_errors(ROOT)) + errors.extend(model_tier_errors(ROOT)) # Validate the divergence ledger in spec/divergences.json when present, so a mistyped repo name or disposition fails CI rather than silently dropping a burn-down row. dispositions = ("re-vendor", "track", "accepted", "upstream-candidate", "investigate", "retire") diff --git a/tests/test_cli_usage.py b/tests/test_cli_usage.py index 78301f49f..5ebe8520b 100755 --- a/tests/test_cli_usage.py +++ b/tests/test_cli_usage.py @@ -68,7 +68,9 @@ "host-setup/agent-safety/claude/gh-write-guard.py": "a Claude Code hook", "host-setup/agent-safety/claude/stray-process-sweep.py": "a Claude Code hook", "host-setup/agent-safety/claude/tool-containment.py": "a Claude Code hook", + "host-setup/agent-safety/claude/model-tier-guard.py": "a Claude Code hook", "host-setup/agent-safety/claude/test_install.py": "a test module", + "host-setup/agent-safety/claude/test_model_tier_guard.py": "a test module", "lib/sh/secrets.sh": "a library a script sources", } diff --git a/tests/test_spec_validate.py b/tests/test_spec_validate.py index 274bab858..5a10c594a 100755 --- a/tests/test_spec_validate.py +++ b/tests/test_spec_validate.py @@ -339,6 +339,98 @@ def test_hub_instruction_documents_name_no_version(self) -> None: self.assertEqual(validate.version_literal_errors(root), []) +class ModelTierCase(unittest.TestCase): + """spec/model-tiers.json keeps its invariants, and AGENTS.md's tier table renders it exactly.""" + + TABLE = ( + "| Tool | `highest` | `frontier` | `standard` | `bulk` |\n" + "| --- | --- | --- | --- | --- |\n" + "| Claude Code | `fable` | `opus` | `sonnet` | `haiku` |\n" + ) + + def setUp(self) -> None: + self.root = Path(self.enterContext(tempfile.TemporaryDirectory())) + (self.root / "spec").mkdir() + self.models: dict[str, object] = { + "highest": "fable", + "frontier": "opus", + "standard": "sonnet", + "bulk": "haiku", + } + self.config: dict[str, object] = { + "tiers": ["highest", "frontier", "standard", "bulk"], + "maintainerOnly": "highest", + "defaultTier": "frontier", + "tools": {"claude-code": {"name": "Claude Code", "models": self.models}}, + } + + def errors(self, table: str = TABLE) -> list[str]: + (self.root / "spec/model-tiers.json").write_text(json.dumps(self.config), encoding="utf-8") + (self.root / "AGENTS.md").write_text( + f"# Agents\n\n### Model Tiers\n\n{table}\nAfter.\n", encoding="utf-8" + ) + return validate.model_tier_errors(self.root) + + def test_accepts_a_table_that_renders_the_config(self) -> None: + self.assertEqual(self.errors(), []) + + def test_rejects_a_table_naming_another_model(self) -> None: + errors = self.errors(self.TABLE.replace("| `opus` |", "| `fable` |")) + + self.assertEqual(1, len(errors)) + self.assertIn("does not match", errors[0]) + + def test_rejects_the_maintainer_only_tier_as_the_default(self) -> None: + self.config["defaultTier"] = "highest" + + self.assertTrue(any("defaultTier" in e for e in self.errors())) + + def test_rejects_a_tool_missing_a_tier(self) -> None: + del self.models["bulk"] + + self.assertTrue(any("exactly the declared tiers" in e for e in self.errors())) + + def test_rejects_a_model_that_is_not_a_non_empty_string(self) -> None: + for value in (None, 7, ""): + with self.subTest(value=value): + self.models["highest"] = value + + self.assertTrue(any("non-empty model name" in e for e in self.errors())) + + def test_rejects_a_tier_sharing_the_maintainer_only_model(self) -> None: + self.models["frontier"] = "Fable" + + self.assertTrue(any("no other tier shares" in e for e in self.errors())) + + def test_reports_a_maintainer_only_value_that_is_not_a_string(self) -> None: + self.config["maintainerOnly"] = [] + + self.assertTrue(any("maintainerOnly" in e for e in self.errors())) + + def test_rejects_a_repeated_tier_name(self) -> None: + self.config["tiers"] = ["highest", "frontier", "standard", "bulk", "bulk"] + + self.assertTrue(any("must not repeat a name" in e for e in self.errors())) + + def test_rejects_a_tools_list_instead_of_an_object(self) -> None: + self.config["tools"] = [] + + self.assertTrue(any("tools an object" in e for e in self.errors())) + + def test_rejects_a_tier_that_is_not_a_string(self) -> None: + self.config["tiers"] = [["highest"]] + + self.assertTrue(any("a list of strings" in e for e in self.errors())) + + def test_rejects_a_tool_entry_that_is_not_an_object(self) -> None: + self.config["tools"] = {"claude-code": "opus"} + + self.assertTrue(any("needs a name and a models object" in e for e in self.errors())) + + def test_the_hub_table_matches_the_hub_config(self) -> None: + self.assertEqual(validate.model_tier_errors(Path(__file__).resolve().parents[1]), []) + + class DescriptionErrorsCase(unittest.TestCase): """registry/repos.json's optional `description` (GOVERNANCE.md "Repository Details")."""