Map Model Tiers in One Config and Refuse an Unrequested Highest-Tier Dispatch by ptr727 · Pull Request #3516 · ptr727/ProjectTemplate

ptr727 · 2026-10-27T16:02:15Z

Summary

Skills name a model tier and leave its model to the cwd repository's carried AGENTS.md. A repository whose AGENTS.md predates the tier table leaves frontier unresolved, and sessions read it as "the frontier model" and dispatched subagents at the most capable, maintainer-only model. This closes that from three sides:

Data home. spec/model-tiers.json states the tier-to-model mapping per tool. Only Claude Code has a row today, and Codex or opencode join as data. spec/validate.py holds the AGENTS.md table equal to it, since a carried file cannot hold a generated include region.
Rule text carried with the skills. AGENTS.md moves the tier rule into a "Model Tiers" subsection. It says frontier is not the industry's frontier model and that an orchestrator names a tier on every dispatch. local-strict-review, unattended-handoff, and backlog-burndown carry that subsection as a generated include, so the mapping resolves even in a repository whose carried AGENTS.md is stale.
Mechanical guard. A model-tier-guard PreToolUse hook on Agent|Workflow refuses a dispatch at the maintainer-only tier's model unless the session was launched with FLEET_ALLOW_HIGHEST=1. The installer deploys it with the mapping and verifies its registration. It is requirement 10 of the agent-safety spec, and the host safety block states the same restriction.

Verification

spec/validate.py passes, and tests/test_spec_validate.py covers the table check (171 tests).
The agent-safety kit's tests pass: 181 passed, 1 skipped, including 15 new subprocess cases for the hook.
test_cli_usage passes (16), build_dist.py --check is current, and ruff and mypy are clean.
The prose gate's default checks are clean with comments allowed. This PR adds code comments, so it carries the comments label.

Local Strict Review

Ten recorded passes ran, the last over 6fdde9b with no finding. Every introduced finding from the earlier passes was fixed on this branch, except one the maintainer deferred:

host-setup/agent-safety/claude/install.py: the tier guard's registration check and its strip-then-register block follow the guard's blocks rather than sharing a helper. Deferred to Share One Registration Helper Between the Agent-Safety Installer's PreToolUse Hooks #3517. (introduced, consolidation)

Pre-existing findings, not filed

host-setup/agent-safety/README.md near lines 6, 168, and 214, and host-setup/agent-safety/claude/install.py:196: sentences over the prose gate's optional sentence-length bound, on lines this change touched only incidentally. (pre-existing, style)

🤖 Generated with Claude Code

…Dispatch Skills name a tier and defer its model to the carried AGENTS.md, and a repository whose AGENTS.md predates the tier table leaves `frontier` unresolved, so a session read it as "the frontier model" and dispatched at the most capable one. Close that from three sides: - spec/model-tiers.json states the tier-to-model mapping as data, per tool, and spec/validate.py holds the AGENTS.md table equal to it. - AGENTS.md moves the tier rule into its own "Model Tiers" subsection, says `frontier` is not the industry's frontier model, and has an orchestrator name a tier on every dispatch. The three skills that dispatch by tier carry that subsection as a generated include. - A model-tier-guard PreToolUse hook refuses an Agent or Workflow dispatch at the maintainer-only tier's model unless the session was launched with FLEET_ALLOW_HIGHEST=1, and the host safety block states the same restriction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Declare the hook and its test module in test_cli_usage, and run the test module in the validate action's coverage step. - Match a quoted `model` key in a Workflow script, as JavaScript allows. - State that the maintainer selects the highest tier at session start, which is what the hook enforces, in AGENTS.md and the safety block. - Show each tool's dispatch alias verbatim in the tier table, so a future Codex or opencode row needs no display transform. - Add the guard as requirement 10 of the agent-safety spec and to its audit list, and fold the two matcher helpers into one. - Shorten the new rule sentences the prose gate's sentence check found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-27T16:02:25Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

Source mode: Auto
Repository: ptr727/ProjectTemplate
Branch: feature/model-tier-guard
File: .coderabbit.yaml
Commit: 565bf2a
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: ade28f55-aa7a-48a7-a2b5-040d68119e82

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

ptr727 · 2026-10-27T16:02:27Z

A recorded local strict-review pass covers head 1da6a7d317344e018dfc6c7f731225b550bad8bb, the content this pull request carries at that commit against develop, and it recorded 8 findings.

- Resolve a relative or home-relative Workflow scriptPath against the session's cwd, and refuse a script file the hook cannot read rather than passing it unseen. - Name the configured default tier in the deny text, and match the model name case-insensitively without folding the key's case. - Say the kit refuses a dispatch that names the highest-tier model, which is what the hook can see, and state why requirement 10 sits in the host layer and what it lets through. - Add the guard's self-test to both Verify lists, and drop the incidental requirement counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

codecov · 2026-10-27T16:07:33Z

Codecov Report

❌ Patch coverage is 80.43478% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.53%. Comparing base (3dbbe67) to head (565bf2a).

Files with missing lines	Patch %	Lines
spec/validate.py	80.43%	9 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3516      +/-   ##
===========================================
- Coverage    81.54%   81.53%   -0.01%     
===========================================
  Files           26       26              
  Lines         8918     8964      +46     
===========================================
+ Hits          7272     7309      +37     
- Misses        1646     1655       +9

Flag	Coverage Δ
python-3.13	`81.53% <80.43%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

🟡 Changes recommended

Unresolved fail-closed guard behavior and validation/documentation gaps block safe approval.

14 open issues

Fail closed when scriptPath cannot be read · New
Use consistent model-key matching in fail-closed paths · New
Check missing configuration before launch allowance · New
Describe the configured default model tier · New
Validate the model-tiers configuration structure · New
Limit host-hook claims to inspectable dispatches · New
Add requirement 10 to the specification decision model · New
Remove incidental requirement and hook counts · New
Include the model guard self-test in verification lists · New
Narrow highest-tier enforcement claims to inspectable inputs · New
Share PreToolUse registration validation logic · New
Reuse one helper for hook group registration · New
Remove the incidental requirement count · New
Remove the incidental requirement count · New

What changed in this PR

This pull request centralizes model-tier mappings, propagates tier guidance through fleet skills, and adds a Claude Code guard against unrequested highest-tier dispatches.

Changes:

Adds canonical model-tier configuration and validation against AGENTS.md.
Deploys and tests the Claude Code model-tier guard and registration logic.
Updates safety documentation, generated skills, digests, and CI coverage.

File	Description
`tests/test_spec_validate.py`	Tests model-tier configuration and table invariants.
`tests/test_cli_usage.py`	Classifies the new hook and tests for CLI checks.
`spec/validate.py`	Validates model-tier configuration and documentation consistency.
`spec/model-tiers.json`	Defines the canonical tier-to-model mapping.
`host-setup/agent-safety/README.md`	Documents requirement 10 and host-safety auditing guidance.
`host-setup/agent-safety/opencode/README.md`	Updates opencode audit instructions.
`host-setup/agent-safety/codex/README.md`	Updates Codex audit instructions.
`host-setup/agent-safety/claude/test_model_tier_guard.py`	Tests model-tier guard decision behavior.
`host-setup/agent-safety/claude/test_install.py`	Tests guard deployment, registration, configuration, and digests.
`host-setup/agent-safety/claude/README.md`	Documents the model-tier hook and installation layout.
`host-setup/agent-safety/claude/model-tier-guard.py`	Implements highest-tier dispatch protection.
`host-setup/agent-safety/claude/install.py`	Deploys the mapping and registers the model-tier hook.
`host-setup/agent-safety/claude/claude-md-safety.md`	Adds host-wide model-tier safety guidance.
`AGENTS.md`	Defines model-tier rules and the mapping table.
`.github/skills/unattended-handoff/SKILL.md`	Carries model-tier guidance into the generated skill source.
`.github/skills/local-strict-review/SKILL.md`	Carries model-tier guidance into the generated skill source.
`.github/skills/backlog-burndown/SKILL.md`	Carries model-tier guidance into the generated skill source.
`.github/actions/validate/action.yml`	Runs the new model-tier guard tests in CI.
`.claude-plugin/fleet-skills/skills/unattended-handoff/SKILL.md`	Updates the generated Claude skill distribution.
`.claude-plugin/fleet-skills/skills/local-strict-review/SKILL.md`	Updates the generated Claude skill distribution.
`.claude-plugin/fleet-skills/skills/backlog-burndown/SKILL.md`	Updates the generated Claude skill distribution.
`.claude-plugin/fleet-skills/.source-digests/unattended-handoff`	Refreshes the unattended-handoff skill digest.
`.claude-plugin/fleet-skills/.source-digests/local-strict-review`	Refreshes the local-strict-review skill digest.
`.claude-plugin/fleet-skills/.source-digests/backlog-burndown`	Refreshes the backlog-burndown skill digest.
`.agents/skills/unattended-handoff/SKILL.md`	Updates the authored unattended-handoff skill.
`.agents/skills/local-strict-review/SKILL.md`	Updates the authored local-strict-review skill.
`.agents/skills/backlog-burndown/SKILL.md`	Updates the authored backlog-burndown skill.

_{Review effort: Lite}

ptr727 · 2026-10-27T16:09:17Z

A recorded local strict-review pass covers head b8ef30b3db27bde69222747e58664cbc2119dbc0, the content this pull request carries at that commit against develop, and it recorded 6 findings.

…Tier - unattended-handoff and backlog-burndown dispatch at `highest` only where the session was also launched with it allowed, and stop on a refusal rather than continuing at a tier nobody chose. AGENTS.md names the launch setting and says naming the tier in a prompt is not it. - Match the blocked alias as a word inside a Workflow model value, so a full model id is refused too. - Run the guard's self-test under coverage in the validate action, since the subprocess tests are not traced. - Name the data file in the payload docstring and the --report problem, assert the stale-config test's message, and update the spec's scope and decision-flow summaries for requirement 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The guard's deny text says to stop and report where the maintainer asked for the tier, and names the default tier only otherwise, so it agrees with the skills that forbid dispatching lower. - The unattended-handoff picker step takes the same launch-allowance condition and refusal handling as the worker step. - The guard's documented blind spots name a child workflow a script starts, beside the saved workflow already listed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ptr727 · 2026-10-27T16:20:07Z

A recorded local strict-review pass covers head 20dd597feefe112c78337bcdf953289e42b6f18e, the content this pull request carries at that commit against develop, and it recorded 1 finding.

A PreToolUse hook cannot read the session's own model, so a session launched on the highest tier gives that tier to every dispatch naming no model, whatever the launch setting says. AGENTS.md now says such a session names a tier on every dispatch, and the guard's documented blind spots list the inheriting dispatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Claude Code ignores a fork's model parameter, so naming a tier on a fork does not move it off a highest-tier session's model. A dispatch naming no model also takes its agent definition's model before the session's. AGENTS.md now says both, and the guard's blind-spot lists name the fork. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The guard reads a fork's model like any other dispatch's, so only a fork naming another model passes unseen. "Every other dispatch" read as alternate dispatches, so the rule now says every dispatch meant to run below the highest tier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ptr727 · 2026-10-27T16:28:46Z

A recorded local strict-review pass covers head eeb42a325c184b8b64c98303412c8a3a36c4e73b, the content this pull request carries at that commit against develop, and it recorded 1 finding.

The refusal sentence had drifted past the inheritance sentences, so its "any other dispatch" read as every dispatch naming the tier, allowance or not. The bullet now states who selects the tier, how, and that the kit refuses a dispatch naming it only without the launch setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The rewrite left "naming it partway through a session means restarting the session with it allowed" with no actor, which reads as an instruction to whichever agent sees the name. The bullet now says the maintainer restarts the session, and an agent never sets the allowance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ptr727 · 2026-10-27T16:31:35Z

A recorded local strict-review pass covers head 6fdde9ba6fa836c48201461cd81cbda3ca3545bb, the content this pull request carries at that commit against develop, and it recorded 0 findings.

spec/validate.py now reports a tiers value that is not a list, a tools value that is not an object, and a tool entry missing its name or models object, rather than raising on the first one it meets. The agent-safety status table also drops its count of PreToolUse hooks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A tier that is not a string would otherwise pass the list check and then raise on set(), so the check reuses the existing helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ptr727 · 2026-10-27T16:36:31Z

A recorded local strict-review pass covers head 565bf2a00c94ebb3e2347da78ffdbd61f738df16, the content this pull request carries at that commit against develop, and it recorded 0 findings.

ptr727 and others added 2 commits October 27, 2026 08:47

Copilot AI lite review requested due to automatic review settings October 27, 2026 16:02

ptr727 added the comments Permits the comment lines the pull request adds or edits, which the prose gate otherwise refuses label Oct 27, 2026

Copilot started reviewing on behalf of ptr727 October 27, 2026 16:02

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

ptr727 and others added 2 commits October 27, 2026 09:11

ptr727 mentioned this pull request Oct 27, 2026

Share One Registration Helper Between the Agent-Safety Installer's PreToolUse Hooks #3517

Open

ptr727 and others added 4 commits October 27, 2026 09:24

Split Two Long Sentences in the Guard's Blind-Spot Wording

eeb42a3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ptr727 and others added 3 commits October 27, 2026 09:29

Split the Highest Tier's Launch-Setting Sentence

2a0060b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ptr727 and others added 2 commits October 27, 2026 09:32

Reuse is_str_list for the Tier Config's Tiers Check

565bf2a

A tier that is not a string would otherwise pass the list check and then raise on set(), so the check reuses the existing helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ptr727 merged commit cb82cda into develop Oct 27, 2026
13 checks passed

ptr727 deleted the feature/model-tier-guard branch October 27, 2026 17:14

ptr727 mentioned this pull request Oct 27, 2026

Share One Registration-Check Skeleton Between the Installer's PreToolUse and SessionEnd Hooks #3519

Open

ptr727 mentioned this pull request Oct 27, 2026

Promote Model Tier Config and Shared Installer Hook Registration to Main #3521

Open

Conversation

ptr727 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification

Local Strict Review

Pre-existing findings, not filed

Uh oh!

coderabbitai Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ptr727 commented Oct 27, 2026

Uh oh!

codecov Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ptr727 commented Oct 27, 2026

Uh oh!

ptr727 commented Oct 27, 2026

Uh oh!

ptr727 commented Oct 27, 2026

Uh oh!

ptr727 commented Oct 27, 2026

Uh oh!

ptr727 commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ptr727 commented Oct 27, 2026 •

edited

Loading

coderabbitai Bot commented Oct 27, 2026 •

edited

Loading

codecov Bot commented Oct 27, 2026 •

edited

Loading