Model routing: route Claude Code User Documentation Review instead of defaulting to claude-sonnet-5 #73025

Summary

Route Claude Code User Documentation Review with model routing, using goal: cost and mode: economy, instead of running every day on the Claude engine's default model, claude-sonnet-5. Only the frontmatter of .github/workflows/claude-code-user-docs-review.md and the recompiled claude-code-user-docs-review.lock.yml change. The agent then reaches Claude through GitHub Copilot instead of the Anthropic API key. Estimated saving: about 930 AI Credits (AIC; 1 AIC = $0.01) a week.

This is one of four issues filed together for gh-aw's daily Claude Code report workflows. The others are Copilot Session Insights (#73024), Agent Job Health Monitor (#73026) and Safe Output Health Monitor (#73027).

Why model routing

Claude Code User Documentation Review runs every task on one model, however easy. Model routing picks the model and reasoning effort for each run, based on its task, so the workflow no longer depends on one model name staying available. We're moving gh-aw's own workflows over one at a time, one per PR, and measuring cost and outcomes after each.

The first, Duplicate Code Detector, has been routed since 17 October (#69499):

With so few runs, we read the quality result as "no worse", not "better". Avenger is next (#73009).

With the Claude engine, model routing runs through GitHub Copilot (model-provider: github) and chooses only among the claude-* models that Copilot offers. smoke-claude-routed already uses this engine configuration, with its candidates limited to claude-sonnet-5. Its last 12 runs, 18 to 29 October, all succeeded.

Why Claude Code User Documentation Review next

• Almost all of its cost is the agent on claude-sonnet-5. Claude Code User Documentation Review cost 1,608, 1,985, 1,449 and 595 AIC in the weeks starting 28 September, 5, 12 and 19 October. A run whose agent executes costs about 146 AIC: about 145 for the agent and about 1 for threat detection and evals, which already use their default models. At its recent run rate, that's about 1,020 AIC a week.
• It reads data and writes a report. It reviews gh-aw's documentation as a Claude Code user who doesn't use Copilot, and posts the gaps it finds as a discussion. The persona comes from the prompt, so it doesn't depend on which provider serves the model. We expect the router to classify a run as a medium-difficulty explanation within one subsystem.
• A much cheaper choice scores about the same. For that task, with goal: cost and mode: economy, the routing tables in AWF v0.29.0, gh-aw's current default, pick claude-haiku-5.5 at xhigh effort. On the router's benchmarks it scores 40.7, against about 37 for claude-sonnet-5 across its efforts. It costs about 8 AIC per benchmark task, against 85–107 for claude-sonnet-5 at medium and high. The benchmarks are coding tasks, so they're weaker evidence for a report than for a code fix. We estimate routed runs at about 13 AIC each, about 95 AIC a week.
• A weaker run does little harm. Its only output is one discussion in the audits category (an issue, if the discussion can't be created). All 28 runs from 28 September to 25 October succeeded, and 9 of its last 10. That's the baseline to watch.

Why economy. Claude Code User Documentation Review reads data and reports on it, and people read the report before acting on it. economy already picks a model that scores above claude-sonnet-5 on the router's benchmarks. If reports get worse, balanced picks claude-opus-5.5 at low effort. That scores 48.4 at about 31 AIC per benchmark task, still well below claude-sonnet-5.

Why not auto. Claude Code User Documentation Review's prompt is the same every day; only the data it reads changes. The docs recommend a fixed mode when you know what kind of tasks a workflow gets.

Proposed change

In .github/workflows/claude-code-user-docs-review.md:

• Replace engine: claude with an engine block that keeps id: claude.
• Add engine.model-provider: github. The Claude engine needs it for model routing.
• Add engine.model-routing with goal: cost and mode: economy.

The resulting engine block:

engine:
  id: claude
  model-provider: github
  model-routing:
    goal: cost
    mode: economy

Then run make recompile. Only claude-code-user-docs-review.md and claude-code-user-docs-review.lock.yml should change. In the lock file, expect these changes:

• The AWF router image is added to the pinned images.
• A "Prepare model-routing conversation" step is added.
• Model routing is enabled in the agent's AWF config, with github-copilot/claude-* as the candidates.
• The agent no longer uses ANTHROPIC_API_KEY. The secret, its validation step and the GH_AW_MODEL_AGENT_CLAUDE default are removed. Claude Code reaches Copilot through the AWF API proxy, authenticated with the workflow's github.token, because the workflow already has copilot-requests: write.
• The routed model, effort and routing status are passed to the downstream jobs.
• Threat detection runs on Copilot CLI with its detection default model, instead of on Claude Code. Evals stay on Claude Code with the evals default model, now through Copilot.

On current main (e6afe3424d), this change compiles to exactly that diff.

Compiling prints two new warnings: the doc-reader and engine-example-counter sub-agents use model: small, which "may be incompatible with the claude engine's model-routing API", because the small alias also lists non-Claude models. At runtime the alias resolves to the first available model in its list, and that's a Claude Haiku model on Copilot, so we expect the sub-agents to keep working. If they don't, give them an explicit Claude model.

The docs note that Claude Code fails at none or minimal effort. The current routing tables have no such choices for Claude models, so the router can't pick them.

After merge

Over the first two weeks:

• Check the first routed runs with gh aw audit <run>. Its Model Routing section shows the selected model and effort, as do the firewall.model_routing records in usage/aw_session.jsonl. Expect mostly claude-haiku-5.5 at xhigh. The router classifies each run, so some runs can differ.
• Compare AIC per run, for the agent and the threat detection and evals jobs together, with about 146 AIC today.
• Check that the doc-reader and engine-example-counter sub-agents ran on a Claude model.
• Compare the success rate with 9 of its last 10 runs, and read the first few reports against recent ones. If the reports get worse, switch to mode: balanced. To roll back, remove model-provider and model-routing and recompile.

Merge order. We suggest merging Copilot Session Insights (#73024) and Claude Code User Documentation Review (#73025) first. Merge the two health monitors, Agent Job Health Monitor (#73026) and Safe Output Health Monitor (#73027), after the first routed runs of those two look right, on different days, so each change can be measured on its own.

Out of scope

• Other workflows. Each gets its own issue.
• Claude Code User Documentation Review's prompt, schedule, tools, permissions, limits and safe outputs.
• The sub-agents' model: small.
• Changes to the router or its tables.

Acceptance criteria

• claude-code-user-docs-review.md sets engine.model-provider: github and engine.model-routing with goal: cost and mode: economy, and keeps engine.id: claude.
• claude-code-user-docs-review.lock.yml is regenerated with make recompile, and no other workflow changes.