Model routing: route Copilot Session Insights instead of defaulting to claude-sonnet-5 #73024

Summary

Route Copilot Session Insights with model routing, using goal: cost and mode: economy, instead of running every day on the Claude engine's default model, claude-sonnet-5. Only the frontmatter of .github/workflows/copilot-session-insights.md and the recompiled copilot-session-insights.lock.yml change. The agent then reaches Claude through GitHub Copilot instead of the Anthropic API key. Estimated saving: about 1,000 AI Credits (AIC; 1 AIC = $0.01) a week.

This is one of four issues filed together for gh-aw's daily Claude Code report workflows. The others are Claude Code User Documentation Review (#73025), Agent Job Health Monitor (#73026) and Safe Output Health Monitor (#73027).

Why model routing

Copilot Session Insights runs every task on one model, however easy. Model routing picks the model and reasoning effort for each run, based on its task, so the workflow no longer depends on one model name staying available. We're moving gh-aw's own workflows over one at a time, one per PR, and measuring cost and outcomes after each.

The first, Duplicate Code Detector, has been routed since 17 October (#69499):

With so few runs, we read the quality result as "no worse", not "better". Avenger is next (#73009).

With the Claude engine, model routing runs through GitHub Copilot (model-provider: github) and chooses only among the claude-* models that Copilot offers. smoke-claude-routed already uses this engine configuration, with its candidates limited to claude-sonnet-5. Its last 12 runs, 18 to 29 October, all succeeded.

Why Copilot Session Insights next

• Almost all of its cost is the agent on claude-sonnet-5. Copilot Session Insights cost 1,510, 25, 1,262 and 826 AIC in the weeks starting 28 September, 5, 12 and 19 October. All seven runs in the week of 5 October failed, so that week cost almost nothing. A run whose agent executes costs about 185 AIC: about 184 for the agent and about 1 for threat detection and evals, which already use their default models. At its recent run rate, that's about 1,110 AIC a week.
• It reads data and writes a report. It analyzes Copilot coding agent sessions and posts usage patterns, success rates and performance metrics as a discussion. We expect the router to classify a run as a medium-difficulty explanation within one subsystem.
• A much cheaper choice scores about the same. For that task, with goal: cost and mode: economy, the routing tables in AWF v0.29.0, gh-aw's current default, pick claude-haiku-5.5 at xhigh effort. On the router's benchmarks it scores 40.7, against about 37 for claude-sonnet-5 across its efforts. It costs about 8 AIC per benchmark task, against 85–107 for claude-sonnet-5 at medium and high. The benchmarks are coding tasks, so they're weaker evidence for a report than for a code fix. We estimate routed runs at about 17 AIC each, about 100 AIC a week.
• A weaker run does little harm. Its only outputs are one discussion in the audits category (an issue, if the discussion can't be created) and its repo memory. All of its last 10 runs, 20 to 29 October, succeeded. That's the baseline to watch.

Why economy. Copilot Session Insights reads data and reports on it, and people read the report before acting on it. economy already picks a model that scores above claude-sonnet-5 on the router's benchmarks. If reports get worse, balanced picks claude-opus-5.5 at low effort. That scores 48.4 at about 31 AIC per benchmark task, still well below claude-sonnet-5.

Why not auto. Copilot Session Insights's prompt is the same every day; only the data it reads changes. The docs recommend a fixed mode when you know what kind of tasks a workflow gets.

Proposed change

In .github/workflows/copilot-session-insights.md:

• Replace engine: claude with an engine block that keeps id: claude.
• Add engine.model-provider: github. The Claude engine needs it for model routing.
• Add engine.model-routing with goal: cost and mode: economy.

The resulting engine block:

engine:
  id: claude
  model-provider: github
  model-routing:
    goal: cost
    mode: economy

Then run make recompile. Only copilot-session-insights.md and copilot-session-insights.lock.yml should change. In the lock file, expect these changes:

• The AWF router image is added to the pinned images.
• A "Prepare model-routing conversation" step is added.
• Model routing is enabled in the agent's AWF config, with github-copilot/claude-* as the candidates.
• The agent no longer uses ANTHROPIC_API_KEY. The secret, its validation step and the GH_AW_MODEL_AGENT_CLAUDE default are removed. Claude Code reaches Copilot through the AWF API proxy, authenticated with the COPILOT_GITHUB_TOKEN secret.
• The routed model, effort and routing status are passed to the downstream jobs.
• Threat detection runs on Copilot CLI with its detection default model, instead of on Claude Code. Evals stay on Claude Code with the evals default model, now through Copilot.

On current main (e6afe3424d), this change compiles to exactly that diff.

The docs note that Claude Code fails at none or minimal effort. The current routing tables have no such choices for Claude models, so the router can't pick them.

After merge

Over the first two weeks:

• Check the first routed runs with gh aw audit <run>. Its Model Routing section shows the selected model and effort, as do the firewall.model_routing records in usage/aw_session.jsonl. Expect mostly claude-haiku-5.5 at xhigh. The router classifies each run, so some runs can differ.
• Compare AIC per run, for the agent and the threat detection and evals jobs together, with about 185 AIC today.
• Check that the repo memory it writes still looks right, since later runs read it.
• Compare the success rate with 10 of its last 10 runs, and read the first few reports against recent ones. If the reports get worse, switch to mode: balanced. To roll back, remove model-provider and model-routing and recompile.

Merge order. We suggest merging Copilot Session Insights (#73024) and Claude Code User Documentation Review (#73025) first. Merge the two health monitors, Agent Job Health Monitor (#73026) and Safe Output Health Monitor (#73027), after the first routed runs of those two look right, on different days, so each change can be measured on its own.

Out of scope

• Other workflows. Each gets its own issue.
• Copilot Session Insights's prompt, schedule, tools, permissions, limits and safe outputs.
• Changes to the router or its tables.

Acceptance criteria

• copilot-session-insights.md sets engine.model-provider: github and engine.model-routing with goal: cost and mode: economy, and keeps engine.id: claude.
• copilot-session-insights.lock.yml is regenerated with make recompile, and no other workflow changes.