Model routing: route Avenger instead of pinning gpt-6-astra #73009

Summary

Replace Avenger's copilot/gpt-6-astra pin with model routing, using goal: cost and mode: balanced. Only the frontmatter of .github/workflows/avenger.md and the recompiled avenger.lock.yml change. Estimated saving: about 10,900 AI Credits (AIC; 1 AIC = $0.01) a week.

Why model routing

A pinned model handles every task, however easy. It also pins the threat-detection and evals jobs, which then cost tens of AIC more per run than they would on their small default models. Model routing picks the model and reasoning effort for each run, based on its task, so the workflow no longer depends on one model name staying available. We're moving gh-aw's own workflows over one at a time, one per PR, and measuring cost and outcomes after each.

The first, Duplicate Code Detector, has been routed since 17 October (#69499):

With so few runs, we read the quality result as "no worse", not "better".

Why Avenger next

• It's one of the most expensive workflows, and its cost is rising. Avenger cost 104, 3,227, 7,173 and 15,462 AIC in the weeks starting 28 September, 5, 12 and 19 October. That is about 11,400 AIC a week over the last two. Its agent runs only while CI on main is red, so its weekly cost follows CI health.
• Its task is the kind the router's benchmarks measure best. Avenger fixes failing builds and tests, which are coding tasks like the ones the router's benchmarks use. The router should classify a run as a medium-difficulty fix within one subsystem.
• A much cheaper choice scores about the same. The routing tables in AWF v0.29.0, gh-aw's current default, pick gpt-5.6-luna at max effort for that task with goal: cost and mode: balanced. On the router's coding benchmarks it scores 46.6, against about 45 for gpt-6-astra across its efforts. It costs about 11 AIC per benchmark task, against 120–300 for gpt-6-astra. We estimate routed Avenger at about 490 AIC a week, including threat detection and evals on their default models.
• A maintainer reviews every change. Avenger opens PRs, and a maintainer merges them, so a weaker fix is caught before it lands. 10 of the 13 Avenger PRs opened from 28 September to 25 October were merged (77%). That's the baseline to watch.

Why balanced, not robust. The docs suggest robust for workflows that open PRs. Here, balanced already matches today's model on benchmark score, and a maintainer reviews every PR. If the merge rate drops, robust picks claude-opus-5.5 at medium effort. That scores 55.5 at about 49 AIC per benchmark task, still well below gpt-6-astra.

Why not auto. Avenger's prompt is the same every hour, apart from the CI run details. The docs recommend a fixed mode when you know what kind of tasks a workflow gets.

Proposed change

In .github/workflows/avenger.md:

• Remove model: copilot/gpt-6-astra.
• Add engine.model-routing with goal: cost and mode: balanced.
• Keep engine.id: copilot and engine.model-provider: github. TestCheckoutWorkflowsUseBashCapableEngine in pkg/workflow/avenger_engine_regression_test.go requires GitHub inference for Avenger.

The resulting engine block:

engine:
  id: copilot
  model-provider: github
  model-routing:
    goal: cost
    mode: balanced

Then run make recompile. Only avenger.md and avenger.lock.yml should change. In the lock file, expect these changes:

• The AWF router image is added to the pinned images.
• A "Prepare model-routing conversation" step is added.
• Model routing is enabled in the agent's AWF config.
• The agent step no longer sets COPILOT_MODEL.
• The routed model, effort and routing status are passed to the downstream jobs.
• Threat detection and evals use the detection and evals default models instead of gpt-6-astra.

On current main (e6afe3424d), this change compiles to exactly that diff, and the test above passes.

After merge

Over the first two weeks:

• Check the first routed runs with gh aw audit <run>. Its Model Routing section shows the selected model and effort, as do the firewall.model_routing records in usage/aw_session.jsonl. Expect mostly gpt-5.6-luna at max. The router classifies each run, so some runs can differ.
• Compare AIC per run, for the agent, threat-detection and evals jobs together, with the pinned baseline.
• Compare the PR merge rate with 77%. If it drops, switch to mode: robust. To roll back, restore the pin and recompile.

Out of scope

• Other workflows. Each gets its own issue.
• Avenger's prompt, schedule, tools, permissions and safe outputs.
• Changes to the router or its tables.

Acceptance criteria

• avenger.md uses engine.model-routing with goal: cost and mode: balanced, and no longer pins a model.
• avenger.lock.yml is regenerated with make recompile, and no other workflow changes.
• TestCheckoutWorkflowsUseBashCapableEngine passes.