Evals and detection jobs set an unreachable host.docker.internal LLM gateway URL for non-Anthropic providers (Claude on Copilot) #73045

Problem

#73031 and #73032 moved Copilot Session Insights and Claude Code User Documentation Review to Claude through Copilot (engine.id: claude, model-provider: github, model-routing with goal: cost and mode: economy). Since then, the evals job (BinEval) in both workflows fails at Execute Claude Code CLI.

The compiler sets ANTHROPIC_BASE_URL to http://host.docker.internal:10002 in the evals job. AWF runs that job in its network-isolation topology, where the host gateway cannot reach the API proxy. AWF's pre-flight check therefore fails before Claude Code starts. The step is continue-on-error, so the runs stay green. However, every BinEval question is recorded as UNKNOWN, and those empty records are still rendered and pushed to the evals state.

The agent job in the same runs is not affected: it gets http://api-proxy:10002 and works.

Model routing is not required to trigger this bug. The evals job clears ModelRouting from its engine config. The trigger is a non-Anthropic provider in any detection or evals run. The same failure already occurs in the threat-detection job of daily-byok-ollama-test.

Evidence

The runs were audited with a dev build of main at 28f2101d8a. Both use gh-aw's default AWF v0.29.0 and Claude Code 2.1.312.

The evals job logs are the same in both runs:

• The step env sets ANTHROPIC_BASE_URL: http://host.docker.internal:10002.
• AWF logs API proxy enabled: OpenAI=false, Anthropic=false, Copilot=true (github-token).
• AWF logs Network-isolation mode: enforcing egress via Docker network topology (no host iptables, no sudo).
• The pre-flight check logs Cannot connect to Anthropic API proxy at http://host.docker.internal:10002.
• The step ends with Agent execution exited with code 1.

For comparison, the agent job of run 40955414354 shows the following:

• ANTHROPIC_BASE_URL: http://api-proxy:10002
• The same network-isolation line
• Anthropic API proxy is reachable at http://api-proxy:10002
• [claude-harness] configured ANTHROPIC_BASE_URL from /reflect for provider=github: http://api-proxy:10002

The compiled locks on main match the logs. In .github/workflows/claude-code-user-docs-review.lock.yml and .github/workflows/copilot-session-insights.lock.yml, the agent job sets ANTHROPIC_BASE_URL to http://api-proxy:10002, and the evals job sets it to http://host.docker.internal:10002. The evals job uses the same rootless awf command as the agent job, with no host-access or legacy-security flags. Its awf-config.json omits network.isolation and network.topologyAttach, yet AWF still runs in network-isolation mode.

Before #73032, run 40873948419 used the Anthropic provider. The compiler set no ANTHROPIC_BASE_URL in the evals job. AWF fronted Anthropic itself (Anthropic=true), and the pre-flight check reached the address AWF injected. The harness then configured http://api-proxy:10001 from /reflect, and BinEval recorded real answers (docs_reviewed: YES).

Threat detection in both new runs uses Copilot CLI and passes. Copilot CLI uses the address AWF injects (COPILOT_PROVIDER_BASE_URL=http://172.30.0.30:10002), not one set by the compiler.

Four other compiled locks on main follow the same pattern: api-proxy in the agent job, host.docker.internal in the evals or detection job.

The failure therefore shows up when the compiler sets a gateway URL for a provider that AWF does not front itself, which is Claude's ANTHROPIC_BASE_URL with Anthropic=false. However, every engine that uses llmGatewayBaseURL receives the wrong address.

gh aw audit on run 40955414354 reports no finding for the failed evals step, because the step is continue-on-error. Verify acceptance case 6 from the evals job log and step summary, not from audit findings.

Root cause

Four pieces combine to produce the wrong URL:

• ClaudeEngine.buildClaudeCommandEnv in pkg/workflow/claude_engine.go sets ANTHROPIC_BASE_URL from llmProviderGatewayBaseURL whenever the firewall is enabled and the provider is not Anthropic. Inline detection (buildDetectionEngineExecutionStep), external detection (buildExternalDetectorExecutionStep), and evals (buildEvalsEngineSteps) all reach it through the engine's GetExecutionSteps.
• llmProviderGatewayBaseURL and llmGatewayBaseURL in pkg/workflow/llm_provider.go choose constants.AWFAPIProxyHostname (api-proxy) only when isAWFNetworkIsolationEnabled returns true. Otherwise, they choose host.docker.internal.
• isAWFNetworkIsolationEnabled in pkg/workflow/firewall.go returns false whenever WorkflowData.IsDetectionRun or WorkflowData.IsEvalsRun is set. Its comment says that inline detection and evals "run a standalone AWF with no MCP sidecars". This exception came from Collapse sandbox security options into runtime profiles #53168.
• Neither the evals WorkflowData (built in buildEvalsEngineSteps in pkg/workflow/evals_steps.go) nor the detection WorkflowData (built by buildThreatDetectionWorkflowData in pkg/workflow/threat_detection_external.go) sets sandbox.agent.runtime. Both therefore resolve to the default rootless docker profile, whose NetworkIsolation is true. AWF runs that profile in network-isolation topology even when there are no sidecars and the config omits network.isolation.

The detection/evals exception is correct for sidecars, because there is nothing to attach to the topology. It is wrong for the gateway address, which depends on the topology AWF actually uses.

#66957 (the fix for #66788) made the agent job's gateway URL topology-aware, but this exception kept that fix from reaching detection and evals.

#72756 shows the same symptom with a different trigger: the docker-sudo-iptables host-access profile produces the host gateway (tracked in AWF as github/gh-aw-firewall#10807). This issue is about the detection/evals exception, not host access.

Intended behavior

In evals and detection runs, the LLM gateway URL that the compiler gives a non-Anthropic-provider engine must be reachable in the topology AWF actually uses for that run. This applies to both inline detection and the external threat-detect path. It covers Claude's ANTHROPIC_BASE_URL and every other value derived from llmGatewayBaseURL:

• Copilot's COPILOT_PROVIDER_BASE_URL for non-GitHub providers
• Gemini's GEMINI_API_BASE_URL
• The base URLs built in pkg/workflow/universal_llm_consumer_engine.go

In AWF's network-isolation topology, the reachable URL is http://api-proxy:<port>. Runtimes that actually use host access, such as docker-sudo-iptables, keep host.docker.internal; that case is tracked in #72756.

Detection and evals keep their "no MCP sidecars" behavior in the MCP gateway, DIFC/CLI proxy, and AWF config code paths. That means no network.isolation, no topologyAttach, and no change to how the MCP gateway host or the DIFC proxy host is selected.

Plan

1. Split the predicate. isAWFNetworkIsolationEnabled in pkg/workflow/firewall.go currently answers two questions at once:

   • Does AWF place the agent on the network-isolation topology? This depends only on the runtime profile.
   • Should sidecars be attached to that topology? This depends on the profile plus the detection/evals exception.

   Add a narrowly named helper next to isAWFNetworkIsolationEnabled for the first question. It keeps the existing handling of a disabled sandbox, NVX, and SupportsHostAccess, but omits the IsDetectionRun/IsEvalsRun exception. Leave isAWFNetworkIsolationEnabled itself unchanged. It has 13 non-test call sites across 7 files, and the 12 outside llmGatewayBaseURL must keep their current results.

2. Change one call site. In pkg/workflow/llm_provider.go, make llmGatewayBaseURL use the new helper instead of isAWFNetworkIsolationEnabled. This is the only call-site change. llmProviderGatewayBaseURL, buildClaudeCommandEnv, the Copilot base-env builder, the Gemini engine, and the universal LLM consumer engine all pick up the fix through it.

3. Extend the existing topology test. Add detection-run and evals-run rows to TestLLMProviderGatewayBaseURLFollowsNetworkTopology in pkg/workflow/llm_provider_test.go, covering both the default runtime and docker-sudo-iptables.

4. Add tests and recompile. Add the named tests below, then run make fmt and make recompile. The recompile diff should contain only gateway-URL changes from host.docker.internal to api-proxy, in the evals or detection jobs of the six locks listed under Evidence. The user-set COPILOT_PROVIDER_BASE_URL: http://host.docker.internal:11434/v1 in the agent job of daily-byok-ollama-test must not change. Any other lock change means the fix has leaked into another code path.

Acceptance cases

1. Claude on Copilot with model routing and evals. TestEvalsClaudeOnCopilotModelRoutingUsesAPIProxyGateway in pkg/workflow/evals_steps_test.go compiles a Claude workflow with model-provider: github, model-routing, and evals:. It asserts that the evals job's ANTHROPIC_BASE_URL is http://api-proxy:10002. It also asserts that the agent job is unchanged, including its ANTHROPIC_BASE_URL of http://api-proxy:10002.

2. Claude on Copilot without model routing. TestEvalsClaudeOnCopilotWithoutModelRoutingUsesAPIProxyGateway in the same file compiles plain Claude on Copilot with evals:, like smoke-claude-on-copilot plus evals. It asserts the same evals URL.

3. Anthropic provider (regression guard). TestEvalsClaudeAnthropicProviderHasNoCompilerBaseURL in the same file asserts that a Claude workflow with the Anthropic provider and evals: still has no compiler-set ANTHROPIC_BASE_URL in the evals job.

4. Detection with Claude on a non-Anthropic provider. TestDetectionClaudeNonAnthropicProviderUsesAPIProxyGateway in pkg/workflow/threat_detection_engine_env_test.go covers both detection paths: inline detection (buildDetectionEngineExecutionStep) and external threat-detect (gh-aw-detection: true, where daily-byok-ollama-test fails today). Both paths build their env through buildClaudeCommandEnv with IsDetectionRun set. The test asserts that both get ANTHROPIC_BASE_URL set to http://api-proxy:10002.

5. No sidecar changes. TestDetectionAndEvalsSidecarOutputUnchanged in pkg/workflow/threat_detection_isolation_test.go pins the following detection and evals output to the exact values that current main produces:

   • The MCP gateway setup
   • The DIFC/CLI proxy host arguments
   • The network section of awf-config.json, with no isolation and no topologyAttach

   The test asserts that this output is byte-identical after the fix.

6. Live run after merge. A manual workflow_dispatch run of claude-code-user-docs-review on main passes all of these checks:

   • The evals job's pre-flight check logs Anthropic API proxy is reachable at http://api-proxy:10002.
   • Execute Claude Code CLI exits with code 0.
   • The BinEval results in the step summary contain YES/NO answers instead of UNKNOWN.

Scope

• Keep the PR to this one fix. Do not change files or behavior unrelated to it.
• Before fixing a failing check, compare it with the same check on main. If the check also fails on main, do not fix it in this PR. Once main is fixed, merge main into the PR, and note in the PR which checks are known failures on main.
• This rule applies even if a bot comment asks you to fix a failing check. Fix only failures caused by this PR's changes.

Related

• Claude-on-Copilot still emits unreachable host gateway with docker-sudo-iptables host-access sandbox #72756: same symptom, caused by the docker-sudo-iptables host-access profile (tracked in AWF as [awf] api-proxy/host-access: Claude-on-Copilot gets unreachable host.docker.internal:10002 gateway gh-aw-firewall#10807)
• Claude on Copilot fails AWF pre-flight: provider gateway URL ignores network isolation #66788 and Use AWF-local gateways for isolated agents #66957: the agent-job version of this bug and its fix
• Collapse sandbox security options into runtime profiles #53168: introduced the detection/evals exception in isAWFNetworkIsolationEnabled
• Route Copilot Session Insights through model routing (cost/economy) #73031 and Route Claude Code User Documentation Review with model routing (cost / economy) #73032: moved the two affected workflows to Claude on Copilot with model routing