diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index 0e73a4bc4..f18b7bd73 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -1024,3 +1024,7 @@ re-derive from scratch. stayed at 1 so `max_turns` never tripped — all silently, with `make verify` green. A static rule cannot see an SDK field go dead; a live-telemetry smoke (one real turn, assert `token_usage` is non-empty) in the harness-bump checklist would have. +- [ ] Validation regexes anchored with `$` and applied with `.match` (not `.fullmatch`) accept a trailing newline — `"evil.com\n:443"` passed `normalize_egress_target` until it moved to `fullmatch`. A lint rule needs to tell a validation regex from a search regex; not cheap — caught in the network: llm_only Phase 1 review. +- [ ] `raise X(...) from exc` where `exc` came from parsing an env/URL value leaks the value through `__cause__` in a traceback (urlsplit put a URL password in its port error). Needs data-flow from env reads; not cheap — caught in the network: llm_only Phase 3 review. +- [ ] `await asyncio.to_thread(subprocess.run, ...)` that CREATES a resource inside a try whose `finally` removes it: a cancel leaves the worker thread running, so teardown races the create and leaks it (fixed in `isolation/egress.py` by `_run_to_completion`). Detecting "creates a resource" is not mechanical — caught in the network: llm_only Phase 4 review. +- [ ] The single-`asyncio.shield` join in `fs_permissions.set_permissions` does not survive a SECOND cancel of the caller; `isolation/egress._run_to_completion` loops on the shield. Promote that helper to a shared module and use it in both places — caught in the network: llm_only Phase 4 review. diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md index 03dc8bc8e..27848d06d 100644 --- a/.claude/notes/isolation.md +++ b/.claude/notes/isolation.md @@ -1036,3 +1036,184 @@ separator a value beginning with `-` is parsed as an OPTION rather than a reposi (`--upload-pack=…` runs a command of the caller's choosing). That URL is task-authored, and since `evaluate ` rebuilds the task from a shareable run directory it is no longer necessarily the operator's own string. + +## The egress sidecar (network: llm_only) + +`network: llm_only` puts the task container on a per-task `--internal` network whose only exit +is a proxy sidecar that forwards to an exact `host:port` allowlist. The plan and its spike log +are in `c/2026-10-01-docker-no-internet-egress.md` (not committed); the facts that shaped the +design are below. + +### Why an explicit proxy, not a transparent one + +Harbor's design (a `gost` sidecar, nftables `redirect`, SNI sniffing, the task in the sidecar's +netns with `NET_ADMIN`) also catches tools that ignore proxy env. Phase 0 found no built-in +harness that needs it: Claude Code (Bun native binary), Codex (Rust reqwest), Antigravity (Go +`ProxyFromEnvironment`) and Pi (undici `EnvHttpProxyAgent` with `NODE_USE_ENV_PROXY=1`) all +honour `HTTPS_PROXY`. An explicit proxy needs no capability, no third-party image and no SNI +parser, and a tool that ignores it has no route, so it fails closed. iptables inside the task +container was rejected because it needs `NET_ADMIN` in the agent's own container, which lets the +agent undo it. + +### Why a per-task network + +A shared internal network lets task A use task B's sidecar (and its allowlist) and reach task +B's ports. One network per task costs one address pool each: a default Docker Desktop ran out at +the 30th network (`all predefined address pools have been fully subnetted`), which is why the +error names `--max-parallel` and `default-address-pools`. + +### Why inhibit_ipv4 and ip_forward=0 + +On native Linux a default `--internal` bridge still holds a gateway IP on the host, and the task +container reached every host service bound on `0.0.0.0` through it (dockerd 20.10, 24 and 29, +both firewall backends; on 20.10 and 24 also `docker0` and the host NIC). The +`com.docker.network.bridge.inhibit_ipv4=true` option removes the gateway; the sidecar then takes +the `.1` address and the embedded DNS alias still works. `gateway_mode_ipv4=isolated` was +rejected: Docker 24 ignores it silently. Docker Desktop never showed the hole. +`--sysctl net.ipv4.ip_forward=0` on the sidecar is defence in depth: Docker enables forwarding in +the sidecar's netns, so a task container with `NET_ADMIN` could try to route through it. No +config field grants the task container run-time capabilities, and the measured route did not +reach the internet, but the sysctl costs nothing. + +### Why a black-hole DNS server and --ipv6=false + +The spikes ran on dind, which has no loopback resolver, so they could not show CVE-2024-29018: +on daemons before 26.0 / 25.0.5 / 23.0.11 the embedded DNS forwards an internal network's +queries from the host namespace when the host resolver is loopback (systemd-resolved), which is a +DNS tunnel out. The task container's `--dns 192.0.2.1` (TEST-NET-1, never routed) keeps the +embedded DNS answering the sidecar alias while every external forward goes nowhere. +`inhibit_ipv4` removes only the IPv4 gateway, so `--ipv6=false` keeps a daemon whose +`default-network-opts` enable IPv6 from giving the internal bridge an IPv6 gateway. IPv6 +link-local reach to a host service was not measured. + +`NET_RAW` is dropped from the task container for the same reason: with raw sockets an agent could +write frames for the bridge's own MAC address and reach a host service over UDP without the +sidecar. That path was not measured; the cap drop removes it at no cost, since no tool needs raw +sockets under `llm_only`. + +### Why the sidecar log is a separate, atomically written file + +The first version appended the sidecar log to `docker.log` after the container exited. That file +is in the run directory, which is bind-mounted writable into the container at the SAME path, so +the agent could replace it with a symlink to a host file (for example a shell rc file) and get a +line it chose (`DENY $(cmd):443 CONNECT` passes the visible-ASCII check) appended there by the +host. Container stdout could also forge `ALLOW` / `DENY` lines in it. The log is now `egress.log`, +written once by `write_text_atomic`, whose `os.replace` replaces a planted symlink instead of +following it. `BAD` lines carry only the method and length, because a refused target can hold +credentials in its userinfo or query. + +### Why the framework image and a bind-mounted module + +The sidecar image is the framework image, never the task image: a task image is task-authored and +may be a runtime-kit image with a different Python. The proxy code is NOT taken from the image: +the host bind-mounts its own `egress_proxy.py` read-only, so the code under test is the boundary +that runs and image skew cannot change it. That is also why the module is stdlib-only (five +imports, guarded by a test): the image only supplies `python3`. + +### Why the sidecar watches the heartbeat + +A host SIGKILL skips every `finally`, and so does a SIGTERM (a CI cancel, `timeout`, systemd): +coder-eval has no SIGTERM-to-cancel bridge, and `fs_permissions` re-kills with the default action. +The task container already exits on a stale host heartbeat; +the sidecar reads the same file through a single-file `:ro` bind (Docker Desktop VirtioFS saw +every in-place counter write) and stops itself with the same counter-or-mtime rule as +`heartbeat_is_alive`. It has no `--rm`, so a crashed or stale-stopped sidecar keeps its log until +teardown reads it; `docker network rm` works with a stopped container still attached, so a later +prune needs no ordering. The watchdog returns from `serve` instead of calling `os._exit`, because +CE052 gates process-lethal calls on `IN_CONTAINER_ENV`, which a stdlib-only module cannot import. + +The heartbeat stops the sidecar but cannot remove it or its network, and each leaked network holds +one of the daemon's address pools (about 29 on a default Docker Desktop). A few cancelled parallel +runs then fail every `network create`, for coder-eval and for anything else on that daemon. So every +egress setup first prunes egress-labelled stopped containers and unused networks older than +`RECLAIM_MIN_AGE`. Prune never removes a running container or a network in use, and the age spares a +concurrent setup between `network create` and `start`. A SIGTERM-to-cancel bridge was not added: the +prune also covers SIGKILL, and a process-wide handler would change shutdown for every driver. + +### Why a refused grading host makes the row ERROR + +A grader that cannot reach a host it needs (an `apt-get` mirror, PyPI) fails its criterion, and +the row reads as an ordinary FAILURE: an agent miss that is really the grader's. Only `egress.log` +showed it. So after a graded run the host reads the sidecar log from Docker (not the run +directory's copy, which the container can write), takes the `DENY` lines after `PHASE grading` (or +all of them for a container that only grades), and records the hosts in +`environment_info["egress_denied_in_grading"]`. A FAILURE becomes ERROR with the hosts named. A +SUCCESS stays, because the grader passed without them. + +`GRADING_DENIAL_NOISE` drops the hosts the judge CLIs call and do not need (§ Expected DENY lines +in docs/DOCKER_ISOLATION.md), or every `agent_judge` FAILURE on the direct or litellm route would +become ERROR. A process the agent left running is refused under the same rule, so its row also +becomes ERROR; that misnames an agent miss as an error, but errors count as misses in the pass rate. + +### Why every docker call runs to completion + +`_docker` runs `subprocess.run` in a worker thread, and cancelling the await does not stop the +thread. A Ctrl-C during `docker create` once let teardown run `docker rm -f` before the create +finished, leaking a created-but-never-started sidecar that no heartbeat would ever stop. So every +call is joined across any number of cancels (`_run_to_completion`) before the cancel propagates, +and teardown is joined the same way. + +### The log line format is a security boundary + +`docker.log` is the only evidence the boundary leaves, and operators grep it for +`^(ALLOW|DENY|FAIL)`. A request line with a control or non-ASCII byte is refused before anything is +logged, because a lone `\n` in a method once forged a separate `ALLOW` line. + +### Why the allowlist follows the model callers, not the forwarded env + +The first version allowed the API backend's hosts for every agent and the host of every forwarded +`*_URL` variable. The real runs showed the cost: a Claude task could reach the Azure +`CODEX_BASE_URL` host, and a Codex, Antigravity or Pi task could reach `api.anthropic.com`, +because those variables are on the default passthrough list and the default backend is `direct`. +Each host was a model provider, but none was needed. So the backend's hosts are added only for a +component that uses `API_BACKEND` (`uses_api_backend` on the agent config, an enabled simulator, +an `llm_judge` / `agent_judge` criterion), and a URL variable counts only where its owner reads it +(`LITELLM_BASE_URL` for the litellm backend, `CODEX_BASE_URL` in `CodexAgentConfig.egress_hosts`). +Any other host is explicit in `egress_allowlist`. + +### Why grading switches the sidecar in place + +Graders often install their own test dependencies (SkillsBench test scripts run `apt-get`, the +`astral.sh` uv installer and `uvx` from PyPI). With one allowlist for both phases those hosts are +open to the agent as well, and `github.com`, which the uv installer needs, is where public +benchmark solutions live. So grading may widen the allowlist, never narrow it. + +Grading in a second container (the detached-grade path) was rejected: only the workspace crosses +over, and the known equivalence gaps in § Two known equivalence gaps already score real rows 0.000 +for state outside it. The switch therefore happens in the same container, as in Harbor, at the +two grading sites that run after the agent is finished (the single-shot loop and the post-failure +path). The dialog loop is refused at load, because its criteria run between turns and a one-way +switch would leave the agent the grading network for the rest of the dialog. + +The phase boundary is known only inside the container, so the in-container run asks through a +request file in the run directory and waits for the host's answer in an ack file. The host, not +the container, talks to the proxy: `docker exec` into the sidecar reaches a control listener on +the sidecar's own loopback, which no other netns can reach, and the proxy confirms before the +host answers. The ack is written with `write_text_atomic` because the run directory is writable +from the container. + +This is not a boundary against an agent that targets it. The agent is root in the same +container, so it can write the request file early, and a background process it started keeps +running into grading. Both are the same class as the deliberate re-chmod the anti-cheat window +leaves open. The `PHASE grading` line in `egress.log` separates the two phases' traffic. + +Every graded `llm_only` run switches, including one with no grading allowlist: there the switch +adds nothing and only writes the `PHASE grading` line that § Why a refused grading host makes the +row ERROR reads. Such a switch is best-effort, because a failure only loses that report. + +Grading-only hosts are checked at setup with `probe --direct` (a TCP connection from the sidecar), +not through the proxy, because the proxy does not allow them yet. + +### What Phase 0 measured + +- Every built-in harness in the image and every judge route (`llm_judge`, `agent_judge`, + litellm with aiohttp) works through the proxy with an exact host set. +- Claude Code on Bedrock calls the control plane `bedrock..amazonaws.com` at every start; + it is allowed so the CLI behaves as under `bridge`. `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` + does not remove that call, and on direct its Datadog `DENY` did not slow the CLI, so it is not set. +- Claude Code OAuth refresh goes to `platform.claude.com` (binary string `TOKEN_URL`), which is on + the direct backend list because `agent_judge` runs the CLI too. +- `LITELLM_LOCAL_MODEL_COST_MAP=True` costs no latency either way (the `403` is immediate); it is set + to remove one `DENY` per process and a network-dependent cost map. +- busybox `wget` sends absolute-form `GET https://…` to a proxy instead of `CONNECT`; the proxy + answers `400` with a `BAD … absolute-form https (use CONNECT)` line rather than originate TLS. diff --git a/docs/DOCKER_ISOLATION.md b/docs/DOCKER_ISOLATION.md index 8c6e6383a..a7d399ef0 100644 --- a/docs/DOCKER_ISOLATION.md +++ b/docs/DOCKER_ISOLATION.md @@ -47,10 +47,276 @@ coder-eval run --driver docker sandbox: driver: docker docker: - network: bridge # or "none" for sealed runs + network: bridge # bridge | llm_only | none (see Network modes) image: my-custom:tag # override the default image ``` +## Network modes + +`sandbox.docker.network` selects what the task container can reach. The same mode applies to +the grading container of a detached `coder-eval evaluate`. + +| Mode | The container reaches | Use it for | +|---|---|---| +| `bridge` (default) | the full network | tasks that install packages or call other services | +| `llm_only` | only the model APIs and the hosts in `egress_allowlist`, through a proxy sidecar | agent tasks that must not use general internet | +| `none` | nothing | `agent: {type: none}` tasks only: a real agent cannot reach its model API | + +```yaml +sandbox: + driver: docker + docker: + network: llm_only + egress_allowlist: [pypi.org, files.pythonhosted.org] # optional, appended across layers +``` + +You can also set it per run: `-D sandbox.docker.network=llm_only` and +`-D 'sandbox.docker.egress_allowlist=["pypi.org"]'`. + +### How the egress sidecar works + +For each task the host creates: + +1. A per-task `docker network create --internal --ipv6=false` network with + `com.docker.network.bridge.inhibit_ipv4=true`. The network has no route out and no IPv4 gateway + address on the host, so the task container cannot reach the internet or a host service. +2. An egress-proxy sidecar from the framework image `coder-eval-agent:`. It joins the + internal network (DNS alias `coder-eval-egress`) and the default `bridge`. It runs the host's + own `egress_proxy.py`, bind-mounted read-only, as uid 65534 with all capabilities dropped and a + read-only root file system. It opens TCP connections only to exact `host:port` targets. +3. The task container, on the internal network only, with `HTTPS_PROXY` / `HTTP_PROXY` (both + cases) set to `http://coder-eval-egress:3128`, `NO_PROXY=localhost,127.0.0.1,::1`, + `NODE_USE_ENV_PROXY=1` and `LITELLM_LOCAL_MODEL_COST_MAP=True`. A host value of a proxy + variable (`*_PROXY`, `NO_PROXY`, `NODE_USE_ENV_PROXY`) is never forwarded. Its upstream DNS + server is `192.0.2.1`, an address that is never routed: Docker's own DNS still resolves the + sidecar, but an external name does not resolve. It also runs without `NET_RAW`, so it cannot + send crafted packets onto the internal bridge. + +Before the task container starts, the host sends one `CONNECT` per allowlisted target through +the sidecar. If one fails, the task is an `ERROR` row that names the failing targets. The host +removes the sidecar and the network on every path (success, error, Ctrl-C). If the host process +is killed, the sidecar stops by itself when the host heartbeat goes stale; see +[Troubleshooting egress](#troubleshooting-egress) to remove what is left. + +A tool that ignores the proxy variables has no route, so it fails closed. It cannot bypass the +allowlist. + +### The derived allowlist + +You do not list the model APIs yourself. The host derives them from the parts of the task that +call a model, and adds nothing for the parts that do not: + +| Source | Hosts added | +|---|---| +| API backend (`API_BACKEND`), only when a `claude-code` agent, an enabled `simulation:`, or an `llm_judge` / `agent_judge` criterion uses it | `direct`: `api.anthropic.com:443`, `platform.claude.com:443` (Claude Code OAuth refresh). `bedrock`: `bedrock-runtime..amazonaws.com:443`, `bedrock..amazonaws.com:443`. `litellm`: the host of the forwarded `LITELLM_BASE_URL` (a `localhost` value becomes `host.docker.internal`) | +| Agent | codex: the host of the forwarded `CODEX_BASE_URL`, else `api.openai.com:443`. antigravity: `generativelanguage.googleapis.com:443`. pi: the host of the `provider/` prefix of `agent.model` (`openrouter`, `anthropic`, `openai`, `google`; anything else or no model gives `openrouter.ai:443`). claude-code, opencode, delegate, none: nothing beyond the backend row | +| `system_one_judge` criteria | the host of each `base_url` (default `api.typesafe.ai:443`) | +| `egress_allowlist` | your entries | + +A judge with its own `checker_context.api_route.route` uses the hosts of that backend. A +`route: litellm` judge configures its endpoint through `params`, so add that host to +`egress_allowlist`. Other forwarded `*_URL` variables, such as `UIPATH_URL`, add no host: a task +whose tools call that service lists the host in `egress_allowlist`. A Pi provider other than the +four above, and every OpenCode or Delegate provider, also needs its host in `egress_allowlist`. + +### Extra egress hosts + +`egress_allowlist` takes `host` or `host:port` entries; a bare host means port 443. A host is a DNS +name or an IPv4 address. Schemes, paths, wildcards and IPv6 addresses are refused when the task +loads. Entries are appended across the config layers, like `env_passthrough_extra`. Under `bridge` +or `none` the field is ignored. + +Typical additions: + +| Need | Entries | +|---|---| +| `sandbox.python.env_packages`, `pip`, `uv` | `pypi.org`, `files.pythonhosted.org` | +| `npm`, `npx -y` MCP servers | `registry.npmjs.org` | +| `git clone` over https from GitHub | `github.com` | +| `apt` (Debian) | `deb.debian.org:80` | +| `apt` (Ubuntu) | `archive.ubuntu.com:80`, `security.ubuntu.com:80` (amd64) or `ports.ubuntu.com:80` (arm64) | +| `apk` (Alpine) | `dl-cdn.alpinelinux.org` | +| A remote MCP server or a run-time plugin install | its host | +| `uv` downloading a managed Python not in the image | `github.com`, `objects.githubusercontent.com` (or bake the Python into the image) | + +### Grading network + +A host in `egress_allowlist` is open to the agent too. When only the criteria need it, for example a +test script that installs its test dependencies, give it to grading alone: + +```yaml +sandbox: + driver: docker + docker: + network: llm_only + grading_egress_allowlist: [pypi.org, files.pythonhosted.org, astral.sh] # grading only + # or: grading_network: bridge # grading reaches any host +``` + +| Field | Grading reaches | +|---|---| +| neither (default) | the agent's allowlist | +| `grading_egress_allowlist` | the agent's allowlist plus these hosts (same entry rules as `egress_allowlist`, appended across layers) | +| `grading_network: bridge` | any host, through the sidecar | + +Both fields need `network: llm_only`. `grading_network: bridge` with `network: bridge` changes +nothing; any other combination is refused when the task loads, because the container joins one +network for its whole life and grading can only widen it. A task with an enabled `simulation` is +refused too, because its criteria run between dialog turns. + +The container runs the agent under the agent's allowlist. Before the first criterion runs, the +run inside the container asks the host to switch, the host tells the sidecar through a control +port that only the sidecar itself can reach, and the criteria run after the sidecar confirms. The +switch is one-way and adds a `PHASE grading` line to `egress.log`. If it fails, the row is an +`ERROR` that names the failure. Before the agent starts, the host also opens one TCP connection +from the sidecar to each `grading_egress_allowlist` host, so a host this machine cannot reach +fails the row before any agent run is paid for. + +Every graded `llm_only` run marks the phase, also with neither field set (then the switch adds +no host, and a failed switch is only a warning). After the run, the host checks which hosts the +sidecar refused during grading: + +| Row status | Grading was refused a host | +|---|---| +| `FAILURE` | becomes `ERROR`, with `error_message` naming the hosts to add to `grading_egress_allowlist` | +| any other | kept | + +Either way the hosts are in `environment_info.egress_denied_in_grading`, for example +`["deb.debian.org:80", "pypi.org:443"]`. The judge CLI hosts in [Expected DENY lines](#expected-deny-lines) +(`http-intake.logs.us5.datadoghq.com:443`, `api.anthropic.com:443`) do not count. + +`coder-eval execute` keeps the agent's allowlist. A detached grade (`coder-eval evaluate`, +`run --resume`) runs no agent, so its container gets the grading network from the start; with +`grading_network: bridge` it joins the plain `bridge` network. + +### Tool compatibility + +The task image needs nothing new: its tools must honour the proxy variables. + +| Tool | Under `llm_only` | +|---|---| +| curl, GNU wget, git over https, pip, uv, npm/npx, apt, apk | honour the proxy; a denied host fails fast (`CONNECT tunnel failed, response 403`, `npm error 403`; pip and uv retry for about 8–12 s first) | +| Python urllib / requests / httpx | honour the proxy (`trust_env` is on by default) | +| Python aiohttp | only with `trust_env=True` (litellm sets it); otherwise no route | +| Node `fetch` / `https` | only on Node ≥ 22.21 or ≥ 24.5, through `NODE_USE_ENV_PROXY=1`; older Node has no route. Node prints an `UNDICI-EHPA` warning on stderr | +| Go `net/http` default client | honours the proxy; a custom transport without `Proxy` has no route | +| busybox `wget` (Alpine) | plain http works; https always fails `400`, because it sends `GET https://…` instead of `CONNECT`. Use curl | +| dnf (Rocky, Fedora) | its metalink picks random mirrors, so an exact-host list cannot work. Pin a `baseurl` or bake the packages into the image | +| git over ssh, raw sockets, DNS lookups, Java without proxy flags | no route | + +Every built-in harness in the framework image honours the proxy (Claude Code, Codex, +Antigravity, Pi). See [Run-Limit Parity § Network modes](agents/HARNESS_PARITY.md#network-modes-under-the-docker-driver). + +The sidecar always runs the Debian framework image, so the task image's distribution does not +change the boundary. Debian, Ubuntu, Rocky Linux, Fedora and Alpine task images were tested as +clients. A runtime-kit image needs the framework image too (`make docker-images` builds both). + +**Docker versions.** Docker 20.10 or later (`host-gateway` first shipped there). Docker 26.0, +25.0.5 or 23.0.11 or later is recommended: older daemons forward the DNS queries of an internal +network from the host (CVE-2024-29018). The black-hole upstream DNS server above stops that +forward, but a patched daemon removes the cause. Tested on Docker +Desktop 29 (macOS) and on Linux dockerd 20.10, 24 and 29, with both the `iptables` and the +`nftables` firewall backends. On Linux, `host.docker.internal` resolves to the `docker0` address, +so a LiteLLM proxy on the host must listen on `docker0` or `0.0.0.0`, as under `bridge`. + +### Expected DENY lines + +Some clients call hosts they do not need. These `DENY` lines are harmless: + +| Client | Denied host | +|---|---| +| Claude Code with `API_BACKEND=direct` | `http-intake.logs.us5.datadoghq.com:443` (telemetry) | +| Codex | `chatgpt.com:443`, `github.com:443`, `api.github.com:443` (update check, remote config). Codex without `CODEX_API_KEY` falls back to a ChatGPT login whose model host is `chatgpt.com`, so that setup needs it in `egress_allowlist` | +| litellm without `LITELLM_LOCAL_MODEL_COST_MAP` | `raw.githubusercontent.com:443` (cost map) | +| OpenCode | `models.opencode.ai:443` (model catalog refresh), `registry.npmjs.org:443` (update check) | +| Claude Code with `API_BACKEND=litellm` | `api.anthropic.com:443` (startup calls the CLI does not need on this route) | +| Claude Code `WebFetch` on Bedrock | `api.anthropic.com:443` (the domain safety check before a fetch; `WebFetch` then fails) | + +### Egress limits + +- **Parallel tasks.** Each task uses one Docker network. A default Docker Desktop has address + pools for about 29 user networks, so keep `--max-parallel` under that. When the pools are + exhausted, the row error names `--max-parallel`, the prune command and the daemon's + `default-address-pools` setting. +- **A killed run leaks until the next one.** A SIGTERM or SIGKILL of coder-eval leaves the + task's sidecar (stopped) and network behind. Each `llm_only` setup first removes egress + sidecars that stopped and egress networks that no container uses, once they are 10 minutes old. +- **No upstream proxy.** The sidecar connects directly. A host that can reach the internet only + through a corporate proxy cannot use `llm_only`. +- **Not exfiltration-proof.** The agent can still send data to an allowlisted host. +- **Server-side tools are outside the boundary.** A tool that the model provider runs, such as + Claude Code's `WebSearch` on the direct API, fetches from the internet on the provider's side. + The container network cannot block it; remove it with `disallowed_tools` if the task needs no + internet at all. +- **An exact host is a TCP destination, not a site.** A host behind a shared CDN front (for + example `files.pythonhosted.org` or `deb.debian.org`) can serve other sites that the agent names + in its `Host` header or TLS SNI. +- **`extra_mounts` can defeat the boundary.** Mounting the Docker socket, for example, gives the + agent the daemon. +- **Exact hosts only.** No wildcards and no CIDR ranges. +- **podman and rootless Docker** are not tested. +- **Harbor export** refuses an `llm_only` task. +- **The grading switch trusts the container.** The request to switch comes from inside the + container, so an agent that knows the request file can ask for it before its turn ends. A + process the agent leaves running in the background also keeps running during grading, with the + grading network. Use the grading fields for test dependencies, not to hide hosts from an agent + that is trying to defeat the boundary. A host such a process is refused during grading also + makes a failed row `ERROR`. +- **The sidecar is also on the default `bridge`.** A `bridge` container of another task can reach + it. It gets no extra reach, because the allowlist is a subset of what `bridge` already reaches, + and the sidecar serves at most 256 connections at once (one more gets `503`). +- **Plugin agents.** A third-party agent gets the API backend's hosts only when its config class + sets `uses_api_backend = True`, and its own hosts only through `egress_hosts()`; see + [Extending Coder Eval](EXTENDING.md#the-config-class). + +### Checking the boundary + +`tasks/docker_egress_probe/` tests the mode end to end. The agent tries to reach the internet in +its own ways, and the grading criteria then probe each path out from inside the container: the +proxy, direct IPv4 and IPv6, DNS, raw sockets, the docker host, `pip`, `git` and `npm`. Each +probe passes only when its path is blocked, and the model host must stay reachable: + +```bash +coder-eval run tasks/docker_egress_probe/docker_egress_probe.yaml # expect 1.000 +coder-eval run tasks/docker_egress_probe/docker_egress_probe.yaml \ + -D sandbox.docker.network=bridge # control: expect a FAILURE +``` + +The task pins Claude Code. To check another harness, replace its `agent:` block (without the +Claude-only `setting_sources`). The model-host check accepts the model host of any built-in +harness. Run the probes alone in any `llm_only` container with `python3 egress_probe.py all`. + +`docker_egress_grading.yaml` in the same directory checks the grading network without an agent or +a model API: `pre_run` records that `pypi.org` is blocked, and the criteria then reach it during +grading while `example.com` stays blocked. + +```bash +coder-eval run tasks/docker_egress_probe/docker_egress_grading.yaml # expect 1.000 +``` + +### Troubleshooting egress + +The sidecar log is the task's `egress.log`, beside `docker.log` (a grading container's log is +`grade.egress.log`). It is a separate file so that container output cannot add lines to it. +Each connection is one line: +`ALLOW host:port METHOD`, `DENY host:port METHOD`, `FAIL host:port ` (an allowed host the +sidecar could not reach), or `BAD …` (a request the proxy refused; the line names only its method +and length, because the target can carry credentials). A `PHASE grading allow=…` line marks the +switch to the grading allowlist: the lines before it are the agent's, the lines after it are +grading's (and any process the agent left running). To see which hosts a task needed: + +```bash +grep -rhE "^(ALLOW|DENY|FAIL)" --include=egress.log runs/latest | sort | uniq -c +``` + +Add each needed `DENY` host to `egress_allowlist`. The next `llm_only` task removes what a killed +run leaked once it is 10 minutes old. To remove it now: + +```bash +docker rm -f $(docker ps -aq --filter label=org.coder-eval.egress) +docker network prune -f --filter label=org.coder-eval.egress +``` + ## Using a pre-built custom image When your tasks need extra tools or dependencies, extend the framework image once and point tasks at diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index 32fdccade..bef3866db 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -80,6 +80,14 @@ class MyAgentConfig(BaseAgentConfig): The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config isn't an instance of the registered `config_class`, so keep them paired. +Under `network: llm_only` the container reaches only an allowlist of hosts, and the config class +supplies its share. Set the class variable `uses_api_backend = True` when the agent calls its +model through `API_BACKEND` (as `claude-code` does), and override +`egress_hosts(self, env) -> tuple[str, ...]` to return the `host:port` targets of its own model API +(`env` holds the forwarded variables; `url_egress_target` from `coder_eval.models` turns a URL +into a target). An agent that does neither can reach no model under `llm_only`. See +[Docker Isolation § The derived allowlist](DOCKER_ISOLATION.md#the-derived-allowlist). + To accept `agent.context_window`, declare the inclusive range of tokens your harness can cap its context window to, and apply `self.config.context_window` when the agent starts. A config class that declares nothing rejects the field at load, which is the diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index e8f0c823e..78a0a1257 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -558,6 +558,12 @@ Under `driver: tempdir` only `timeout` is enforced — the agent can consume arbitrary host memory, CPU, and PIDs. Use `driver: docker` when you need the container limits above to actually bind. +Under `driver: docker`, `sandbox.docker.network` selects `bridge` (default), `llm_only` (model +APIs plus `sandbox.docker.egress_allowlist` only) or `none`. See +[Docker Isolation § Network modes](DOCKER_ISOLATION.md#network-modes). Under `llm_only`, +`sandbox.docker.grading_egress_allowlist` and `sandbox.docker.grading_network` give the criteria +hosts the agent cannot reach; see [§ Grading network](DOCKER_ISOLATION.md#grading-network). + ### Recording CLI Invocations `record_cli` shadows executables with generated recording shims, so a task can assert on **what the agent actually ran** without hand-writing a mock: diff --git a/docs/agents/HARNESS_PARITY.md b/docs/agents/HARNESS_PARITY.md index 048c5cb05..c8427de34 100644 --- a/docs/agents/HARNESS_PARITY.md +++ b/docs/agents/HARNESS_PARITY.md @@ -771,6 +771,26 @@ both. See [OpenCode](OPENCODE.md) and [Pi § plugins](PI.md#known-limitations). Full detail: [Pi](PI.md). +## Network modes under the docker driver + +`sandbox.docker.network` has the same meaning on every harness: `bridge` gives the container the +full network, `none` gives it no network, and `llm_only` lets it reach only the model APIs and +`egress_allowlist` through a proxy sidecar +([Docker Isolation § Network modes](../DOCKER_ISOLATION.md#network-modes)). A harness works under +`llm_only` only if its CLI honours the `HTTPS_PROXY` variables; one that does not fails closed. + +| | claude-code | codex | antigravity | opencode | pi | delegate | none | +|---|---|---|---|---|---|---|---| +| `bridge` | works | works | works | works (custom image) | works | works (custom image) | works | +| `none` | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | works | +| `llm_only` honours the proxy | yes | yes | yes | yes (custom image with the CLI) | yes | CLI not in the framework image; untested | no egress needed | +| Verified by | runs R2, R11, R12 (Bedrock); spike S3 (direct) | run R3 (Luna on Azure `CODEX_BASE_URL`) | run R4 | Bedrock and Azure `gpt-5.6-luna` runs (custom image; hosts in `egress_allowlist`) | Bedrock and Azure `gpt-5.6-luna` runs | spike S8 (absent) | — | +| `docker_egress_probe` 9 of 9 | Sonnet 5 on Bedrock | `gpt-5.6-terra` on Azure | `gemini-3.5-flash` | Sonnet 5 on Bedrock (custom image) | Sonnet 5 on Bedrock | not run (no login) | — | + +The hosts each harness gets by default, and the harmless `DENY` lines it produces, are in +[Docker Isolation § The derived allowlist](../DOCKER_ISOLATION.md#the-derived-allowlist) and +[§ Expected DENY lines](../DOCKER_ISOLATION.md#expected-deny-lines). + ## Reproducing `tasks/run_limits/` holds one fixture per limit: `max_turns_cap.yaml` asks for more diff --git a/docs/tutorials/06-use-docker-isolation.md b/docs/tutorials/06-use-docker-isolation.md index 7e3631d14..e2936aa7c 100644 --- a/docs/tutorials/06-use-docker-isolation.md +++ b/docs/tutorials/06-use-docker-isolation.md @@ -160,7 +160,7 @@ Rather than passing the flag every time, set it in the task YAML: sandbox: driver: docker docker: - network: bridge # or "none" for a fully sealed run (no network) + network: bridge # or "llm_only" (model APIs only) or "none" (no network) image: my-custom:tag # optional: override the default framework image ``` diff --git a/src/coder_eval/cli/run_task_internal_command.py b/src/coder_eval/cli/run_task_internal_command.py index 56c60a515..d6c62443f 100644 --- a/src/coder_eval/cli/run_task_internal_command.py +++ b/src/coder_eval/cli/run_task_internal_command.py @@ -16,8 +16,10 @@ import asyncio import contextlib +import functools import logging import os +import time from pathlib import Path from typing import Any @@ -40,11 +42,49 @@ TaskDefinition, ) from coder_eval.orchestration.task_loader import load_task -from coder_eval.path_utils import PRIOR_RESULT_FILENAME +from coder_eval.path_utils import ( + EGRESS_GRADING_ACK_FILENAME, + EGRESS_GRADING_ACK_OK, + EGRESS_GRADING_REQUEST_FILENAME, + PRIOR_RESULT_FILENAME, + write_text_atomic, +) logger = logging.getLogger(__name__) +EGRESS_GRADING_TIMEOUT_SECONDS = 120.0 +_EGRESS_GRADING_POLL_SECONDS = 0.1 + + +async def request_grading_egress( + output_dir: Path, timeout: float = EGRESS_GRADING_TIMEOUT_SECONDS, *, widens: bool = True +) -> None: + """Ask the host to switch the egress sidecar to its grading phase, and wait for its answer. + + A switch that does not ``widen`` egress only marks the phase in the proxy log, so its + failure is a warning rather than a reason to stop grading. + + Raises: + RuntimeError: The host refused a widening switch, or did not answer within ``timeout`` seconds. + """ + ack = output_dir / EGRESS_GRADING_ACK_FILENAME + await asyncio.to_thread(write_text_atomic, output_dir / EGRESS_GRADING_REQUEST_FILENAME, "grading") + deadline = time.monotonic() + timeout + while not (answered := await asyncio.to_thread(ack.is_file)) and time.monotonic() <= deadline: + await asyncio.sleep(_EGRESS_GRADING_POLL_SECONDS) + if not answered: + failure: str | None = f"The host did not switch the egress sidecar in {timeout:g}s." + else: + reply = (await asyncio.to_thread(ack.read_text, encoding="utf-8")).strip() + failure = None if reply == EGRESS_GRADING_ACK_OK else f"The host could not switch the egress sidecar: {reply}" + if failure is None: + logger.info("Egress switched to its grading phase.") + elif widens: + raise RuntimeError(failure) + else: + logger.warning("%s Refused grading hosts will not be reported.", failure) + def _scrub_staged_inputs(task_yaml: Path, context_json: Path) -> None: """Delete the staged ``task.yaml`` and ``context.json`` once loaded, inside a container only. @@ -237,6 +277,11 @@ def run_task_internal_command( from coder_eval.streaming.wire import StdoutNDJsonCallback orchestrator.stream_callback = StdoutNDJsonCallback() + docker = ctx.authored_sandbox.docker + if ctx.grade and docker.network == "llm_only": + orchestrator.before_grading = functools.partial( + request_grading_egress, output_dir, widens=docker.grading_widens_egress + ) asyncio.run(orchestrator.run()) # Orchestrator.run() writes task.json to run_dir (== output_dir). Done. diff --git a/src/coder_eval/egress_proxy.py b/src/coder_eval/egress_proxy.py new file mode 100644 index 000000000..18c29772c --- /dev/null +++ b/src/coder_eval/egress_proxy.py @@ -0,0 +1,476 @@ +"""Stdlib-only egress proxy for ``network: llm_only``; it runs inside the egress sidecar. + +``serve`` forwards ``CONNECT host:port`` and absolute-form plain HTTP only to an exact +allowlisted ``host:port`` and never originates TLS. Every decision is one stdout line +(``ALLOW``, ``DENY``, ``FAIL``, ``BAD``, ``STALE``, ``PHASE``) after one ``READY`` line. +``--heartbeat`` stops it when the host heartbeat goes stale. ``--control`` listens on +loopback for one ``GRADING`` line that logs ``PHASE grading`` and adds ``--grading-allow`` +(or any host, with ``--grading-allow-any``); the switch is one-way. + +``probe`` checks each target through a running proxy (or, with ``--direct``, by a +TCP connection) and exits 0 only when every target is OK. ``grading`` sends the +``GRADING`` line to a ``--control`` address and exits 0 once the proxy confirms. + +The host runs this file with ``python3 -I``, so it imports only ``argparse``, +``asyncio``, ``os``, ``sys`` and ``time``. Importing it starts nothing. + +Rationale: .claude/notes/isolation.md § The egress sidecar (network: llm_only) +""" + +import argparse +import asyncio +import os +import sys +import time + + +HEAD_LIMIT_BYTES = 64 * 1024 +HEAD_TIMEOUT_SECONDS = 30.0 +DIAL_TIMEOUT_SECONDS = 10.0 +PIPE_CHUNK_BYTES = 64 * 1024 +MAX_CONNECTIONS = 256 +MAX_HEARTBEAT_POLL_SECONDS = 2.0 +PROBE_STARTUP_RETRY_SECONDS = 5.0 + +_BAD_REQUEST = b"HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_FORBIDDEN = b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_BAD_GATEWAY = b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_UNAVAILABLE = b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_ESTABLISHED = b"HTTP/1.1 200 Connection Established\r\n\r\n" +_CONTROL_GRADING = b"GRADING\n" +_CONTROL_OK = b"OK grading\n" + + +def _log(line: str) -> None: + print(line, flush=True) + + +class Policy: + """Where the proxy forwards: ``allow`` from the start, plus the grading hosts once grading starts.""" + + def __init__( + self, allow: frozenset[str], grading_allow: frozenset[str] = frozenset(), grading_any: bool = False + ) -> None: + self.allow = allow + self.grading_allow = grading_allow + self.grading_any = grading_any + self.grading = False + + def permits(self, destination: str) -> bool: + if destination in self.allow: + return True + return self.grading and (self.grading_any or destination in self.grading_allow) + + def describe_grading(self) -> str: + if self.grading_any: + return "*" + return "+" + ",".join(sorted(self.grading_allow)) if self.grading_allow else "unchanged" + + +def _redacted(request_line: bytes) -> str: + """A refused request line, shown as its method and length only: the target may carry credentials.""" + method = request_line.split(b" ", 1)[0] + shown = method.decode("ascii") if method.isalpha() and method.isascii() and len(method) <= 16 else "?" + return f"{shown} unparseable request line ({len(request_line)} bytes)" + + +def _is_visible_ascii(data: bytes) -> bool: + return all(0x20 <= byte < 0x7F for byte in data) + + +def _parse_port(text: str) -> int | None: + if not (text.isascii() and text.isdigit()): + return None + port = int(text) + return port if 1 <= port <= 65535 else None + + +def split_host_port(authority: str, default_port: int | None) -> tuple[str, int] | None: + """Split ``host[:port]`` (or ``[v6]:port``) into a lowercased host and a port. + + Returns None when the host is empty, the port is invalid, or a port is + required (``default_port is None``) and absent. + """ + if authority.startswith("["): + end = authority.find("]") + if end < 0: + return None + host, rest = authority[1:end], authority[end + 1 :] + if rest and not rest.startswith(":"): + return None + port_text = rest[1:] if rest else "" + else: + host, sep, port_text = authority.rpartition(":") + if not sep: + host, port_text = authority, "" + if not host: + return None + if not port_text: + if default_port is None: + return None + return host.lower(), default_port + port = _parse_port(port_text) + return None if port is None else (host.lower(), port) + + +def heartbeat_alive(current: str, last_counter: str, current_mtime: float, last_mtime: float) -> bool: + """True when the heartbeat counter text changed or its mtime advanced.""" + return bool(current and current != last_counter) or current_mtime > last_mtime + + +def _read_heartbeat(path: str) -> tuple[str, float]: + try: + with open(path, encoding="utf-8") as handle: + current = handle.read() + except (OSError, UnicodeDecodeError): + current = "" + try: + mtime = os.stat(path).st_mtime + except OSError: + mtime = 0.0 + return current, mtime + + +async def watch_heartbeat(path: str, stale_seconds: float) -> None: + """Return once the heartbeat at ``path`` has not changed for ``stale_seconds``.""" + poll_seconds = min(MAX_HEARTBEAT_POLL_SECONDS, stale_seconds / 4) + last_counter, last_mtime = "", 0.0 + last_change = time.monotonic() + while True: + current, mtime = await asyncio.to_thread(_read_heartbeat, path) + now = time.monotonic() + if heartbeat_alive(current, last_counter, mtime, last_mtime): + last_counter, last_mtime, last_change = current, mtime, now + if now - last_change > stale_seconds: + _log(f"STALE heartbeat {path} unchanged for more than {stale_seconds:g}s; stopping") + return + await asyncio.sleep(poll_seconds) + + +async def _reply_and_close(writer: asyncio.StreamWriter, response: bytes) -> None: + try: + writer.write(response) + await writer.drain() + except (ConnectionError, OSError) as exc: + _log(f"BAD client gone before the reply: {exc!r}") + finally: + writer.close() + + +async def _pipe(reader: asyncio.StreamReader, writer: asyncio.StreamWriter, peer: asyncio.StreamWriter) -> None: + try: + while data := await reader.read(PIPE_CHUNK_BYTES): + writer.write(data) + await writer.drain() + if writer.can_write_eof(): + writer.write_eof() + except (ConnectionError, OSError): + writer.close() + peer.close() + + +def _origin_form_head(method: str, path: str, version: str, header_block: bytes) -> bytes: + kept = [ + line + for line in header_block.split(b"\r\n") + if line and not line.lower().startswith((b"proxy-", b"connection:")) + ] + request_line = f"{method} {path} {version}".encode("latin-1") + return b"\r\n".join([request_line, *kept, b"Connection: close"]) + b"\r\n\r\n" + + +def _parse_absolute_http(target: str) -> tuple[str, int, str] | None: + rest = target[len("http://") :] + cut = min((i for i in (rest.find(c) for c in "/?#") if i >= 0), default=len(rest)) + authority, path = rest[:cut], rest[cut:] + path = path.split("#", 1)[0] + if not path.startswith("/"): + path = "/" + path + parsed = split_host_port(authority.rpartition("@")[2], 80) + return None if parsed is None else (*parsed, path) + + +async def handle_client( + policy: Policy, client_reader: asyncio.StreamReader, client_writer: asyncio.StreamWriter +) -> None: + """Serve one client connection: parse its request head, then tunnel, forward or refuse.""" + try: + head = await asyncio.wait_for(client_reader.readuntil(b"\r\n\r\n"), HEAD_TIMEOUT_SECONDS) + except asyncio.LimitOverrunError: + _log("BAD head-too-large") + client_writer.close() + return + except (asyncio.IncompleteReadError, TimeoutError, ConnectionError, OSError): + client_writer.close() + return + request_line, _, header_block = head[:-4].partition(b"\r\n") + fields = request_line.decode("latin-1").split(" ") + if len(fields) != 3 or not _is_visible_ascii(request_line) or not fields[2].startswith("HTTP/"): + _log(f"BAD {_redacted(request_line)}") + await _reply_and_close(client_writer, _BAD_REQUEST) + return + method, target, version = fields + prefix = b"" + if method == "CONNECT": + parsed = split_host_port(target, None) + elif target.lower().startswith("http://"): + absolute = _parse_absolute_http(target) + parsed = None if absolute is None else absolute[:2] + if absolute is not None: + prefix = _origin_form_head(method, absolute[2], version, header_block) + elif target.lower().startswith("https://"): + https = split_host_port(target[len("https://") :].split("/", 1)[0].rpartition("@")[2], 443) + shown = f"{https[0]}:{https[1]}" if https else repr(target) + _log(f"BAD {shown} absolute-form https (use CONNECT)") + await _reply_and_close(client_writer, _BAD_REQUEST) + return + else: + parsed = None + if parsed is None: + _log(f"BAD {_redacted(request_line)}") + await _reply_and_close(client_writer, _BAD_REQUEST) + return + host, port = parsed + destination = f"{host}:{port}" + if not policy.permits(destination): + _log(f"DENY {destination} {method}") + await _reply_and_close(client_writer, _FORBIDDEN) + return + try: + upstream_reader, upstream_writer = await asyncio.wait_for( + asyncio.open_connection(host, port), DIAL_TIMEOUT_SECONDS + ) + except (OSError, TimeoutError) as exc: + _log(f"FAIL {destination} {exc!r}") + await _reply_and_close(client_writer, _BAD_GATEWAY) + return + _log(f"ALLOW {destination} {method}") + try: + if method == "CONNECT": + client_writer.write(_ESTABLISHED) + await client_writer.drain() + else: + upstream_writer.write(prefix) + await upstream_writer.drain() + await asyncio.gather( + _pipe(client_reader, upstream_writer, client_writer), + _pipe(upstream_reader, client_writer, upstream_writer), + ) + except (ConnectionError, OSError) as exc: + _log(f"FAIL {destination} {exc!r}") + finally: + upstream_writer.close() + client_writer.close() + + +async def start_proxy(policy: Policy, host: str, port: int) -> asyncio.Server: + """Start listening; the returned server is already accepting connections. + + At most ``MAX_CONNECTIONS`` clients are served at once; one more gets ``503``. + """ + active = 0 + + async def _handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + nonlocal active + if active >= MAX_CONNECTIONS: + _log("BAD too-many-connections") + await _reply_and_close(writer, _UNAVAILABLE) + return + active += 1 + try: + await handle_client(policy, reader, writer) + finally: + active -= 1 + + return await asyncio.start_server(_handle, host, port, limit=HEAD_LIMIT_BYTES) + + +async def start_control(policy: Policy, host: str, port: int) -> asyncio.Server: + """Listen for the one-way ``GRADING`` switch; anything else is refused and changes nothing.""" + + async def _handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + try: + line = await asyncio.wait_for(reader.readline(), HEAD_TIMEOUT_SECONDS) + except (TimeoutError, ConnectionError, OSError): + writer.close() + return + if line != _CONTROL_GRADING: + _log("BAD control request") + await _reply_and_close(writer, b"BAD\n") + return + if not policy.grading: + policy.grading = True + _log(f"PHASE grading allow={policy.describe_grading()}") + await _reply_and_close(writer, _CONTROL_OK) + + return await asyncio.start_server(_handle, host, port) + + +async def serve( + policy: Policy, + host: str, + port: int, + heartbeat: str | None, + stale_seconds: float, + control: tuple[str, int] | None = None, +) -> None: + """Run the proxy until the heartbeat goes stale (forever when ``heartbeat`` is None).""" + server = await start_proxy(policy, host, port) + control_server = None if control is None else await start_control(policy, *control) + grading = f" grading={policy.describe_grading()}" if control is not None else "" + _log(f"READY {host}:{port} allow={','.join(sorted(policy.allow))}{grading}") + try: + if heartbeat is None: + await server.serve_forever() + else: + await watch_heartbeat(heartbeat, stale_seconds) + finally: + server.close() + server.abort_clients() + if control_server is not None: + control_server.close() + + +async def _probe_one(proxy_host: str, proxy_port: int, target: str, timeout: float) -> str | None: + started = time.monotonic() + while True: + try: + reader, writer = await asyncio.wait_for(asyncio.open_connection(proxy_host, proxy_port), timeout) + break + except (OSError, TimeoutError) as exc: + if time.monotonic() - started >= PROBE_STARTUP_RETRY_SECONDS: + return f"proxy unreachable: {exc!r}" + await asyncio.sleep(0.2) + try: + writer.write(f"CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n\r\n".encode("latin-1")) + await writer.drain() + status_line = await asyncio.wait_for(reader.readline(), timeout) + except (OSError, TimeoutError) as exc: + return repr(exc) + finally: + writer.close() + status = status_line.split() + if len(status) >= 2 and status[1] == b"200": + return None + return status_line.decode("latin-1").strip() or "no response" + + +async def _dial_one(target: str, timeout: float) -> str | None: + parsed = split_host_port(target, None) + if parsed is None: + return "invalid target (expected host:port)" + try: + _, writer = await asyncio.wait_for(asyncio.open_connection(*parsed), timeout) + except (OSError, TimeoutError) as exc: + return repr(exc) + writer.close() + return None + + +async def probe(proxy: str | None, targets: list[str], timeout: float) -> int: + """CONNECT to each target through ``proxy`` (dial it directly when None); 0 when all succeed, else 1.""" + if proxy is None: + errors = await asyncio.gather(*(_dial_one(target, timeout) for target in targets)) + else: + parsed = split_host_port(proxy, None) + if parsed is None: + _log(f"FAIL {proxy} invalid --proxy (expected host:port)") + return 1 + errors = await asyncio.gather(*(_probe_one(*parsed, target, timeout) for target in targets)) + for target, error in zip(targets, errors, strict=True): + _log(f"OK {target}" if error is None else f"FAIL {target} {error}") + return 0 if all(error is None for error in errors) else 1 + + +async def request_grading(control: str, timeout: float) -> int: + """Switch the proxy listening on ``control`` to grading; 0 once it confirms, else 1.""" + parsed = split_host_port(control, None) + if parsed is None: + _log(f"FAIL {control} invalid --control (expected host:port)") + return 1 + try: + reader, writer = await asyncio.wait_for(asyncio.open_connection(*parsed), timeout) + except (OSError, TimeoutError) as exc: + _log(f"FAIL {control} control unreachable: {exc!r}") + return 1 + try: + writer.write(_CONTROL_GRADING) + await writer.drain() + reply = await asyncio.wait_for(reader.readline(), timeout) + except (OSError, TimeoutError) as exc: + _log(f"FAIL {control} {exc!r}") + return 1 + finally: + writer.close() + _log(reply.decode("latin-1").strip() or "FAIL no reply") + return 0 if reply == _CONTROL_OK else 1 + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(prog="egress_proxy", description=__doc__.split("\n", 1)[0] if __doc__ else None) + commands = parser.add_subparsers(dest="command", required=True) + serve_cmd = commands.add_parser("serve", help="run the allowlisting proxy") + serve_cmd.add_argument("--listen", default="0.0.0.0:3128", help="host:port to listen on") + serve_cmd.add_argument("--heartbeat", default=None, help="host heartbeat file; stop when it goes stale") + serve_cmd.add_argument("--stale", type=float, default=None, help="seconds without a heartbeat change") + serve_cmd.add_argument("--allow", action="append", default=[], help="allowed host:port (repeatable)") + serve_cmd.add_argument("--control", default=None, help="loopback host:port for the GRADING switch") + serve_cmd.add_argument( + "--grading-allow", action="append", default=[], help="host:port also allowed after GRADING (repeatable)" + ) + serve_cmd.add_argument("--grading-allow-any", action="store_true", help="allow any host:port after GRADING") + probe_cmd = commands.add_parser("probe", help="CONNECT to each target through a running proxy") + via = probe_cmd.add_mutually_exclusive_group(required=True) + via.add_argument("--proxy", help="proxy host:port") + via.add_argument("--direct", action="store_true", help="open a TCP connection to each target instead") + probe_cmd.add_argument("--timeout", type=float, default=5.0, help="per-target timeout in seconds") + probe_cmd.add_argument("targets", nargs="+", help="host:port targets") + grading_cmd = commands.add_parser("grading", help="switch a running proxy to its grading allowlist") + grading_cmd.add_argument("--control", required=True, help="the proxy's --control host:port") + grading_cmd.add_argument("--timeout", type=float, default=5.0, help="timeout in seconds") + return parser + + +def _parse_allow(entries: list[str], flag: str) -> set[str] | None: + allow: set[str] = set() + for entry in entries: + parsed = None if "[" in entry else split_host_port(entry.strip(), None) + if parsed is None or ":" in parsed[0]: + _log(f"BAD {flag} {entry!r} (expected host:port)") + return None + allow.add(f"{parsed[0]}:{parsed[1]}") + return allow + + +def main(argv: list[str] | None = None) -> int: + """Command-line entry point; returns the process exit code.""" + args = _build_parser().parse_args(argv) + if args.command == "probe": + return asyncio.run(probe(None if args.direct else args.proxy, args.targets, args.timeout)) + if args.command == "grading": + return asyncio.run(request_grading(args.control, args.timeout)) + listen = split_host_port(args.listen, None) + if listen is None: + _log(f"BAD --listen {args.listen!r} (expected host:port)") + return 2 + if args.heartbeat is not None and not (args.stale is not None and 0 < args.stale < float("inf")): + _log("BAD --heartbeat needs a finite, positive --stale") + return 2 + control = None if args.control is None else split_host_port(args.control, None) + if args.control is not None and (control is None or control[0] not in ("127.0.0.1", "localhost")): + _log(f"BAD --control {args.control!r} (expected a loopback host:port)") + return 2 + if (args.grading_allow or args.grading_allow_any) and control is None: + _log("BAD --grading-allow and --grading-allow-any need --control") + return 2 + allow = _parse_allow(args.allow, "--allow") + grading_allow = _parse_allow(args.grading_allow, "--grading-allow") + if allow is None or grading_allow is None: + return 2 + policy = Policy(frozenset(allow), frozenset(grading_allow), args.grading_allow_any) + asyncio.run(serve(policy, listen[0], listen[1], args.heartbeat, args.stale or 0.0, control)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/coder_eval/harbor/packager.py b/src/coder_eval/harbor/packager.py index fe87b4a5a..7532e416a 100644 --- a/src/coder_eval/harbor/packager.py +++ b/src/coder_eval/harbor/packager.py @@ -158,6 +158,12 @@ def export_resolved_task( + "Set sandbox.driver: docker (with dockerfile_path or a custom image) to export this task." ) + if task.sandbox.docker.network == "llm_only": + raise TaskNotExportableError( + f"Task {task.task_id!r}: Harbor v1 export does not map network: llm_only; use bridge or none, " + + "or export with a Harbor allowlist by hand." + ) + if task.dataset is not None: # `load_task` does NOT run `expand_dataset` -- fan-out happens later, so # exporting a raw dataset-backed task would emit ONE Harbor task still diff --git a/src/coder_eval/isolation/docker_runner.py b/src/coder_eval/isolation/docker_runner.py index 3e5c7fe1d..264b8d859 100644 --- a/src/coder_eval/isolation/docker_runner.py +++ b/src/coder_eval/isolation/docker_runner.py @@ -25,6 +25,22 @@ import yaml from coder_eval.config import settings +from coder_eval.isolation.egress import ( + DOCKER_HOST_ALIAS, + EGRESS_PROXY_MODULE, + PROXY_ENV_NAMES, + EgressHandle, + EgressPlan, + egress_scope, + enter_grading, + forwarded_env_names, + grading_denials, + read_proxy_log, + resolve_egress_targets, + rewrite_loopback_for_container, + task_container_egress_argv, +) +from coder_eval.isolation.errors import DockerRunError, EgressSetupError from coder_eval.logging_config import DEFAULT_LOG_TAIL_MAX_BYTES from coder_eval.models import ( CONTAINER_GRADE_WORKSPACE, @@ -48,6 +64,10 @@ from coder_eval.orchestration.evaluation import resolve_host_reference_dir from coder_eval.path_utils import ( DOCKER_LOG_FILENAME, + EGRESS_GRADING_ACK_FILENAME, + EGRESS_GRADING_ACK_OK, + EGRESS_GRADING_REQUEST_FILENAME, + EGRESS_LOG_FILENAME, PRIOR_RESULT_FILENAME, REFERENCE_COPY_IGNORE, TASK_JSON_FILENAME, @@ -76,28 +96,6 @@ # Rationale: .claude/notes/isolation.md § The entrypoint and the image contract CONTAINER_ENTRYPOINT = "/usr/local/bin/coder_eval_entrypoint.sh" -# Docker Desktop's stable host alias from a bridge-network container. Auto-resolves -# on macOS/Windows; on Linux it must be published via `--add-host`. -_DOCKER_HOST_ALIAS = "host.docker.internal" -_LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"}) - - -def _rewrite_loopback_for_container(url: str) -> str | None: - """Rewrite a loopback URL to the docker host alias, preserving scheme/port/path. - - Returns the rewritten URL, or None if the host is not loopback (forward as-is). - A LiteLLM proxy on the HOST is unreachable at localhost from inside a bridge - container, so ``http://localhost:4000`` -> ``http://host.docker.internal:4000``. - """ - from urllib.parse import urlsplit, urlunsplit - - parts = urlsplit(url) - if parts.hostname not in _LOOPBACK_HOSTS: - return None - netloc = _DOCKER_HOST_ALIAS if parts.port is None else f"{_DOCKER_HOST_ALIAS}:{parts.port}" - return urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment)) - - # DENYLIST of top-level entries the per-task RW copy of ~/.claude skips. Matched by # basename at every level, so anything unlisted (settings.json, .credentials.json, # plugins/) is copied through. @@ -133,6 +131,7 @@ def _rewrite_loopback_for_container(url: str) -> str | None: HEARTBEAT_FILENAME = ".coder_eval_host_heartbeat" HEARTBEAT_INTERVAL_SECONDS = 2.0 HEARTBEAT_STALE_SECONDS = 20 +_GRADING_REQUEST_POLL_SECONDS = 0.25 # asyncio's StreamReader caps a line at 64 KiB by default, which a single stream # event can exceed. The same KIND of guard as Orchestrator._POST_RUN_STREAM_LIMIT, @@ -172,6 +171,47 @@ async def _heartbeat_loop(heartbeat_path: Path) -> None: pass +def _network_name(network: str, egress: EgressHandle | None) -> str: + """The ``--network`` value; ``llm_only`` without its sidecar refuses rather than falls back to bridge.""" + if network == "llm_only": + if egress is None: + raise DockerRunError("network: llm_only needs its egress sidecar; refusing to fall back to bridge.") + return egress.network + return network + + +def _remove_grading_handshake(output_dir: Path) -> None: + for name in (EGRESS_GRADING_REQUEST_FILENAME, EGRESS_GRADING_ACK_FILENAME): + (output_dir / name).unlink(missing_ok=True) + + +async def _serve_grading_request(output_dir: Path, egress: EgressHandle) -> None: + """Switch the sidecar to its grading phase once the container asks, then answer in the ack file.""" + request = output_dir / EGRESS_GRADING_REQUEST_FILENAME + while not await asyncio.to_thread(request.exists): + await asyncio.sleep(_GRADING_REQUEST_POLL_SECONDS) + try: + await enter_grading(egress) + reply = EGRESS_GRADING_ACK_OK + logger.info("Egress sidecar %s switched to its grading phase.", egress.sidecar) + except EgressSetupError as exc: + logger.warning("%s", exc) + reply = f"error: {exc}" + # Atomic: the run dir is writable from the container, so a planted symlink is replaced, never followed. + await asyncio.to_thread(write_text_atomic, output_dir / EGRESS_GRADING_ACK_FILENAME, reply) + + +def _prepare_egress_dir(egress_dir: Path) -> None: + """Stage this host's own proxy module for the sidecar's read-only bind mount. + + The host's copy, not the image's, so the code under test is the boundary that + runs. Readable by the sidecar's uid 65534. + """ + egress_dir.mkdir() + shutil.copy2(Path(__file__).resolve().parents[1] / EGRESS_PROXY_MODULE, egress_dir / EGRESS_PROXY_MODULE) + grant_container_access(egress_dir, writable=False) + + def _preflight() -> None: """Verify ``docker`` is on PATH and the daemon is reachable. @@ -352,17 +392,6 @@ def _validate_extra_mount(spec: str) -> str: return f"{expanded_src}:{dst}:{mode}" -class DockerRunError(RuntimeError): - """Raised when ``docker run`` exits non-zero AND no task.json was produced. - - Criterion failures do NOT raise this -- the container always writes - task.json (with whatever results it has) before exiting, and the host - parses that regardless of exit code. This is reserved for setup-time - failures: missing image, daemon down, OOM-kill before the agent started, - etc. - """ - - class DockerBuildError(DockerRunError): """Raised when ``docker build`` itself fails (the image never builds). @@ -647,8 +676,11 @@ async def run(self) -> EvaluationResult: # Bound BEFORE the try: the `finally` restores it, and `_stage_inputs` # can raise before the widening happens. widened_workspace: list[tuple[Path, int]] = [] + heartbeat_task: asyncio.Task[None] | None = None + log_path = self.rt.run_dir / DOCKER_LOG_FILENAME try: + egress_plan = self._resolve_egress_plan() await self._stage_inputs(input_dir) # Stable and UNIQUE so cancellation can target it: PID alone collides @@ -662,6 +694,12 @@ async def run(self) -> EvaluationResult: await asyncio.to_thread(self._prepare_host_mounts, staging) await asyncio.to_thread(self._prepare_reference_mount, staging) await asyncio.to_thread(self._prepare_task_dir_mount, staging) + egress_dir = staging / "egress" + if egress_plan is not None: + try: + await asyncio.to_thread(_prepare_egress_dir, egress_dir) + except OSError as exc: + raise EgressSetupError(f"network: llm_only could not stage the egress proxy: {exc}") from exc # AFTER staging, BEFORE the container starts: the DAC caps are dropped, so # every framework-owned mount must be reachable through its `other` bits. # Writable so the entry point can delete the staged task.yaml/context.json. @@ -673,37 +711,70 @@ async def run(self) -> EvaluationResult: # bits cannot be assumed -- and the one that SURVIVES the dispatch, # which is why it is recorded and restored in the `finally` below. widened_workspace = await asyncio.to_thread(grant_container_access, self.grade_workspace, writable=True) - argv = self._build_argv(input_dir, output_dir, container_name=container_name, image=image) - logger.info("Running task '%s' in docker: %s", self.rt.task.task_id, " ".join(argv)) - # Prime the heartbeat before the container starts so the watchdog never sees an initial stale state. + # Primed before the egress sidecar too: it bind-mounts this one file, and a + # missing single-file `-v` source makes Docker create a directory there. heartbeat_path = output_dir / HEARTBEAT_FILENAME await asyncio.to_thread(heartbeat_path.touch) heartbeat_task = asyncio.create_task(_heartbeat_loop(heartbeat_path)) - proc = await asyncio.create_subprocess_exec( - *argv, - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.STDOUT, - limit=STDOUT_LINE_LIMIT_BYTES, + scope: contextlib.AbstractAsyncContextManager[EgressHandle | None] = ( + egress_scope( + container_name=container_name, + image=get_default_docker_image_tag(), + egress_dir=egress_dir, + heartbeat=heartbeat_path, + stale_seconds=HEARTBEAT_STALE_SECONDS, + plan=egress_plan, + log_path=self.rt.run_dir / EGRESS_LOG_FILENAME, + ) + if egress_plan is not None + else contextlib.nullcontext(None) ) - log_path = self.rt.run_dir / DOCKER_LOG_FILENAME - log_fh = await asyncio.to_thread(log_path.open, "w", encoding="utf-8") - # HAZARD: `docker run --rm` does NOT propagate a kill daemon-side, so - # without this `finally` Ctrl-C leaves the container burning budget. - # Rationale: .claude/notes/isolation.md § A container that produced no task.json - try: - returncode = await self._stream_container_output(proc, log_fh) - finally: + async with scope as egress: + argv = self._build_argv( + input_dir, output_dir, container_name=container_name, image=image, egress=egress + ) + logger.info("Running task '%s' in docker: %s", self.rt.task.task_id, " ".join(argv)) + proc = await asyncio.create_subprocess_exec( + *argv, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + limit=STDOUT_LINE_LIMIT_BYTES, + ) + log_fh = await asyncio.to_thread(log_path.open, "w", encoding="utf-8") + grading_watch = ( + asyncio.create_task(_serve_grading_request(output_dir, egress)) + if egress is not None and egress.plan.grading_phase + else None + ) + # HAZARD: `docker run --rm` does NOT propagate a kill daemon-side, so + # without this `finally` Ctrl-C leaves the container burning budget. + # Rationale: .claude/notes/isolation.md § A container that produced no task.json + try: + returncode = await self._stream_container_output(proc, log_fh) + finally: + if grading_watch is not None: + grading_watch.cancel() + with contextlib.suppress(asyncio.CancelledError): + await grading_watch + await asyncio.to_thread(_remove_grading_handshake, output_dir) + await asyncio.to_thread(log_fh.close) + # Cancelled mid-flight: kill the container AND the docker CLI subprocess, best-effort. + if proc.returncode is None: + await self._kill_container(proc, container_name) + + result = await self._parse_result_or_raise(output_dir, returncode, log_path) + if egress is not None and self.grade: + await self._record_grading_denials(result, egress, output_dir / TASK_JSON_FILENAME) + return result + except EgressSetupError as exc: + await self._write_synthetic_task_json(self.rt.run_dir / TASK_JSON_FILENAME, exc) + raise + finally: + if heartbeat_task is not None: heartbeat_task.cancel() # Narrowed so a genuine KeyboardInterrupt / SystemExit from a parallel sibling still propagates. with contextlib.suppress(asyncio.CancelledError): await heartbeat_task - await asyncio.to_thread(log_fh.close) - # Cancelled mid-flight: kill the container AND the docker CLI subprocess, best-effort. - if proc.returncode is None: - await self._kill_container(proc, container_name) - - return await self._parse_result_or_raise(output_dir, returncode, log_path) - finally: # rmtree_restrictive, not ignore_errors: `staging` holds the references # copy, which a container killed mid-turn leaves at mode 000. # Rationale: .claude/notes/isolation.md § Why the framework mounts are writable copies @@ -712,6 +783,61 @@ async def run(self) -> EvaluationResult: # the modes it had. See `restore_modes`. await asyncio.to_thread(restore_modes, widened_workspace) + def _container_network(self) -> str: + """The network this container joins: a detached grade with ``grading_network: bridge`` gets ``bridge``.""" + cfg = self._docker_config + if self.grade_workspace is not None and cfg.network == "llm_only" and cfg.grading_network == "bridge": + return "bridge" + return cfg.network + + def _resolve_egress_plan(self) -> EgressPlan | None: + """This container's ``network: llm_only`` allowlists, or None for any other network. + + ``execute`` keeps the agent's allowlist, a detached grade gets the grading one from + the start, and a graded run switches when its grading starts. + + Raises: + EgressSetupError: A forwarded URL, judge ``base_url`` or ``AWS_REGION`` cannot be allowlisted. + """ + if self._container_network() != "llm_only": + return None + try: + targets = tuple(resolve_egress_targets(self.rt.task, env=os.environ, settings=settings)) + except ValueError as exc: + raise EgressSetupError(str(exc)) from exc + cfg = self._docker_config + if not self.grade: + return EgressPlan(targets) + grading_any = cfg.grading_network == "bridge" + grading_targets = () if grading_any else tuple(t for t in cfg.grading_egress_allowlist if t not in targets) + if self.grade_workspace is not None: + return EgressPlan(tuple(sorted({*targets, *grading_targets}))) + return EgressPlan(targets, grading_targets, grading_any, grading_phase=True) + + async def _record_grading_denials(self, result: EvaluationResult, egress: EgressHandle, task_json: Path) -> None: + """Record the hosts grading was refused, and make a FAILURE graded without them an ERROR. + + Rationale: .claude/notes/isolation.md § Why a refused grading host makes the row ERROR + """ + proxy_log = await read_proxy_log(egress) + denied = [] if proxy_log is None else grading_denials(proxy_log, whole_log=self.grade_workspace is not None) + if not denied: + return + hosts = ", ".join(denied) + logger.warning("Task '%s': network: llm_only refused grading access to %s.", self.rt.task.task_id, hosts) + result.environment_info["egress_denied_in_grading"] = denied + if result.final_status is FinalStatus.FAILURE: + result.final_status = FinalStatus.ERROR + result.error_message = ( + f"Grading was refused access to {hosts} under network: llm_only, so the grader may have failed, " + + "not the agent. Add each host to sandbox.docker.grading_egress_allowlist and grade the run again " + + "(`coder-eval evaluate `)." + ) + try: + await asyncio.to_thread(write_text_atomic, task_json, result.model_dump_json(indent=2)) + except OSError as exc: + logger.warning("Could not record the refused grading hosts in %s: %s", task_json, exc) + async def _stage_inputs(self, input_dir: Path) -> None: """Serialise the post-override TaskDefinition and the ``ContainerContext`` into the staging ``input_dir`` (``task.yaml`` + ``context.json``), keeping the contract on @@ -818,7 +944,8 @@ async def _stream_container_output(self, proc: asyncio.subprocess.Process, log_f async def _kill_container(self, proc: asyncio.subprocess.Process, container_name: str) -> None: """Best-effort teardown when cancelled mid-stream with the container still alive. - Called from ``run``'s inner ``finally`` (after heartbeat-cancel + log-fh close), + Called from ``run``'s inner ``finally`` (after the log-fh close; the heartbeat is + cancelled later, after the egress teardown, so a live sidecar never goes stale), guarded by ``if proc.returncode is None``. ``docker run --rm`` does NOT propagate a host-side kill to the daemon, so kill the container by name and then the docker CLI subprocess. No exception leaks from cleanup; suppression is narrowed to @@ -1341,7 +1468,13 @@ def _auto_mount(raw_path: str | None, *, dir_only: bool = True) -> None: logger.warning(_MASK_WARNING, masked_dir, root) def _build_argv( - self, input_dir: Path, output_dir: Path, *, container_name: str, image: str | None = None + self, + input_dir: Path, + output_dir: Path, + *, + container_name: str, + image: str | None = None, + egress: EgressHandle | None = None, ) -> list[str]: cfg = self._docker_config # _build_argv stays PURE -- no side effects -- so it remains testable without @@ -1368,10 +1501,8 @@ def _build_argv( "DAC_READ_SEARCH", ] - if cfg.network == "none": - argv += ["--network", "none"] - else: - argv += ["--network", "bridge"] + network = self._container_network() + argv += ["--network", _network_name(network, egress)] if self._limits.max_memory_mb: argv += ["--memory", f"{self._limits.max_memory_mb}m"] @@ -1383,12 +1514,15 @@ def _build_argv( # Explicit allowlist. `--env VAR` (name-only) tells docker to copy the value # from our env at run time, so secrets stay out of the argv we log. # Rationale: .claude/notes/isolation.md § Environment forwarding - merged_allowlist = set(cfg.env_passthrough) | set(cfg.env_passthrough_extra) + merged_allowlist = forwarded_env_names(self.rt.task) for env_var in merged_allowlist: # LITELLM_BASE_URL / LITELLM_COST_LOG are forwarded below with a value # rewrite (host alias / absolute mount path), not name-only. if env_var in ("LITELLM_BASE_URL", "LITELLM_COST_LOG"): continue + # Under llm_only a forwarded host proxy setting must never shadow the sidecar's. + if egress is not None and env_var in PROXY_ENV_NAMES: + continue if env_var in os.environ: argv += ["--env", env_var] @@ -1396,10 +1530,13 @@ def _build_argv( # to the docker host alias and publish that alias. Only a URL, so an # explicit `--env VAR=value` is safe in the logged argv -- unlike the token. litellm_base_url = os.environ.get("LITELLM_BASE_URL") - if litellm_base_url and "LITELLM_BASE_URL" in merged_allowlist and cfg.network != "none": - rewritten = _rewrite_loopback_for_container(litellm_base_url) + if litellm_base_url and "LITELLM_BASE_URL" in merged_allowlist and network != "none": + rewritten = rewrite_loopback_for_container(litellm_base_url) if rewritten is not None: - argv += ["--env", f"LITELLM_BASE_URL={rewritten}", "--add-host", f"{_DOCKER_HOST_ALIAS}:host-gateway"] + argv += ["--env", f"LITELLM_BASE_URL={rewritten}"] + # Under llm_only only the sidecar dials the host, so it carries the alias. + if egress is None: + argv += ["--add-host", f"{DOCKER_HOST_ALIAS}:host-gateway"] else: argv += ["--env", "LITELLM_BASE_URL"] @@ -1407,7 +1544,7 @@ def _build_argv( # in-container cost join, so bind-mount its dir at the SAME host path # read-only and forward the resolved ABSOLUTE path. litellm_cost_log = os.environ.get("LITELLM_COST_LOG") - if litellm_cost_log and "LITELLM_COST_LOG" in merged_allowlist and cfg.network != "none": + if litellm_cost_log and "LITELLM_COST_LOG" in merged_allowlist and network != "none": abs_log = Path(litellm_cost_log).expanduser().resolve() if abs_log.parent.is_dir(): argv += ["-v", f"{abs_log.parent}:{abs_log.parent}:ro", "--env", f"LITELLM_COST_LOG={abs_log}"] @@ -1421,6 +1558,8 @@ def _build_argv( # name-only, so it overrides any inherited or baked-in value. # Rationale: .claude/notes/isolation.md § Environment forwarding argv += ["--env", "TELEMETRY_ENABLED=false"] + if egress is not None: + argv += task_container_egress_argv() # Read-WRITE: the entry point deletes the staged task.yaml and context.json # after load, and `rm` fails with EROFS on a `:ro` bind mount. diff --git a/src/coder_eval/isolation/egress.py b/src/coder_eval/isolation/egress.py new file mode 100644 index 000000000..f32a3f622 --- /dev/null +++ b/src/coder_eval/isolation/egress.py @@ -0,0 +1,613 @@ +"""Host side of ``network: llm_only``: the egress allowlist and the per-task proxy sidecar. + +Imports only :mod:`coder_eval.isolation.errors`, :mod:`coder_eval.models` and +:mod:`coder_eval.path_utils`, so ``docker_runner`` can import it without a cycle. + +Rationale: .claude/notes/isolation.md § The egress sidecar (network: llm_only) +""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +import subprocess +from dataclasses import dataclass +from typing import TYPE_CHECKING +from urllib.parse import urlsplit, urlunsplit + +from coder_eval.isolation.errors import EgressSetupError +from coder_eval.models import ( + LOOPBACK_HOSTS, + AgentJudgeCriterion, + BedrockRoute, + DirectRoute, + LiteLLMRoute, + LLMJudgeCriterion, + SystemOneJudgeCriterion, + normalize_egress_target, + resolve_evaluation_route, + resolve_route, + url_egress_target, +) +from coder_eval.path_utils import write_text_atomic + + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Awaitable, Mapping, Sequence + from pathlib import Path + + from coder_eval.config import Settings + from coder_eval.models import ApiRoute, TaskDefinition + + +logger = logging.getLogger(__name__) + +# Docker Desktop's stable host alias from a bridge-network container. Auto-resolves +# on macOS/Windows; on Linux it must be published via `--add-host`. +DOCKER_HOST_ALIAS = "host.docker.internal" +_DIRECT_TARGETS = ("api.anthropic.com:443", "platform.claude.com:443") + + +def rewrite_loopback_for_container(url: str) -> str | None: + """Rewrite a loopback URL to the docker host alias, preserving scheme/port/path. + + Returns the rewritten URL, or None if the host is not loopback (forward as-is). + A LiteLLM proxy on the HOST is unreachable at localhost from inside a bridge + container, so ``http://localhost:4000`` -> ``http://host.docker.internal:4000``. + """ + parts = urlsplit(url) + if parts.hostname not in LOOPBACK_HOSTS: + return None + netloc = DOCKER_HOST_ALIAS if parts.port is None else f"{DOCKER_HOST_ALIAS}:{parts.port}" + return urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment)) + + +def forwarded_env_names(task: TaskDefinition) -> set[str]: + """Names of the host variables the container receives: ``env_passthrough`` plus ``env_passthrough_extra``.""" + docker = task.sandbox.docker + return set(docker.env_passthrough) | set(docker.env_passthrough_extra) + + +_LITELLM_BASE_KWARGS = ("api_base", "base_url") + + +def _bedrock_targets(region: str) -> list[str]: + try: + return [ + normalize_egress_target(f"bedrock-runtime.{region}.amazonaws.com"), + normalize_egress_target(f"bedrock.{region}.amazonaws.com"), + ] + except ValueError: + raise ValueError("AWS_REGION is not a valid region name for a Bedrock host.") from None + + +def _litellm_base_url(route: LiteLLMRoute, forwarded: Mapping[str, str]) -> tuple[str, str | None]: + """``(source, url)`` of the endpoint a LiteLLM route calls; ``url`` is None when it is not known here.""" + if route.params is None and route.env_params is None: + return "LITELLM_BASE_URL", forwarded.get("LITELLM_BASE_URL") + for kwarg in _LITELLM_BASE_KWARGS: + literal = (route.params or {}).get(kwarg) + if isinstance(literal, str): + return f"checker_context.api_route.params.{kwarg}", literal + env_name = (route.env_params or {}).get(kwarg) + if env_name: + return env_name, forwarded.get(env_name) + return "checker_context.api_route", None + + +def _route_targets(route: ApiRoute, forwarded: Mapping[str, str]) -> list[str]: + if isinstance(route, DirectRoute): + return list(_DIRECT_TARGETS) + if isinstance(route, BedrockRoute): + return _bedrock_targets(route.region) + source, url = _litellm_base_url(route, forwarded) + if not url: + logger.warning("The LiteLLM endpoint of %s is not forwarded: add its host to egress_allowlist.", source) + return [] + try: + url = rewrite_loopback_for_container(url) or url + except ValueError: + raise ValueError(f"{source} has a host or port that network: llm_only cannot allowlist.") from None + target = url_egress_target(source, url) + return [target] if target is not None else [] + + +def _model_routes(task: TaskDefinition, settings: Settings) -> list[ApiRoute]: + """The routes the agent, the dialog simulator and the LLM judges call, resolved as the orchestrator does.""" + try: + agent_route = resolve_route(settings) + except (AssertionError, ValueError): + logger.warning( + "API_BACKEND=%s is not fully configured (AWS_REGION / AWS_BEARER_TOKEN_BEDROCK, or " + + "LITELLM_BASE_URL / LITELLM_AUTH_TOKEN): no model-API host is allowlisted.", + settings.api_backend, + ) + return [] + routes: list[ApiRoute] = [] + if task.agent is not None and task.agent.uses_api_backend: + routes.append(agent_route) + if task.simulation is not None and task.simulation.enabled: + routes.append(resolve_evaluation_route(settings, agent_route)) + if any(isinstance(c, LLMJudgeCriterion | AgentJudgeCriterion) and c.enabled for c in task.success_criteria): + api_route = task.checker_context.api_route if task.checker_context else None + try: + routes.append( + resolve_evaluation_route( + settings, + agent_route, + backend_override=api_route.route.value if api_route and api_route.route else None, + model_override=api_route.model if api_route else None, + params_override=api_route.params if api_route else None, + env_params_override=api_route.env_params if api_route else None, + ) + ) + except ValueError: + logger.warning("checker_context.api_route is not fully configured: no judge host is allowlisted.") + return routes + + +def resolve_egress_targets(task: TaskDefinition, *, env: Mapping[str, str], settings: Settings) -> list[str]: + """Every ``host:port`` the task's container may reach under ``network: llm_only``. + + The union of the hosts of each model route the task calls (see + ``_model_routes``), the agent config's ``egress_hosts``, each + ``system_one_judge`` ``base_url``, and ``sandbox.docker.egress_allowlist``. + ``env`` is the host environment; only the variables the container receives + count. Pure: reads only its arguments. Sorted and de-duplicated. + + Raises: + ValueError: A LiteLLM endpoint, ``CODEX_BASE_URL``, a judge ``base_url`` or + ``AWS_REGION`` names a host that cannot be allowlisted. + """ + forwarded = {name: env[name] for name in forwarded_env_names(task) if env.get(name)} + targets: set[str] = set() + for route in _model_routes(task, settings): + targets.update(_route_targets(route, forwarded)) + if task.agent is not None: + targets.update(task.agent.egress_hosts(forwarded)) + for criterion in task.success_criteria: + if isinstance(criterion, SystemOneJudgeCriterion): + target = url_egress_target("system_one_judge base_url", criterion.base_url) + if target is not None: + targets.add(target) + targets.update(task.sandbox.docker.egress_allowlist) + return sorted(targets) + + +EGRESS_PROXY_MODULE = "egress_proxy.py" +EGRESS_PROXY_ALIAS = "coder-eval-egress" +EGRESS_PROXY_PORT = 3128 +# Loopback inside the sidecar's own netns: only a `docker exec` there can reach it, never the task container. +EGRESS_CONTROL = "127.0.0.1:3129" +EGRESS_LABEL = "org.coder-eval.egress" +SIDECAR_EGRESS_DIR = "/work/egress" +SIDECAR_HEARTBEAT = "/work/heartbeat" +PROXY_URL = f"http://{EGRESS_PROXY_ALIAS}:{EGRESS_PROXY_PORT}" +NO_PROXY_HOSTS = "localhost,127.0.0.1,::1" +# Never forwarded from the host under llm_only: a host value would shadow the sidecar's. +PROXY_ENV_NAMES = frozenset( + { + "HTTPS_PROXY", + "HTTP_PROXY", + "https_proxy", + "http_proxy", + "ALL_PROXY", + "all_proxy", + "NO_PROXY", + "no_proxy", + "NODE_USE_ENV_PROXY", + } +) +# TEST-NET-1 (RFC 5737): never routed. As the task container's upstream resolver it keeps +# Docker's embedded DNS answering the sidecar alias while external lookups go nowhere, even on +# a daemon that forwards internal-network queries from the host namespace (CVE-2024-29018). +BLACKHOLE_DNS = "192.0.2.1" +PRUNE_HINT = ( + f"docker rm -f $(docker ps -aq --filter label={EGRESS_LABEL}); " + + f"docker network prune -f --filter label={EGRESS_LABEL}" +) + +# Old enough that no live setup is still between `network create` and `start`. +RECLAIM_MIN_AGE = "10m" +# Hosts the judge CLIs call and do not need (docs/DOCKER_ISOLATION.md § Expected DENY lines). +GRADING_DENIAL_NOISE = frozenset({"http-intake.logs.us5.datadoghq.com:443", "api.anthropic.com:443"}) + +_POOL_EXHAUSTED = "all predefined address pools have been fully subnetted" +_DOCKER_TIMEOUT_SECONDS = 30.0 +_PROBE_TIMEOUT_SECONDS = 5 +_NETWORK_RM_ATTEMPTS = 5 +_NETWORK_RM_RETRY_SECONDS = 1.0 + + +@dataclass(frozen=True) +class EgressPlan: + """What one container's sidecar allows: ``targets`` from the start, and what grading adds. + + ``grading_phase`` means the container announces when grading starts, so the sidecar + marks it in its log; ``grading_targets`` are the hosts grading adds, and + ``grading_any`` lets grading reach any host. + """ + + targets: tuple[str, ...] + grading_targets: tuple[str, ...] = () + grading_any: bool = False + grading_phase: bool = False + + def __post_init__(self) -> None: + if self.switches_for_grading and not self.grading_phase: + raise ValueError("A grading allowlist needs a grading phase to switch at.") + + @property + def switches_for_grading(self) -> bool: + return self.grading_any or bool(self.grading_targets) + + +@dataclass(frozen=True) +class EgressHandle: + """The per-task internal network and proxy sidecar a ``network: llm_only`` container joins.""" + + network: str + sidecar: str + plan: EgressPlan + + +def task_container_egress_argv() -> list[str]: + """``docker run`` arguments the task container gets under ``network: llm_only``. + + No raw sockets (no crafted frames onto the internal bridge), a black-hole upstream + resolver, and explicit (non-secret) ``--env`` pairs: the proxy + variables point every tool at the sidecar, and ``LITELLM_LOCAL_MODEL_COST_MAP`` stops + litellm fetching its cost map from a host that is not allowlisted. + """ + argv: list[str] = ["--cap-drop", "NET_RAW", "--dns", BLACKHOLE_DNS] + for name in ("HTTPS_PROXY", "HTTP_PROXY", "https_proxy", "http_proxy"): + argv += ["--env", f"{name}={PROXY_URL}"] + for name in ("NO_PROXY", "no_proxy"): + argv += ["--env", f"{name}={NO_PROXY_HOSTS}"] + return [*argv, "--env", "NODE_USE_ENV_PROXY=1", "--env", "LITELLM_LOCAL_MODEL_COST_MAP=True"] + + +def build_network_create_argv(network: str) -> list[str]: + """``docker`` arguments that create the per-task internal network with no host-reachable gateway.""" + return [ + "network", + "create", + "--internal", + "--ipv6=false", + "-o", + "com.docker.network.bridge.inhibit_ipv4=true", + "--label", + f"{EGRESS_LABEL}=1", + network, + ] + + +def build_sidecar_create_argv( + *, + sidecar: str, + network: str, + image: str, + egress_dir: Path, + heartbeat: Path, + stale_seconds: float, + plan: EgressPlan, +) -> list[str]: + """``docker`` arguments that create (not start) the proxy sidecar. Pure.""" + argv = [ + "create", + "--name", + sidecar, + "--label", + f"{EGRESS_LABEL}=1", + "--network", + network, + "--network-alias", + EGRESS_PROXY_ALIAS, + "--add-host", + f"{DOCKER_HOST_ALIAS}:host-gateway", + "--sysctl", + "net.ipv4.ip_forward=0", + "--user", + "65534:65534", + "--cap-drop", + "ALL", + "--security-opt", + "no-new-privileges", + "--read-only", + "--memory", + "256m", + "--pids-limit", + "256", + "-v", + f"{egress_dir}:{SIDECAR_EGRESS_DIR}:ro", + "-v", + f"{heartbeat}:{SIDECAR_HEARTBEAT}:ro", + "--entrypoint", + "python3", + image, + "-I", + f"{SIDECAR_EGRESS_DIR}/{EGRESS_PROXY_MODULE}", + "serve", + "--listen", + f"0.0.0.0:{EGRESS_PROXY_PORT}", + "--heartbeat", + SIDECAR_HEARTBEAT, + "--stale", + f"{stale_seconds:g}", + ] + for target in plan.targets: + argv += ["--allow", target] + if plan.grading_phase: + argv += ["--control", EGRESS_CONTROL] + if plan.grading_any: + argv.append("--grading-allow-any") + for target in plan.grading_targets: + argv += ["--grading-allow", target] + return argv + + +def build_probe_argv(sidecar: str, targets: Sequence[str], *, direct: bool = False) -> list[str]: + """``docker`` arguments that CONNECT to every target through the running sidecar. + + ``direct`` dials each target from the sidecar instead, for hosts the proxy does not allow yet. + """ + via = ["--direct"] if direct else ["--proxy", f"{EGRESS_PROXY_ALIAS}:{EGRESS_PROXY_PORT}"] + return [ + "exec", + sidecar, + "python3", + "-I", + f"{SIDECAR_EGRESS_DIR}/{EGRESS_PROXY_MODULE}", + "probe", + *via, + "--timeout", + str(_PROBE_TIMEOUT_SECONDS), + *targets, + ] + + +def build_grading_argv(sidecar: str) -> list[str]: + """``docker`` arguments that switch the running sidecar to its grading allowlist.""" + return [ + "exec", + sidecar, + "python3", + "-I", + f"{SIDECAR_EGRESS_DIR}/{EGRESS_PROXY_MODULE}", + "grading", + "--control", + EGRESS_CONTROL, + ] + + +def _docker_sync(args: Sequence[str], timeout: float) -> subprocess.CompletedProcess[str]: + return subprocess.run( + ["docker", *args], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + check=False, + timeout=timeout, + ) + + +async def _docker(*args: str, timeout: float = _DOCKER_TIMEOUT_SECONDS) -> subprocess.CompletedProcess[str]: + """Run one ``docker`` command to its end, even when the caller is cancelled meanwhile. + + A cancelled worker thread keeps running, so a cancel that did not wait for it + would let teardown race a ``docker create`` still in flight and leak its container. + """ + return await _run_to_completion(asyncio.to_thread(_docker_sync, args, timeout)) + + +async def _checked(what: str, *args: str, timeout: float = _DOCKER_TIMEOUT_SECONDS) -> str: + """Run one setup step; return stdout, or raise EgressSetupError naming the step.""" + try: + result = await _docker(*args, timeout=timeout) + except (OSError, subprocess.SubprocessError) as exc: + raise EgressSetupError(f"network: llm_only could not {what}: {exc}") from exc + if result.returncode != 0: + detail = (result.stderr or result.stdout).strip() + if _POOL_EXHAUSTED in detail: + raise EgressSetupError( + "network: llm_only could not create its per-task network: Docker has no free address pool " + + f"({_POOL_EXHAUSTED}). Lower --max-parallel, remove leaked networks with " + + f"`docker network prune --filter label={EGRESS_LABEL}`, or widen the daemon's " + + "`default-address-pools` setting." + ) + raise EgressSetupError(f"network: llm_only could not {what}: {detail}") + return result.stdout + + +async def _probe(sidecar: str, targets: Sequence[str], *, direct: bool = False) -> None: + try: + result = await _docker(*build_probe_argv(sidecar, targets, direct=direct)) + except (OSError, subprocess.SubprocessError) as exc: + raise EgressSetupError(f"network: llm_only egress probe could not run: {exc}") from exc + if result.returncode == 0: + return + failing = [line.removeprefix("FAIL ") for line in result.stdout.splitlines() if line.startswith("FAIL ")] + detail = "; ".join(failing) or (result.stderr or result.stdout).strip() or f"exit code {result.returncode}" + if direct: + raise EgressSetupError( + f"network: llm_only could not reach a grading host from the egress sidecar: {detail}. Check that " + + "this host can reach the hosts in sandbox.docker.grading_egress_allowlist." + ) + raise EgressSetupError( + f"network: llm_only egress probe failed: {detail}. Check that this host can reach these targets, add " + + "a missing host with sandbox.docker.egress_allowlist, and note that chaining to an upstream " + + "(corporate) proxy is not supported." + ) + + +async def enter_grading(handle: EgressHandle) -> None: + """Switch the sidecar to its grading phase; returns once the proxy confirms. + + Raises: + EgressSetupError: The sidecar has no grading phase, or did not confirm the switch. + """ + if not handle.plan.grading_phase: + raise EgressSetupError("network: llm_only has no grading phase to switch to.") + try: + result = await _docker(*build_grading_argv(handle.sidecar)) + except (OSError, subprocess.SubprocessError) as exc: + raise EgressSetupError(f"network: llm_only could not switch to the grading allowlist: {exc}") from exc + if result.returncode != 0: + detail = (result.stdout or result.stderr).strip() or f"exit code {result.returncode}" + raise EgressSetupError(f"network: llm_only could not switch to the grading allowlist: {detail}") + + +async def read_proxy_log(handle: EgressHandle) -> str | None: + """The sidecar's log so far, from Docker: the run directory's copy is one the container could write.""" + result = await _best_effort("logs", handle.sidecar) + return result.stdout if result is not None and result.returncode == 0 else None + + +def grading_denials(proxy_log: str, *, whole_log: bool) -> list[str]: + """The ``host:port`` targets the proxy refused once grading started, less ``GRADING_DENIAL_NOISE``. + + ``whole_log`` is for a container that only grades; otherwise grading starts at the + ``PHASE grading`` line, and a log without one has no grading part. + """ + lines = proxy_log.splitlines() + if not whole_log: + start = next((i for i, line in enumerate(lines) if line.startswith("PHASE grading")), None) + if start is None: + return [] + lines = lines[start + 1 :] + denied = {line.split(" ")[1] for line in lines if line.startswith("DENY ") and line.count(" ") >= 2} + return sorted(denied - GRADING_DENIAL_NOISE) + + +async def _best_effort(*args: str, during: str = "teardown") -> subprocess.CompletedProcess[str] | None: + try: + return await _docker(*args) + except (OSError, subprocess.SubprocessError) as exc: + logger.warning("docker %s failed during egress %s: %s", " ".join(args[:2]), during, exc) + return None + + +async def _reclaim_leaked() -> None: + """Remove the stopped sidecars and unused networks that a killed coder-eval left behind. + + Prune never removes a running container or a network a container still uses, so a + concurrent run keeps its own; a daemon busy with another prune just skips this one. + """ + for kind in ("container", "network"): + await _best_effort( + kind, + "prune", + "--force", + "--filter", + f"label={EGRESS_LABEL}", + "--filter", + f"until={RECLAIM_MIN_AGE}", + during="reclaim", + ) + + +async def _teardown(*, network: str | None, sidecar: str | None, log_path: Path) -> None: + if sidecar is not None: + logs = await _best_effort("logs", sidecar) + if logs is not None and logs.returncode == 0: + try: + await asyncio.to_thread(write_text_atomic, log_path, logs.stdout + logs.stderr) + except OSError as exc: + logger.warning("Could not write the egress proxy log to %s: %s", log_path, exc) + await _best_effort("rm", "-f", sidecar) + if network is None: + return + for attempt in range(1, _NETWORK_RM_ATTEMPTS + 1): + removed = await _best_effort("network", "rm", network) + if removed is not None and (removed.returncode == 0 or "not found" in removed.stderr.lower()): + return + if attempt < _NETWORK_RM_ATTEMPTS: + await asyncio.sleep(_NETWORK_RM_RETRY_SECONDS) + logger.warning("Could not remove egress network %s; remove leaked egress resources with: %s", network, PRUNE_HINT) + + +async def _run_to_completion[T](awaitable: Awaitable[T]) -> T: + """Await ``awaitable`` to its end across any number of cancels of the caller, then re-raise the cancel.""" + task = asyncio.ensure_future(awaitable) + cancelled = False + while not task.done(): + try: + await asyncio.shield(task) + except asyncio.CancelledError: + cancelled = True + if cancelled: + if not task.cancelled() and task.exception() is not None: + logger.warning("A docker step failed while its caller was cancelled: %s", task.exception()) + raise asyncio.CancelledError + return task.result() + + +@contextlib.asynccontextmanager +async def egress_scope( + *, + container_name: str, + image: str, + egress_dir: Path, + heartbeat: Path, + stale_seconds: float, + plan: EgressPlan, + log_path: Path, +) -> AsyncIterator[EgressHandle]: + """Create the internal network and proxy sidecar, probe every target, yield, then tear both down. + + ``egress_dir`` holds the proxy module and ``heartbeat`` must already exist on the + host. Setup first reclaims what a killed run leaked. Teardown runs on every exit path + and writes the sidecar log to ``log_path`` (atomically, so a symlink planted at that + path is replaced, never followed). + + Rationale: .claude/notes/isolation.md § Why the sidecar watches the heartbeat + + Raises: + EgressSetupError: Any setup step (image, network, sidecar, probe) failed. + """ + network = f"{container_name}-net" + sidecar = f"{container_name}-egress" + created_network: str | None = None + created_sidecar: str | None = None + try: + await _checked( + f"find the framework image {image} for its egress sidecar (run `make docker-image`)", + "image", + "inspect", + "--format", + "{{.Id}}", + image, + ) + await _reclaim_leaked() + created_network = network + await _checked("create its per-task network", *build_network_create_argv(network)) + created_sidecar = sidecar + await _checked( + "create the egress sidecar", + *build_sidecar_create_argv( + sidecar=sidecar, + network=network, + image=image, + egress_dir=egress_dir, + heartbeat=heartbeat, + stale_seconds=stale_seconds, + plan=plan, + ), + ) + await _checked("attach the egress sidecar to the bridge network", "network", "connect", "bridge", sidecar) + await _checked("start the egress sidecar", "start", sidecar) + if plan.targets: + await _probe(sidecar, plan.targets) + elif not plan.switches_for_grading: + logger.warning("network: llm_only with an empty allowlist: the container can reach no host at all.") + if plan.grading_targets: + await _probe(sidecar, plan.grading_targets, direct=True) + logger.info("Egress sidecar %s on %s allows: %s", sidecar, network, ", ".join(plan.targets) or "(nothing)") + if plan.switches_for_grading: + grading = "any host" if plan.grading_any else ", ".join(plan.grading_targets) + logger.info("Egress sidecar %s also allows, once grading starts: %s", sidecar, grading) + yield EgressHandle(network=network, sidecar=sidecar, plan=plan) + finally: + await _run_to_completion(_teardown(network=created_network, sidecar=created_sidecar, log_path=log_path)) diff --git a/src/coder_eval/isolation/errors.py b/src/coder_eval/isolation/errors.py new file mode 100644 index 000000000..1bebf0766 --- /dev/null +++ b/src/coder_eval/isolation/errors.py @@ -0,0 +1,20 @@ +"""Errors raised by the docker isolation layer.""" + + +class DockerRunError(RuntimeError): + """Raised when ``docker run`` exits non-zero AND no task.json was produced. + + Criterion failures do NOT raise this -- the container always writes + task.json (with whatever results it has) before exiting, and the host + parses that regardless of exit code. This is reserved for setup-time + failures: missing image, daemon down, OOM-kill before the agent started, + etc. + """ + + +class EgressSetupError(DockerRunError): + """Raised when the ``network: llm_only`` egress sidecar cannot be set up. + + The task container never started, so there is no task.json; the runner + writes a synthetic ERROR record before re-raising. + """ diff --git a/src/coder_eval/models/__init__.py b/src/coder_eval/models/__init__.py index e665486bd..3caa44700 100644 --- a/src/coder_eval/models/__init__.py +++ b/src/coder_eval/models/__init__.py @@ -183,6 +183,7 @@ # Sandbox from coder_eval.models.sandbox import ( + LOOPBACK_HOSTS, RECORD_CLI_DIR, RECORD_CLI_LOG, RECORD_CLI_LOG_NAME, @@ -195,6 +196,8 @@ RecordedCli, ResourceLimits, SandboxConfig, + normalize_egress_target, + url_egress_target, validate_template_sources_list, ) from coder_eval.models.system_one import ( @@ -344,6 +347,9 @@ "RECORD_CLI_LOG_NAME", "SIDECAR_MODULES", "ResourceLimits", + "LOOPBACK_HOSTS", + "normalize_egress_target", + "url_egress_target", "validate_template_sources_list", # Telemetry "AssistantMessage", diff --git a/src/coder_eval/models/agent_config.py b/src/coder_eval/models/agent_config.py index e65f627a4..2ff68cb7d 100644 --- a/src/coder_eval/models/agent_config.py +++ b/src/coder_eval/models/agent_config.py @@ -3,6 +3,7 @@ from __future__ import annotations import dataclasses +from collections.abc import Mapping from typing import Annotated, Any, ClassVar, Literal, Self, TypedDict from claude_agent_sdk import ClaudeAgentOptions @@ -19,6 +20,7 @@ from coder_eval.models.enums import AgentKind, PermissionMode from coder_eval.models.merge_strategy import MergeField +from coder_eval.models.sandbox import url_egress_target type SettingSource = Literal["user", "project", "local"] @@ -217,6 +219,18 @@ def check_prompt_exclusivity(self) -> Self: raise ValueError("Only one of 'system_prompt' or 'system_prompt_file' can be provided, not both") return self + uses_api_backend: ClassVar[bool] = False + """True when the agent reaches its model through ``API_BACKEND``; ``llm_only`` then allows the backend's hosts.""" + + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """Normalized ``host:port`` targets this agent's own model API needs under ``network: llm_only``. + + The API backend's hosts are added only when ``uses_api_backend`` is true. + ``env`` holds the host variables forwarded into the container. Pure: no I/O. + Override on a plugin agent's config class. + """ + return () + @model_validator(mode="after") def check_context_window_supported(self) -> Self: """Reject a ``context_window`` the resolved agent type cannot apply. @@ -249,6 +263,7 @@ class ClaudeCodeAgentConfig(BaseAgentConfig): """Claude Code agent configuration.""" type: Literal[AgentKind.CLAUDE_CODE] # type: ignore[assignment] + uses_api_backend: ClassVar[bool] = True # The CLI's own accepted auto-compact window range; it silently drops a # settings value outside it, so the bound is enforced here instead. @@ -362,6 +377,20 @@ class CodexAgentConfig(BaseAgentConfig): # `yes` that loads as 1. _context_window_range: ClassVar[tuple[int, int | None] | None] = (32_768, None) + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """The ``CODEX_BASE_URL`` host when it is forwarded, else ``api.openai.com``. + + ``CodexAgent._resolve_base_url`` is the authority for that variable. + + Raises: + ValueError: ``CODEX_BASE_URL`` names a host or port that cannot be allowlisted. + """ + base_url = env.get("CODEX_BASE_URL") + if not base_url: + return ("api.openai.com:443",) + target = url_egress_target("CODEX_BASE_URL", base_url) + return (target,) if target is not None else () + # Mirrors google.antigravity.types.ThinkingLevel as a plain Literal so this module # imports without the optional SDK -- base installs must load every config class. @@ -387,6 +416,10 @@ class AntigravityAgentConfig(BaseAgentConfig): ), ) + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """The Gemini API host, the only one the local harness contacts.""" + return ("generativelanguage.googleapis.com:443",) + class OpenCodeAgentConfig(BaseAgentConfig): """OpenCode agent configuration (the open-source terminal coding agent). @@ -436,6 +469,13 @@ class OpenCodeAgentConfig(BaseAgentConfig): # forbid valid Pi levels. Confirmed against ``pi --help`` on Pi 0.87.1. type PiThinkingLevel = Literal["off", "minimal", "low", "medium", "high", "xhigh", "max"] +_PI_PROVIDER_HOSTS: dict[str, str] = { + "openrouter": "openrouter.ai:443", + "anthropic": "api.anthropic.com:443", + "openai": "api.openai.com:443", + "google": "generativelanguage.googleapis.com:443", +} + class PiAgentConfig(BaseAgentConfig): """Pi agent configuration (the ``pi`` Node coding agent — https://pi.dev/). @@ -463,6 +503,14 @@ class PiAgentConfig(BaseAgentConfig): description="Pi reasoning effort passed as --thinking (off/minimal/low/medium/high/xhigh/max).", ) + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """The API host of the ``provider/`` prefix of ``model``; OpenRouter when unknown or unset. + + Any other provider needs its host in ``sandbox.docker.egress_allowlist``. + """ + provider, _, model_id = (self.model or "").partition("/") + return (_PI_PROVIDER_HOSTS.get(provider if model_id else "", "openrouter.ai:443"),) + _DELEGATE_SDK_OPTION_FIELDS: frozenset[str] = frozenset({"effort"}) diff --git a/src/coder_eval/models/sandbox.py b/src/coder_eval/models/sandbox.py index d544cd51e..e68965fd5 100644 --- a/src/coder_eval/models/sandbox.py +++ b/src/coder_eval/models/sandbox.py @@ -2,8 +2,11 @@ from __future__ import annotations +import logging +import re import warnings from typing import Literal +from urllib.parse import urlsplit from pydantic import AliasChoices, BaseModel, ConfigDict, Field, field_validator, model_validator @@ -101,6 +104,61 @@ def validate_template_sources_list(sources: list[TemplateSource]) -> None: ) +_EGRESS_HOST = re.compile(r"[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*") +_EGRESS_PORT = re.compile(r"[0-9]{1,5}") + + +def normalize_egress_target(entry: str) -> str: + """Normalize one ``host`` or ``host:port`` egress entry to ``host:port``. + + The host is a DNS name or an IPv4 literal, lowercased; a bare host means port 443. + + Raises: + ValueError: The entry carries a scheme, path, wildcard, IPv6 literal or an + invalid port, or is empty. + """ + raw = entry.strip() + text = raw.lower() + host, sep, port_text = text.rpartition(":") + if not sep: + host, port_text = text, "443" + valid = raw.isascii() and _EGRESS_HOST.fullmatch(host) and _EGRESS_PORT.fullmatch(port_text) + if not valid or not 1 <= int(port_text) <= 65535: + raise ValueError( + f"egress_allowlist entry {entry!r} must be 'host' or 'host:port' (a DNS name or IPv4 address, " + + "port 1-65535), with no scheme, path, wildcard or IPv6 literal." + ) + return f"{host}:{int(port_text)}" + + +LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"}) + +logger = logging.getLogger(__name__) + + +def url_egress_target(source: str, url: str) -> str | None: + """Normalized ``host:port`` of an ``http(s)`` URL, or None when it has no host or another scheme. + + A loopback host is also None, with a warning: the egress sidecar cannot reach it. + ``source`` names the URL in the warning and the error; the URL itself is never + echoed, because it may carry credentials. + + Raises: + ValueError: The URL's host or port cannot be allowlisted. + """ + try: + parts = urlsplit(url) + if parts.scheme not in ("http", "https") or not parts.hostname: + return None + if parts.hostname in LOOPBACK_HOSTS: + logger.warning("%s points at a loopback host, which the egress sidecar cannot reach.", source) + return None + port = parts.port if parts.port is not None else (443 if parts.scheme == "https" else 80) + return normalize_egress_target(f"{parts.hostname}:{port}") + except ValueError: + raise ValueError(f"{source} has a host or port that network: llm_only cannot allowlist.") from None + + class DockerBuildConfig(BaseModel): """``docker build`` customization for a ``dockerfile_path`` task image. @@ -192,9 +250,42 @@ class DockerDriverConfig(BaseModel): "`dockerfile_path` is set. Ignored when building is not in play." ), ) - network: Literal["bridge", "none"] = Field( + network: Literal["bridge", "none", "llm_only"] = Field( default="bridge", - description="Container network. 'bridge' for tasks needing LLM/pkg access; 'none' for fully sealed runs.", + description=( + "Container network. 'bridge' (default): full network. 'llm_only': the container reaches only the " + "model APIs and `egress_allowlist` through a proxy sidecar (see docs/DOCKER_ISOLATION.md § Network " + "modes). 'none': no network at all, so only `agent: {type: none}` tasks can run." + ), + ) + egress_allowlist: list[str] = MergeField( + strategy="append", + default_factory=list, + description=( + "Extra `host` or `host:port` targets (default port 443) the container may reach under " + "`network: llm_only`, beyond the derived model-API hosts. Appended across config layers. " + "Ignored for other network modes. Example: ['pypi.org', 'files.pythonhosted.org']." + ), + ) + grading_network: Literal["bridge", "llm_only"] | None = Field( + default=None, + description=( + "Network for grading, when it differs from the agent's. Needs `network: llm_only`. 'bridge': the " + "criteria reach any host through the proxy sidecar. 'llm_only': the agent's allowlist plus " + "`grading_egress_allowlist`. None (default): grading uses the agent's network. The proxy switches " + "once, when grading starts, so a criterion can install its test dependencies while the agent never " + "reached those hosts. Not supported with an enabled `simulation`, whose criteria run between turns." + ), + ) + grading_egress_allowlist: list[str] = MergeField( + strategy="append", + default_factory=list, + description=( + "Extra `host` or `host:port` targets (default port 443) the criteria may reach under " + "`network: llm_only`, on top of `egress_allowlist` and the derived model-API hosts. The agent " + "cannot reach them. Appended across config layers. Ignored with `grading_network: bridge` or " + "another network mode. Example: ['pypi.org', 'files.pythonhosted.org', 'astral.sh']." + ), ) working_dir: str | None = Field( default=None, @@ -283,6 +374,27 @@ class DockerDriverConfig(BaseModel): description="Extra `-v src:dst[:ro]` mount specs forwarded to `docker run`. Validated for basic syntax.", ) + @field_validator("egress_allowlist", "grading_egress_allowlist") + @classmethod + def _normalize_egress_allowlist(cls, values: list[str]) -> list[str]: + return [normalize_egress_target(v) for v in values] + + @model_validator(mode="after") + def _check_grading_network(self) -> DockerDriverConfig: + if self.grading_network is None or self.network == "llm_only": + return self + if self.network == "bridge" and self.grading_network == "bridge": + return self + raise ValueError( + f"grading_network: {self.grading_network} needs network: llm_only, not {self.network}: the container " + + "joins one network for its whole life, so grading can only widen an llm_only allowlist." + ) + + @property + def grading_widens_egress(self) -> bool: + """True when grading may reach hosts the agent could not, so the proxy switches when grading starts.""" + return self.network == "llm_only" and (self.grading_network == "bridge" or bool(self.grading_egress_allowlist)) + @field_validator("working_dir") @classmethod def _validate_working_dir(cls, v: str | None) -> str | None: diff --git a/src/coder_eval/models/tasks.py b/src/coder_eval/models/tasks.py index 90947acc2..3fc3b8477 100644 --- a/src/coder_eval/models/tasks.py +++ b/src/coder_eval/models/tasks.py @@ -687,6 +687,17 @@ def check_none_agent(self) -> Self: ) return self + @model_validator(mode="after") + def check_grading_network_without_simulation(self) -> Self: + """Refuse a separate grading network on a simulated task: its criteria run between agent turns.""" + if self.simulation is not None and self.simulation.enabled and self.sandbox.docker.grading_widens_egress: + raise ValueError( + "sandbox.docker.grading_network / grading_egress_allowlist cannot widen the network of a task " + + "with an enabled simulation: its criteria run between dialog turns, so the agent would keep " + + "the grading network for the rest of the dialog." + ) + return self + @model_validator(mode="after") def check_reference_consumers_have_a_reference(self) -> Self: """Reject criteria that DEMAND the reference when no ``reference:`` block is set. diff --git a/src/coder_eval/orchestration/batch.py b/src/coder_eval/orchestration/batch.py index 2087d0365..f8b71fe05 100644 --- a/src/coder_eval/orchestration/batch.py +++ b/src/coder_eval/orchestration/batch.py @@ -193,6 +193,12 @@ async def run_single(rt: ResolvedTask) -> TaskResult: # HERE, where the original driver is still visible: the # in-container orchestrator sees it forced to tempdir. driver = sandbox_cfg.driver if sandbox_cfg is not None else "tempdir" + if sandbox_cfg is not None and driver != "docker" and sandbox_cfg.docker.network == "llm_only": + raise ValueError( + f"sandbox.docker.network: llm_only needs sandbox.driver: docker, not {driver!r}; " + + "without the container the agent would get the full host network. " + + "Set driver: docker or pass --driver docker." + ) preservation_mode = resolve_preservation_mode(config.preservation_mode, driver) # DIRECT_WRITE on a non-docker host re-opens the parent-dir # node_modules contamination MOVE_ON_WRITE exists to prevent diff --git a/src/coder_eval/orchestration/regrade.py b/src/coder_eval/orchestration/regrade.py index e301d4954..f0212760d 100644 --- a/src/coder_eval/orchestration/regrade.py +++ b/src/coder_eval/orchestration/regrade.py @@ -37,7 +37,9 @@ ) from coder_eval.path_utils import ( DOCKER_LOG_FILENAME, + EGRESS_LOG_FILENAME, GRADE_DOCKER_LOG_FILENAME, + GRADE_EGRESS_LOG_FILENAME, GRADE_LOG_FILENAME, PRE_GRADE_JSON_FILENAME, TASK_JSON_FILENAME, @@ -284,6 +286,11 @@ def _container_dispatch_commands(task: TaskDefinition, task_file: Path | None) - parts += [f"-v {mount}" for mount in docker.extra_mounts or []] parts += [f"-v {path}" for path in _dispatch_host_exposure(task, task_file)] parts += [f"--env {name}" for name in docker.env_passthrough_extra or []] + if docker.network == "llm_only" and docker.grading_network != "bridge": + parts.append("--network llm_only") + allowed = list(dict.fromkeys([*docker.egress_allowlist, *docker.grading_egress_allowlist])) + if allowed: + parts.append(f"(egress allowed to: {', '.join(allowed)} plus the derived model-API hosts)") parts.append("(with your credentials in its environment and a writable copy of ~/.claude)") return [" ".join(parts)] @@ -688,6 +695,7 @@ def _fold_back_container_logs(container_run_dir: Path, run_dir: Path) -> None: unhonored = f"{TASK_JSON_FILENAME}.unhonored" rescued = ( (DOCKER_LOG_FILENAME, GRADE_DOCKER_LOG_FILENAME), + (EGRESS_LOG_FILENAME, GRADE_EGRESS_LOG_FILENAME), (GRADE_LOG_FILENAME, GRADE_LOG_FILENAME), (unhonored, unhonored), ) diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index 4567ae84c..db264a7ad 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -6,7 +6,7 @@ import tempfile import time import uuid -from collections.abc import Callable, Sequence +from collections.abc import Awaitable, Callable, Sequence from contextlib import suppress from dataclasses import dataclass from datetime import datetime @@ -352,6 +352,10 @@ def __init__( self.artifacts_dir = artifacts_dir self.task_file = task_file self.stream_callback = stream_callback + # Awaited once, before the first criterion of a graded run; the in-container run + # sets it to widen the egress allowlist for grading. + self.before_grading: Callable[[], Awaitable[None]] | None = None + self._grading_started = False self.sandbox = sandbox self.variant_id = variant_id self.source_yaml = source_yaml @@ -856,6 +860,7 @@ async def _evaluate_post_failure_criteria(self) -> None: checked: CriteriaResults = [] if runnable: + await self._start_grading() checked = await self.success_checker.check_all_async( runnable, reference_dir=self._reference_dir, @@ -876,6 +881,11 @@ async def _evaluate_post_failure_criteria(self) -> None: recovered.append(next(checked_iter)) self.result.post_failure_criteria_results = recovered + async def _start_grading(self) -> None: + if self.before_grading is not None and not self._grading_started: + self._grading_started = True + await self.before_grading() + async def _drain_killed_turn(self) -> None: """Move a hard-killed turn's partial record from the agent onto the result. @@ -2208,6 +2218,7 @@ async def _evaluation_loop(self) -> bool: # Check success criteria (reference_dir feeds reference_comparison + judges) logger.debug("Checking success criteria") + await self._start_grading() await self._verify_reference_integrity() criteria_results = await self.success_checker.check_all_async( self.task.success_criteria, diff --git a/src/coder_eval/path_utils.py b/src/coder_eval/path_utils.py index 103e3b825..7c6c9ba22 100644 --- a/src/coder_eval/path_utils.py +++ b/src/coder_eval/path_utils.py @@ -39,6 +39,15 @@ # Rationale: .claude/notes/persistence.md § Run-directory filename constants DOCKER_LOG_FILENAME = "docker.log" GRADE_DOCKER_LOG_FILENAME = "grade.docker.log" +# The egress sidecar's ALLOW/DENY log under ``network: llm_only``, kept apart from +# ``docker.log`` so container stdout cannot forge its lines; and its grading fold-back name. +EGRESS_LOG_FILENAME = "egress.log" +GRADE_EGRESS_LOG_FILENAME = "grade.egress.log" +# The in-container run asks for the grading allowlist through the first file; the host +# answers in the second with EGRESS_GRADING_ACK_OK or "error: ". +EGRESS_GRADING_REQUEST_FILENAME = ".coder_eval_egress_grading_request" +EGRESS_GRADING_ACK_FILENAME = ".coder_eval_egress_grading_ack" +EGRESS_GRADING_ACK_OK = "ok" # The virtualenv directory `setup` creates and `adopt` discovers. Named because # whether it is on PATH decides which binaries a criterion resolves. diff --git a/tasks/README.md b/tasks/README.md index a36d7a0aa..77d102318 100644 --- a/tasks/README.md +++ b/tasks/README.md @@ -69,6 +69,7 @@ criterion, so `pr-checks.yml` preflights it. Run it on its own with `--tags syst |-----------|---------------| | `datasets/` | JSONL datasets referenced by dataset-backed tasks | | `dockerfile_build_example/` | A task that builds its own Dockerfile | +| `docker_egress_probe/` | Thorough `network: llm_only` egress check: grading probes for every path out, plus a `bridge` control; `docker_egress_grading.yaml` checks the grading network with no agent | | `mock_path_dirs_template_dir/` | Template dir (mock CLI bins) consumed by `mock_path_dirs_smoke` | | `python_cli_simulated_judged/` | Simulated multi-turn dialog + judged scoring | | `internal/` | Dev smoke tests exercising harness internals (e.g. session-resumption / context retention) | diff --git a/tasks/docker_egress_probe/docker_egress_grading.yaml b/tasks/docker_egress_probe/docker_egress_grading.yaml new file mode 100644 index 000000000..787e70cf2 --- /dev/null +++ b/tasks/docker_egress_probe/docker_egress_grading.yaml @@ -0,0 +1,27 @@ +task_id: docker_egress_grading +description: | + Check that grading_egress_allowlist opens its hosts to grading only. No agent and no model API: + pre_run runs before grading under the agent's allowlist and records whether pypi.org answered, + then the criteria run under the grading allowlist. Under network bridge the first and last + criteria fail, which is the control. +tags: [docker, network, smoke] +agent: + type: none +sandbox: + driver: docker + docker: + network: llm_only + grading_egress_allowlist: [pypi.org] +pre_run: + - command: "if curl -sS -m 10 -o /dev/null https://pypi.org/simple/; then echo REACHED; else echo BLOCKED; fi > agent_phase.txt" +success_criteria: + - type: file_contains + path: agent_phase.txt + includes: ["BLOCKED"] + description: "Before grading, pypi.org is blocked." + - type: run_command + command: "curl -sS -m 10 -o /dev/null https://pypi.org/simple/" + description: "During grading, pypi.org is reachable." + - type: run_command + command: "! curl -sS -m 10 -o /dev/null https://example.com" + description: "During grading, a host on no allowlist is still blocked." diff --git a/tasks/docker_egress_probe/docker_egress_probe.yaml b/tasks/docker_egress_probe/docker_egress_probe.yaml new file mode 100644 index 000000000..51836ef33 --- /dev/null +++ b/tasks/docker_egress_probe/docker_egress_probe.yaml @@ -0,0 +1,75 @@ +task_id: docker_egress_probe +description: | + Thorough egress check for network llm_only. The agent tries to reach the internet in its own + ways and reports what happened; the grading probes then test each path out of the container + (proxy, direct IPv4/IPv6, DNS, raw sockets, the docker host, package managers, git) and require + every one to be blocked while the model API stays reachable. Under network bridge the probes + fail, which is the control that shows they measure something. +tags: [docker, network, smoke] +agent: + type: claude-code + permission_mode: acceptEdits + allowed_tools: [Bash, Read, Write, WebFetch, WebSearch] + setting_sources: [] +sandbox: + driver: docker + docker: + network: llm_only + limits: + timeout: 600 +initial_prompt: | + You are in a sandbox that should have no general internet access. Test that, and report honestly. + Try each of these once, with a timeout of at most 15 seconds each, and do not retry: + 1. curl https://example.com + 2. curl --noproxy '*' https://example.com + 3. python3 -c "import urllib.request; print(urllib.request.urlopen('https://example.com', timeout=10).status)" + 4. pip download --no-deps --retries 0 --timeout 5 -d /tmp/p six + 5. git clone --depth 1 https://github.com/octocat/Hello-World /tmp/h + 6. getent hosts example.com + 7. Use your WebFetch tool on https://example.com + 8. Use your WebSearch tool to search for "example domain IANA" + Write report.md with one line per attempt: the attempt number, REACHED or BLOCKED, and the short + error or result. For attempt 7, also write webfetch.txt containing exactly the text WebFetch + returned, or the single word BLOCKED if it failed. Finally create notes.txt containing DONE. +success_criteria: + - type: file_contains + path: notes.txt + includes: ["DONE"] + description: "The agent completed its turn (the model API was reachable)." + weight: 1.0 + - type: file_exists + path: report.md + description: "The agent wrote its egress report." + weight: 0.5 + - type: run_command + command: "test -f webfetch.txt && ! grep -qi 'example domain' webfetch.txt" + description: "WebFetch did not return the example.com page." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" model_host_reachable model_host_other_port_blocked + description: "The model host is reachable through the proxy on its own port only." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_proxy_https blocked_proxy_http blocked_proxy_absolute_https blocked_urllib_through_env_proxy blocked_curl + description: "The proxy refuses a host that is not on the allowlist (CONNECT, plain HTTP, absolute-form HTTPS, urllib, curl)." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_direct_ipv4 blocked_direct_ipv6 blocked_default_route blocked_curl_noproxy + description: "No direct route out: IPv4, IPv6, no default route, curl --noproxy." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_external_dns blocked_udp_dns_to_public_resolver + description: "No DNS out: external names do not resolve and a public resolver does not answer." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_docker_host_alias blocked_gateway_ip blocked_sidecar_other_ports blocked_raw_socket + description: "No reach to the docker host, the gateway or other sidecar ports, and no raw sockets." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_pip_download blocked_git_clone blocked_npm_view + description: "Package managers and git cannot fetch from the internet." + weight: 1.0 +run_limits: + max_turns: 30 + task_timeout: 600 + turn_timeout: 420 diff --git a/tasks/docker_egress_probe/egress_probe.py b/tasks/docker_egress_probe/egress_probe.py new file mode 100644 index 000000000..d0d16033e --- /dev/null +++ b/tasks/docker_egress_probe/egress_probe.py @@ -0,0 +1,258 @@ +"""Egress probes for ``network: llm_only``: each check exits 0 when the container behaves as the mode promises. + +Run as ``python3 egress_probe.py `` from a ``run_command`` criterion, or with +``all`` for a table of every check. A ``blocked_*`` check passes when the path out is +closed; a ``model_*`` check passes when the model API stays reachable. +""" + +from __future__ import annotations + +import os +import shutil +import socket +import subprocess +import sys +import urllib.error +import urllib.request +from urllib.parse import urlsplit + + +TIMEOUT = 6.0 +OUTSIDE_HOST = "example.com" +OUTSIDE_IPV4 = "1.1.1.1" +OUTSIDE_IPV6 = "2606:4700:4700::1111" + + +def _proxy() -> tuple[str, int]: + parts = urlsplit(os.environ.get("HTTPS_PROXY", "")) + if not parts.hostname or parts.port is None: + raise RuntimeError("HTTPS_PROXY is not set: this is not an llm_only container") + return parts.hostname, parts.port + + +def _connect_status(target: str) -> str: + with socket.create_connection(_proxy(), TIMEOUT) as sock: + sock.sendall(f"CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n\r\n".encode()) + return sock.recv(256).split(b"\r\n", 1)[0].decode("latin-1") + + +def _raw_status(request: bytes) -> str: + with socket.create_connection(_proxy(), TIMEOUT) as sock: + sock.sendall(request) + return sock.recv(256).split(b"\r\n", 1)[0].decode("latin-1") + + +def _url_target(url: str) -> tuple[str, int] | None: + parts = urlsplit(url) + if not parts.hostname: + return None + return parts.hostname, parts.port or (443 if parts.scheme == "https" else 80) + + +def _model_candidates() -> list[tuple[str, int]]: + """Every ``(host, port)`` a built-in harness may call for its model, from the variables this container got.""" + backend = os.environ.get("API_BACKEND", "direct") + candidates: list[tuple[str, int] | None] = [] + if backend == "bedrock" and os.environ.get("AWS_REGION"): + candidates.append((f"bedrock-runtime.{os.environ['AWS_REGION'].lower()}.amazonaws.com", 443)) + if backend == "direct": + candidates.append(("api.anthropic.com", 443)) + if backend == "litellm": + candidates.append(_url_target(os.environ.get("LITELLM_BASE_URL", ""))) + if os.environ.get("CODEX_BASE_URL"): + candidates.append(_url_target(os.environ["CODEX_BASE_URL"])) + elif os.environ.get("CODEX_API_KEY"): + candidates.append(("api.openai.com", 443)) + if os.environ.get("GEMINI_API_KEY"): + candidates.append(("generativelanguage.googleapis.com", 443)) + if os.environ.get("OPENROUTER_API_KEY"): + candidates.append(("openrouter.ai", 443)) + return [c for c in candidates if c is not None] + + +def _model_target() -> tuple[str, int] | None: + """The first model host the proxy lets through, or None when it allows none of them.""" + for host, port in _model_candidates(): + if " 200 " in f"{_connect_status(f'{host}:{port}')} ": + return host, port + return None + + +def _tcp_refused(host: str, port: int, family: int = socket.AF_INET) -> str | None: + """None when no connection could be made, else a description of the open path.""" + try: + with socket.socket(family, socket.SOCK_STREAM) as sock: + sock.settimeout(TIMEOUT) + sock.connect((host, port)) + except OSError: + return None + return f"connected to {host}:{port}" + + +def _command_fails(argv: list[str], timeout: float = 60) -> str | None: + if shutil.which(argv[0]) is None: + return None + env = dict(os.environ, PIP_DISABLE_PIP_VERSION_CHECK="1", GIT_TERMINAL_PROMPT="0") + try: + done = subprocess.run(argv, capture_output=True, text=True, timeout=timeout, env=env, cwd="/tmp") + except subprocess.TimeoutExpired: + return None + return None if done.returncode != 0 else f"{' '.join(argv)} succeeded" + + +def blocked_proxy_https() -> str | None: + status = _connect_status(f"{OUTSIDE_HOST}:443") + return None if " 403 " in f"{status} " else f"proxy answered {status!r} for {OUTSIDE_HOST}:443" + + +def blocked_proxy_http() -> str | None: + request = f"GET http://{OUTSIDE_HOST}/ HTTP/1.1\r\nHost: {OUTSIDE_HOST}\r\n\r\n".encode() + status = _raw_status(request) + return None if " 403 " in f"{status} " else f"proxy answered {status!r} for http://{OUTSIDE_HOST}/" + + +def blocked_proxy_absolute_https() -> str | None: + request = f"GET https://{OUTSIDE_HOST}/ HTTP/1.1\r\nHost: {OUTSIDE_HOST}\r\n\r\n".encode() + status = _raw_status(request) + return None if status.split(" ")[1:2] in (["400"], ["403"]) else f"proxy answered {status!r}" + + +def blocked_urllib_through_env_proxy() -> str | None: + try: + urllib.request.urlopen(f"https://{OUTSIDE_HOST}/", timeout=TIMEOUT) + except (urllib.error.URLError, OSError): + return None + return f"urllib fetched https://{OUTSIDE_HOST}/" + + +def blocked_direct_ipv4() -> str | None: + return _tcp_refused(OUTSIDE_IPV4, 443) or _tcp_refused(OUTSIDE_IPV4, 80) + + +def blocked_direct_ipv6() -> str | None: + if not socket.has_ipv6: + return None + return _tcp_refused(OUTSIDE_IPV6, 443, socket.AF_INET6) + + +def blocked_default_route() -> str | None: + with open("/proc/net/route", encoding="ascii") as routes: + defaults = [line for line in routes.read().splitlines()[1:] if line.split()[1:2] == ["00000000"]] + return f"default route present: {defaults}" if defaults else None + + +def blocked_gateway_ip() -> str | None: + with open("/proc/net/route", encoding="ascii") as routes: + rows = [line.split() for line in routes.read().splitlines()[1:]] + for row in rows: + destination = socket.inet_ntoa(int(row[1], 16).to_bytes(4, "little")) + gateway = ".".join([*destination.split(".")[:3], "1"]) + for port in (22, 80, 443, 2375, 4000): + opened = _tcp_refused(gateway, port) + if opened: + return opened + return None + + +def blocked_external_dns() -> str | None: + try: + addresses = socket.getaddrinfo(OUTSIDE_HOST, 443) + except OSError: + return None + return f"{OUTSIDE_HOST} resolved to {sorted({a[4][0] for a in addresses})}" + + +def blocked_udp_dns_to_public_resolver() -> str | None: + query = b"\x12\x34\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x07example\x03com\x00\x00\x01\x00\x01" + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: + sock.settimeout(TIMEOUT) + try: + sock.sendto(query, ("8.8.8.8", 53)) + sock.recvfrom(512) + except OSError: + return None + return "8.8.8.8 answered a DNS query" + + +def blocked_docker_host_alias() -> str | None: + for name in ("host.docker.internal", "gateway.docker.internal"): + try: + socket.getaddrinfo(name, 80) + except OSError: + continue + return f"{name} resolves" + return None + + +def blocked_raw_socket() -> str | None: + try: + socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP).close() + except PermissionError: + return None + return "a raw ICMP socket was opened (NET_RAW is not dropped)" + + +def blocked_sidecar_other_ports() -> str | None: + host, proxy_port = _proxy() + for port in (22, 80, 443, 8080): + if port != proxy_port and (opened := _tcp_refused(host, port)): + return opened + return None + + +def blocked_curl_noproxy() -> str | None: + return _command_fails(["curl", "-sS", "-m", "8", "--noproxy", "*", "-o", "/dev/null", f"https://{OUTSIDE_HOST}"]) + + +def blocked_curl() -> str | None: + return _command_fails(["curl", "-sS", "-m", "8", "-o", "/dev/null", f"https://{OUTSIDE_HOST}"]) + + +def blocked_pip_download() -> str | None: + argv = ["pip", "download", "--no-deps", "--retries", "0", "--timeout", "5", "-d", "/tmp/pip-probe", "six"] + return _command_fails(argv) + + +def blocked_git_clone() -> str | None: + return _command_fails(["git", "clone", "--depth", "1", "https://github.com/octocat/Hello-World", "/tmp/git-probe"]) + + +def blocked_npm_view() -> str | None: + return _command_fails(["npm", "view", "left-pad", "version", "--fetch-retries=0", "--fetch-timeout=5000"]) + + +def model_host_reachable() -> str | None: + if _model_target() is not None: + return None + tried = ", ".join(f"{host}:{port}" for host, port in _model_candidates()) or "none known to this probe" + return f"the proxy allows none of the model hosts ({tried})" + + +def model_host_other_port_blocked() -> str | None: + target = _model_target() + if target is None: + return "the proxy allows no model host" + host, port = target + other = 8 if port != 8 else 9 + status = _connect_status(f"{host}:{other}") + return None if " 403 " in f"{status} " else f"proxy answered {status!r} for {host}:{other}" + + +CHECKS = {name: fn for name, fn in globals().items() if name.startswith(("blocked_", "model_")) and callable(fn)} + + +def main(argv: list[str]) -> int: + names = list(CHECKS) if argv[1:] == ["all"] else argv[1:] + failed = 0 + for name in names: + try: + problem = CHECKS[name]() + except Exception as exc: + problem = f"probe error: {exc!r}" + print(f"{'PASS' if problem is None else 'FAIL'} {name}{'' if problem is None else ': ' + problem}") + failed += problem is not None + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/tests/test_docker_egress_config.py b/tests/test_docker_egress_config.py new file mode 100644 index 000000000..60fc0eb5c --- /dev/null +++ b/tests/test_docker_egress_config.py @@ -0,0 +1,114 @@ +"""`network: llm_only` and `egress_allowlist` on DockerDriverConfig: values, normalization, rejection.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from coder_eval.models import DockerDriverConfig, normalize_egress_target + + +def test_network_modes_and_allowlist_defaults(): + assert DockerDriverConfig().network == "bridge" + assert DockerDriverConfig().egress_allowlist == [] + assert DockerDriverConfig(network="llm_only").network == "llm_only" + with pytest.raises(ValidationError): + DockerDriverConfig(network="internal") # type: ignore[arg-type] + + +@pytest.mark.parametrize( + ("entry", "expected"), + [ + (" API.Anthropic.com ", "api.anthropic.com:443"), + ("pypi.org:443", "pypi.org:443"), + ("host.docker.internal:4000", "host.docker.internal:4000"), + ("10.0.0.5:8080", "10.0.0.5:8080"), + ("example.com:080", "example.com:80"), + ], +) +def test_entries_normalize_to_host_port(entry: str, expected: str): + assert normalize_egress_target(entry) == expected + assert DockerDriverConfig(egress_allowlist=[entry]).egress_allowlist == [expected] + + +@pytest.mark.parametrize( + "entry", + [ + "https://pypi.org", + "pypi.org/simple", + "*.googleapis.com", + "pypi.org:0", + "pypi.org:70000", + "pypi.org:", + "", + "[::1]:443", + "-bad.example.com", + "user@pypi.org", + "evil.com\n:443", + "\u212aube.io", + ], +) +def test_invalid_entries_raise_naming_the_entry(entry: str): + with pytest.raises(ValidationError, match="egress_allowlist entry"): + DockerDriverConfig(egress_allowlist=[entry]) + + +async def test_llm_only_without_the_docker_driver_is_refused_before_any_agent_runs(tmp_path): + from unittest.mock import patch + + from coder_eval.models import FinalStatus, ResolvedTask, TaskDefinition + from coder_eval.orchestration.batch import run_batch + from coder_eval.orchestration.config import BatchRunConfig + from coder_eval.orchestrator import Orchestrator + + task = TaskDefinition( + task_id="t", + description="d", + initial_prompt="p", + agent={"type": "claude-code"}, + sandbox={"driver": "tempdir", "docker": {"network": "llm_only"}}, + success_criteria=[{"type": "file_exists", "path": "x.txt", "description": "x"}], + ) + run_dir = tmp_path / "run" + rt = ResolvedTask(task=task, task_file=tmp_path / "t.yaml", run_dir=run_dir / "default" / "t", variant_id="default") + with patch.object(Orchestrator, "__init__", side_effect=AssertionError("an agent must not run")) as init: + _summary, results = await run_batch([rt], BatchRunConfig(run_dir=run_dir, max_parallel=1)) + init.assert_not_called() + [result] = results + assert result.result.final_status == FinalStatus.ERROR + assert "llm_only needs sandbox.driver: docker" in (result.result.error_message or "") + + +def test_grading_allowlist_normalizes_like_the_agent_allowlist(): + cfg = DockerDriverConfig(network="llm_only", grading_egress_allowlist=["PyPI.org", "astral.sh:443"]) + assert cfg.grading_egress_allowlist == ["pypi.org:443", "astral.sh:443"] + with pytest.raises(ValidationError): + DockerDriverConfig(network="llm_only", grading_egress_allowlist=["https://pypi.org"]) + + +@pytest.mark.parametrize( + ("docker", "widens"), + [ + ({"network": "llm_only"}, False), + ({"network": "llm_only", "grading_network": "llm_only"}, False), + ({"network": "llm_only", "grading_network": "bridge"}, True), + ({"network": "llm_only", "grading_egress_allowlist": ["pypi.org"]}, True), + ({"network": "bridge", "grading_network": "bridge"}, False), + ({"network": "bridge", "grading_egress_allowlist": ["pypi.org"]}, False), + ], +) +def test_grading_widens_egress_only_from_llm_only(docker: dict[str, object], widens: bool): + assert DockerDriverConfig(**docker).grading_widens_egress is widens # type: ignore[arg-type] + + +@pytest.mark.parametrize( + "docker", + [ + {"network": "bridge", "grading_network": "llm_only"}, + {"network": "none", "grading_network": "bridge"}, + {"network": "none", "grading_network": "llm_only"}, + ], +) +def test_grading_network_cannot_narrow_or_leave_none(docker: dict[str, object]): + with pytest.raises(ValidationError, match="needs network: llm_only"): + DockerDriverConfig(**docker) # type: ignore[arg-type] diff --git a/tests/test_docker_egress_runner.py b/tests/test_docker_egress_runner.py new file mode 100644 index 000000000..a3fc8d458 --- /dev/null +++ b/tests/test_docker_egress_runner.py @@ -0,0 +1,675 @@ +"""``network: llm_only`` in the docker runner: argv, sidecar lifecycle and teardown (no daemon).""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +import os +import subprocess +import sys +import tempfile +import threading +from datetime import UTC, datetime +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock + +import pytest + +from coder_eval.isolation import docker_runner as dr +from coder_eval.isolation import egress as egress_mod +from coder_eval.isolation.docker_runner import CONTAINER_ENTRYPOINT, DockerRunError, DockerRunner +from coder_eval.isolation.egress import ( + EGRESS_LABEL, + NO_PROXY_HOSTS, + PROXY_URL, + PRUNE_HINT, + EgressHandle, + EgressPlan, + build_network_create_argv, + build_sidecar_create_argv, + egress_scope, + grading_denials, +) +from coder_eval.isolation.errors import EgressSetupError +from coder_eval.models import ( + CONTAINER_INPUT_DIR, + CONTAINER_OUTPUT_DIR, + IN_CONTAINER_ENV, + ApiBackend, + DockerDriverConfig, + EvaluationResult, + FileExistsCriterion, + FinalStatus, + SandboxConfig, + TaskDefinition, + parse_agent_config, +) +from coder_eval.orchestration.regrade import _container_dispatch_commands +from coder_eval.path_utils import TASK_JSON_FILENAME + + +pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="docker driver is POSIX-only") + +SETUP = [ + "image inspect", + "container prune", + "network prune", + "network create", + "create --name", + "network connect", + "start c-egress", + "exec c-egress", +] +TEARDOWN = ["logs c-egress", "rm -f", "network rm"] + + +def _task(**docker: object) -> TaskDefinition: + return TaskDefinition( + task_id="t", + description="d", + initial_prompt="p", + agent=parse_agent_config(type="claude-code"), + sandbox=SandboxConfig(driver="docker", docker=DockerDriverConfig(**docker)), # type: ignore[arg-type] + success_criteria=[FileExistsCriterion(description="c", path="x.txt")], + ) + + +def _runner(**docker: object) -> DockerRunner: + rt = MagicMock() + rt.task = _task(**docker) + rt.run_dir = Path(tempfile.gettempdir()) / "test_run_egress" + rt.task_file = None + return DockerRunner(rt) + + +@pytest.fixture +def hermetic_env(monkeypatch: pytest.MonkeyPatch) -> None: + for name in [*DockerDriverConfig().env_passthrough, *(n for n in os.environ if n.startswith("LITELLM_"))]: + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test") + monkeypatch.setenv("CODER_EVAL_NO_CLAUDE_MOUNT", "1") + + +@pytest.fixture +def dirs(tmp_path: Path) -> tuple[Path, Path]: + input_dir, output_dir = tmp_path / "in", tmp_path / "out" + input_dir.mkdir() + output_dir.mkdir() + return input_dir, output_dir + + +@pytest.mark.parametrize("network", ["bridge", "none"]) +def test_bridge_and_none_argv_is_unchanged(hermetic_env: None, dirs: tuple[Path, Path], network: str) -> None: + input_dir, output_dir = dirs + runner = _runner(network=network, env_passthrough=["ANTHROPIC_API_KEY"], image="img:1") + argv = runner._build_argv(input_dir, output_dir, container_name="c") + assert argv == [ + *["docker", "run", "--rm", "--name", "c", "--entrypoint", CONTAINER_ENTRYPOINT], + *["--cap-drop", "DAC_OVERRIDE", "--cap-drop", "DAC_READ_SEARCH", "--network", network], + *["--env", "ANTHROPIC_API_KEY", "--env", f"{IN_CONTAINER_ENV}=1", "--env", "TELEMETRY_ENABLED=false"], + *["-v", f"{input_dir.resolve()}:{CONTAINER_INPUT_DIR}", "-v", f"{output_dir}:{CONTAINER_OUTPUT_DIR}"], + *["img:1", "--output", CONTAINER_OUTPUT_DIR], + ] + + +def test_llm_only_joins_only_the_internal_network_with_explicit_proxy_env( + hermetic_env: None, dirs: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv("HTTPS_PROXY", "http://corp-proxy:8080") + monkeypatch.setenv("LITELLM_BASE_URL", "http://localhost:4000") + runner = _runner( + network="llm_only", + env_passthrough=["ANTHROPIC_API_KEY", "LITELLM_BASE_URL"], + env_passthrough_extra=["HTTPS_PROXY"], + ) + handle = EgressHandle(network="c-net", sidecar="c-egress", plan=EgressPlan(("api.anthropic.com:443",))) + argv = runner._build_argv(*dirs, container_name="c", egress=handle) + assert argv.count("--network") == 1 + assert argv[argv.index("--network") + 1] == "c-net" + assert argv[argv.index("--dns") + 1] == "192.0.2.1" + assert "--add-host" not in argv + env = [argv[i + 1] for i, token in enumerate(argv) if token == "--env"] + proxy_env = [f"{name}={PROXY_URL}" for name in ("HTTPS_PROXY", "HTTP_PROXY", "https_proxy", "http_proxy")] + assert set(proxy_env) | {f"NO_PROXY={NO_PROXY_HOSTS}", f"no_proxy={NO_PROXY_HOSTS}"} <= set(env) + assert "LITELLM_BASE_URL=http://host.docker.internal:4000" in env + assert "HTTPS_PROXY" not in env, "a host proxy must not be forwarded name-only" + + +def test_llm_only_without_a_handle_refuses_to_fall_back_to_bridge(hermetic_env: None, dirs: tuple[Path, Path]) -> None: + with pytest.raises(DockerRunError, match="refusing to fall back to bridge"): + _runner(network="llm_only")._build_argv(*dirs, container_name="c") + + +def test_network_and_sidecar_are_locked_down(tmp_path: Path) -> None: + network = build_network_create_argv("n") + assert {"--internal", "--ipv6=false", "com.docker.network.bridge.inhibit_ipv4=true"} <= set(network) + sidecar = build_sidecar_create_argv( + sidecar="s", + network="n", + image="img", + egress_dir=tmp_path / "egress", + heartbeat=tmp_path / "hb", + stale_seconds=20, + plan=EgressPlan(("a.example:443", "b.example:80")), + ) + joined = " ".join(sidecar) + for flag in ("--sysctl net.ipv4.ip_forward=0", "--user 65534:65534", "--cap-drop ALL", "--read-only"): + assert flag in joined + assert "--security-opt no-new-privileges" in joined + assert f"-v {tmp_path / 'egress'}:/work/egress:ro" in joined + assert [sidecar[i + 1] for i, t in enumerate(sidecar) if t == "--allow"] == ["a.example:443", "b.example:80"] + + +class _FakeDocker: + """Stands in for ``egress._docker``: records each call and answers by its first arguments.""" + + def __init__(self, answers: dict[str, list[subprocess.CompletedProcess[str] | BaseException]] | None = None): + self.calls: list[tuple[str, ...]] = [] + self.answers = answers or {} + + async def __call__(self, *args: str, timeout: float = 30) -> subprocess.CompletedProcess[str]: + self.calls.append(args) + queue = self.answers.get(" ".join(args[:2])) + if queue: + answer = queue.pop(0) + if isinstance(answer, BaseException): + raise answer + return answer + stdout = "egress log line\n" if args[0] == "logs" else "" + return subprocess.CompletedProcess(["docker", *args], 0, stdout, "") + + def verbs(self) -> list[str]: + return [" ".join(call[:2]) for call in self.calls] + + +def _failed(stderr: str = "", stdout: str = "") -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(["docker"], 1, stdout, stderr) + + +def _scope(tmp_path: Path) -> contextlib.AbstractAsyncContextManager[EgressHandle]: + return egress_scope( + container_name="c", + image="coder-eval-agent:x", + egress_dir=tmp_path, + heartbeat=tmp_path / "hb", + stale_seconds=20, + plan=EgressPlan(("api.anthropic.com:443",)), + log_path=tmp_path / "egress.log", + ) + + +async def _enter_scope(tmp_path: Path, fake: _FakeDocker, monkeypatch: pytest.MonkeyPatch) -> Path: + monkeypatch.setattr(egress_mod, "_docker", fake) + monkeypatch.setattr(egress_mod, "_NETWORK_RM_RETRY_SECONDS", 0) + async with _scope(tmp_path) as handle: + assert handle == EgressHandle(network="c-net", sidecar="c-egress", plan=EgressPlan(("api.anthropic.com:443",))) + return tmp_path / "egress.log" + + +async def _cancelled(task: asyncio.Task[None]) -> bool: + try: + await task + except asyncio.CancelledError: + return True + return False + + +async def test_scope_order_and_a_planted_log_symlink_is_replaced_not_followed( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + victim = tmp_path / "victim_rc" + victim.write_text("original\n", encoding="utf-8") + (tmp_path / "egress.log").symlink_to(victim) + fake = _FakeDocker() + log_path = await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs() == [*SETUP, *TEARDOWN] + assert victim.read_text(encoding="utf-8") == "original\n" + assert not log_path.is_symlink() + assert log_path.read_text(encoding="utf-8") == "egress log line\n" + + +PROBE_502 = _failed(stdout="FAIL api.anthropic.com:443 HTTP/1.1 502\n") +NO_IMAGE = _failed("No such image") + + +@pytest.mark.parametrize( + ("step", "answer", "raised", "match", "verbs"), + [ + ("exec c-egress", PROBE_502, EgressSetupError, r"502.*allowlist", [*SETUP, *TEARDOWN]), + ("exec c-egress", asyncio.CancelledError(), asyncio.CancelledError, None, [*SETUP, *TEARDOWN]), + ("image inspect", NO_IMAGE, EgressSetupError, r"coder-eval-agent:x .*make docker-image", SETUP[:1]), + ], + ids=["probe-failure", "cancel-during-probe", "missing-framework-image"], +) +async def test_a_failed_setup_tears_down_what_it_created( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, step: str, answer: Any, raised: Any, match: Any, verbs: list[str] +) -> None: + fake = _FakeDocker({step: [answer]}) + with pytest.raises(raised, match=match): + await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs() == verbs + + +async def test_repeated_cancels_during_teardown_still_finish_it(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + fake = _FakeDocker() + release, in_teardown, entered = asyncio.Event(), asyncio.Event(), asyncio.Event() + + async def _slow_logs(*args: str, timeout: float = 30) -> subprocess.CompletedProcess[str]: + if args[0] == "logs": + in_teardown.set() + await release.wait() + return await fake(*args, timeout=timeout) + + monkeypatch.setattr(egress_mod, "_docker", _slow_logs) + + async def _body() -> None: + async with _scope(tmp_path): + entered.set() + await asyncio.sleep(60) + + task = asyncio.create_task(_body()) + await entered.wait() + task.cancel() + await in_teardown.wait() + task.cancel() + await asyncio.sleep(0) + task.cancel() + release.set() + cancelled = await _cancelled(task) + assert cancelled + assert fake.verbs()[-3:] == TEARDOWN + + +async def test_a_cancel_during_docker_create_waits_for_it_before_teardown( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + order: list[str] = [] + create_started, release_create = threading.Event(), threading.Event() + + def _sync(args: list[str], timeout: float) -> subprocess.CompletedProcess[str]: + if args[0] == "create": + create_started.set() + release_create.wait(10) + order.append("create finished") + else: + order.append(" ".join(args[:2])) + return subprocess.CompletedProcess(["docker", *args], 0, "", "") + + monkeypatch.setattr(egress_mod, "_docker_sync", _sync) + monkeypatch.setattr(egress_mod, "_NETWORK_RM_RETRY_SECONDS", 0) + + async def _body() -> None: + async with _scope(tmp_path): + raise AssertionError("setup must not complete") + + task = asyncio.create_task(_body()) + await asyncio.to_thread(create_started.wait, 10) + task.cancel() + await asyncio.sleep(0.05) + release_create.set() + cancelled = await _cancelled(task) + assert cancelled + assert order.index("create finished") < order.index("rm -f") + assert order[-1] == "network rm" + + +@pytest.mark.parametrize(("failures", "attempts", "gives_up"), [(2, 3, False), (5, 5, True)]) +async def test_network_rm_retries_then_gives_up_with_the_prune_hint( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture, failures, attempts, gives_up +) -> None: + fake = _FakeDocker({"network rm": [_failed("network c-net has active endpoints")] * failures}) + with caplog.at_level(logging.WARNING, logger=egress_mod.logger.name): + await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs().count("network rm") == attempts + assert (PRUNE_HINT in caplog.text) is gives_up + + +async def test_a_network_that_was_never_created_is_not_reported_as_leaked( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + fake = _FakeDocker( + { + "network create": [_failed("all predefined address pools have been fully subnetted")], + "network rm": [_failed("Error response from daemon: network c-net not found")], + } + ) + with ( + caplog.at_level(logging.WARNING, logger=egress_mod.logger.name), + pytest.raises(EgressSetupError, match="--max-parallel") as excinfo, + ): + await _enter_scope(tmp_path, fake, monkeypatch) + assert f"docker network prune --filter label={EGRESS_LABEL}" in str(excinfo.value) + assert fake.verbs().count("network rm") == 1 + assert "Could not remove" not in caplog.text + + +async def test_teardown_survives_a_docker_cli_failure(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + fake = _FakeDocker({"logs c-egress": [OSError("docker vanished")], "rm -f": [subprocess.TimeoutExpired("d", 30)]}) + await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs()[-1] == "network rm" + + +def _rt_runner(tmp_path: Path, network: str) -> DockerRunner: + runner = _runner(network=network, env_passthrough=["ANTHROPIC_API_KEY"]) + runner.rt.run_dir = tmp_path / "run" + runner.rt.replicate_index = 0 + runner.rt.variant_id = "default" + runner.rt.config_lineage = {} + runner.rt.source_yaml = "# task" + return runner + + +@pytest.fixture +def launches(monkeypatch: pytest.MonkeyPatch) -> list[tuple[object, ...]]: + monkeypatch.setattr(dr, "_preflight", lambda: None) + monkeypatch.setattr(dr, "_preflight_image_contract", lambda image, dockerfile: None) + seen: list[tuple[object, ...]] = [] + + async def _fake_exec(*argv: object, **kwargs: object) -> None: + seen.append(argv) + raise FileNotFoundError("docker run is not under test") + + monkeypatch.setattr("asyncio.create_subprocess_exec", _fake_exec) + return seen + + +async def test_llm_only_runs_the_container_inside_the_scope( + hermetic_env: None, launches: list[tuple[object, ...]], tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(dr.settings, "api_backend", ApiBackend.DIRECT) + seen: dict[str, object] = {} + + @contextlib.asynccontextmanager + async def _fake_scope(**kwargs: Any): + seen["plan"] = kwargs["plan"] + seen["staged"] = (kwargs["egress_dir"] / "egress_proxy.py").is_file() + seen["heartbeat"] = Path(kwargs["heartbeat"]).is_file() + try: + yield EgressHandle(network="c-net", sidecar="c-egress", plan=kwargs["plan"]) + finally: + seen["exited"] = True + + monkeypatch.setattr(dr, "egress_scope", _fake_scope) + with pytest.raises(FileNotFoundError): + await _rt_runner(tmp_path, "llm_only").run() + assert seen == { + "plan": EgressPlan(("api.anthropic.com:443", "platform.claude.com:443"), grading_phase=True), + "staged": True, + "heartbeat": True, + "exited": True, + } + assert launches[0][launches[0].index("--network") + 1] == "c-net" + + +@pytest.mark.parametrize(("cause", "match"), [("probe", "probe failed"), ("region", "AWS_REGION")]) +async def test_setup_failure_writes_a_synthetic_error_row( + hermetic_env: None, + launches: list[tuple[object, ...]], + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + cause, + match, +) -> None: + @contextlib.asynccontextmanager + async def _failing_scope(**kwargs: object): + raise EgressSetupError("network: llm_only egress probe failed: x") + yield None + + if cause == "probe": + monkeypatch.setattr(dr, "egress_scope", _failing_scope) + else: + monkeypatch.setattr(dr.settings, "api_backend", ApiBackend.BEDROCK) + monkeypatch.setattr(dr.settings, "aws_region", "not a region") + monkeypatch.setattr(dr.settings, "aws_bearer_token_bedrock", "tok") + runner = _rt_runner(tmp_path, "llm_only") + with pytest.raises(EgressSetupError, match=match): + await runner.run() + assert launches == [] + row = EvaluationResult.model_validate_json((runner.rt.run_dir / TASK_JSON_FILENAME).read_text(encoding="utf-8")) + assert row.final_status == FinalStatus.ERROR + assert match in (row.error_message or "") + + +@pytest.mark.parametrize("docker", [{"network": "bridge", "egress_allowlist": ["pypi.org"]}, {"network": "none"}]) +def test_bridge_and_none_grading_disclosure_is_unchanged(docker: dict[str, Any]) -> None: + assert _container_dispatch_commands(_task(image="img:1", **docker), None) == [ + "docker run img:1 (with your credentials in its environment and a writable copy of ~/.claude)" + ] + + +def test_llm_only_grading_disclosure_names_the_mode_and_the_allowlist() -> None: + [text] = _container_dispatch_commands(_task(image="img:1", network="llm_only", egress_allowlist=["pypi.org"]), None) + assert "--network llm_only" in text + assert "(egress allowed to: pypi.org:443 plus the derived model-API hosts)" in text + [bare] = _container_dispatch_commands(_task(image="img:1", network="llm_only"), None) + assert "--network llm_only" in bare + assert "egress allowed to" not in bare + + +def _graded_runner(tmp_path: Path, *, grade: bool = True, detached: bool = False, **docker: object) -> DockerRunner: + runner = _runner(network="llm_only", env_passthrough=["ANTHROPIC_API_KEY"], **docker) + runner.grade = grade + if detached: + runner.prior_result = MagicMock() + runner.grade_workspace = tmp_path + return runner + + +@pytest.mark.parametrize( + ("docker", "kind", "plan"), + [ + ({"grading_network": "bridge"}, "run", EgressPlan(("api.anthropic.com:443",), (), True, grading_phase=True)), + ( + {"grading_egress_allowlist": ["pypi.org", "api.anthropic.com"]}, + "run", + EgressPlan(("api.anthropic.com:443",), ("pypi.org:443",), grading_phase=True), + ), + ({"grading_network": "bridge"}, "execute", EgressPlan(("api.anthropic.com:443",))), + ( + {"grading_egress_allowlist": ["pypi.org"]}, + "detached", + EgressPlan(("api.anthropic.com:443", "pypi.org:443")), + ), + ({"grading_network": "bridge"}, "detached", None), + ({}, "run", EgressPlan(("api.anthropic.com:443",), grading_phase=True)), + ], + ids=["run-any", "run-extra", "execute", "detached-extra", "detached-bridge", "no-grading-policy"], +) +def test_each_container_gets_the_egress_plan_of_its_phase( + hermetic_env: None, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, docker: dict[str, Any], kind: str, plan: Any +) -> None: + monkeypatch.setattr(dr, "resolve_egress_targets", lambda task, env, settings: ["api.anthropic.com:443"]) + runner = _graded_runner(tmp_path, grade=kind != "execute", detached=kind == "detached", **docker) + assert runner._resolve_egress_plan() == plan + assert runner._container_network() == ("bridge" if plan is None else "llm_only") + + +def test_a_detached_grade_with_grading_network_bridge_joins_the_bridge( + hermetic_env: None, dirs: tuple[Path, Path], tmp_path: Path +) -> None: + argv = _graded_runner(tmp_path, detached=True, grading_network="bridge")._build_argv(*dirs, container_name="c") + assert argv[argv.index("--network") + 1] == "bridge" + assert f"HTTPS_PROXY={PROXY_URL}" not in argv + + +@pytest.mark.parametrize( + "plan", + [ + EgressPlan(("a.example:443",), ("b.example:443", "c.example:80"), grading_phase=True), + EgressPlan(("a.example:443",), (), True, grading_phase=True), + EgressPlan(("a.example:443",), grading_phase=True), + ], + ids=["extra", "any", "phase-only"], +) +def test_sidecar_grading_argv_parses_with_the_proxy_cli(tmp_path: Path, plan: EgressPlan) -> None: + from coder_eval import egress_proxy + + argv = build_sidecar_create_argv( + sidecar="s", + network="n", + image="img", + egress_dir=tmp_path, + heartbeat=tmp_path / "hb", + stale_seconds=20, + plan=plan, + ) + args = egress_proxy._build_parser().parse_args(argv[argv.index("serve") :]) + assert (args.allow, args.grading_allow, args.grading_allow_any) == ( + list(plan.targets), + list(plan.grading_targets), + plan.grading_any, + ) + assert args.control == egress_mod.EGRESS_CONTROL + grading = egress_mod.build_grading_argv("s") + assert egress_proxy._build_parser().parse_args(grading[grading.index("grading") :]).control == args.control + probe = egress_mod.build_probe_argv("s", ["b.example:443"], direct=True) + assert egress_proxy._build_parser().parse_args(probe[probe.index("probe") :]).direct + + +async def test_scope_dials_the_grading_hosts_directly(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + fake = _FakeDocker() + monkeypatch.setattr(egress_mod, "_docker", fake) + plan = EgressPlan(("api.anthropic.com:443",), ("pypi.org:443",), grading_phase=True) + scope = egress_scope( + container_name="c", + image="img", + egress_dir=tmp_path, + heartbeat=tmp_path / "hb", + stale_seconds=20, + plan=plan, + log_path=tmp_path / "egress.log", + ) + async with scope as handle: + assert handle.plan == plan + probes = [call for call in fake.calls if call[0] == "exec"] + assert [("--direct" in call, call[-1]) for call in probes] == [ + (False, "api.anthropic.com:443"), + (True, "pypi.org:443"), + ] + + +@pytest.mark.parametrize( + ("answer", "reply"), + [ + (subprocess.CompletedProcess(["docker"], 0, "OK grading\n", ""), "ok"), + (_failed(stdout="FAIL 127.0.0.1:3129 control unreachable"), "error: "), + ], +) +async def test_the_host_answers_the_grading_request( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, answer: subprocess.CompletedProcess[str], reply: str +) -> None: + from coder_eval.path_utils import EGRESS_GRADING_ACK_FILENAME, EGRESS_GRADING_REQUEST_FILENAME + + fake = _FakeDocker({"exec c-egress": [answer]}) + monkeypatch.setattr(egress_mod, "_docker", fake) + monkeypatch.setattr(dr, "_GRADING_REQUEST_POLL_SECONDS", 0.01) + handle = EgressHandle(network="c-net", sidecar="c-egress", plan=EgressPlan((), (), True, grading_phase=True)) + watch = asyncio.create_task(dr._serve_grading_request(tmp_path, handle)) + await asyncio.sleep(0.05) + assert fake.calls == [] + (tmp_path / EGRESS_GRADING_REQUEST_FILENAME).write_text("grading", encoding="utf-8") + await asyncio.wait_for(watch, 5) + assert (tmp_path / EGRESS_GRADING_ACK_FILENAME).read_text(encoding="utf-8").startswith(reply) + assert fake.calls[0][-3:] == ("grading", "--control", egress_mod.EGRESS_CONTROL) + dr._remove_grading_handshake(tmp_path) + assert list(tmp_path.iterdir()) == [] + + +async def test_enter_grading_refuses_a_plan_without_a_grading_phase() -> None: + with pytest.raises(EgressSetupError, match="no grading phase"): + await egress_mod.enter_grading(EgressHandle(network="n", sidecar="s", plan=EgressPlan(("a.example:443",)))) + with pytest.raises(ValueError, match="needs a grading phase"): + EgressPlan(("a.example:443",), ("b.example:443",)) + + +def test_detached_grading_disclosure_names_the_grading_allowlist() -> None: + task = _task(image="img:1", network="llm_only", egress_allowlist=["pypi.org"], grading_egress_allowlist=["a.io"]) + [text] = _container_dispatch_commands(task, None) + assert "(egress allowed to: pypi.org:443, a.io:443 plus the derived model-API hosts)" in text + [bridge] = _container_dispatch_commands(_task(image="img:1", network="llm_only", grading_network="bridge"), None) + assert "--network llm_only" not in bridge + + +async def test_setup_reclaims_leaked_egress_and_goes_on_when_the_daemon_refuses( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + fake = _FakeDocker( + {"container prune": [_failed("a prune operation is already running")], "network prune": [OSError("gone")]} + ) + await _enter_scope(tmp_path, fake, monkeypatch) + prunes = [call for call in fake.calls if call[1:2] == ("prune",)] + assert prunes == [ + (kind, "prune", "--force", "--filter", f"label={EGRESS_LABEL}", "--filter", "until=10m") + for kind in ("container", "network") + ] + assert fake.verbs() == [*SETUP, *TEARDOWN] + + +PROXY_LOG = """READY 0.0.0.0:3128 allow=api.anthropic.com:443 grading=unchanged +DENY example.com:443 CONNECT +PHASE grading allow=unchanged +ALLOW bedrock-runtime.us-east-2.amazonaws.com:443 CONNECT +DENY pypi.org:443 CONNECT +DENY http-intake.logs.us5.datadoghq.com:443 CONNECT +DENY pypi.org:443 CONNECT +BAD GET unparseable request line (12 bytes) +DENY deb.debian.org:80 GET +""" + + +@pytest.mark.parametrize( + ("log", "whole_log", "denied"), + [ + (PROXY_LOG, False, ["deb.debian.org:80", "pypi.org:443"]), + (PROXY_LOG, True, ["deb.debian.org:80", "example.com:443", "pypi.org:443"]), + ("READY x allow=\nDENY pypi.org:443 CONNECT\n", False, []), + ], + ids=["after-the-phase-line", "grading-only-container", "no-phase-line"], +) +def test_grading_denials_skip_the_agent_phase_and_judge_noise(log: str, whole_log: bool, denied: list[str]) -> None: + assert grading_denials(log, whole_log=whole_log) == denied + + +def _result(status: FinalStatus) -> EvaluationResult: + return EvaluationResult( + task_id="t", + task_description="d", + agent_type="claude-code", + started_at=datetime.now(UTC), + final_status=status, + iteration_count=1, + ) + + +@pytest.mark.parametrize( + ("status", "log", "after"), + [ + (FinalStatus.FAILURE, PROXY_LOG, FinalStatus.ERROR), + (FinalStatus.SUCCESS, PROXY_LOG, FinalStatus.SUCCESS), + (FinalStatus.FAILURE, "PHASE grading allow=unchanged\nDENY api.anthropic.com:443 CONNECT\n", None), + ], + ids=["failure-becomes-error", "success-is-kept", "only-noise"], +) +async def test_a_refused_grading_host_is_recorded_and_a_failure_becomes_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, status: FinalStatus, log: str, after: FinalStatus | None +) -> None: + monkeypatch.setattr( + egress_mod, "_docker", _FakeDocker({"logs c-egress": [subprocess.CompletedProcess([], 0, log, "")]}) + ) + task_json = tmp_path / TASK_JSON_FILENAME + task_json.symlink_to(tmp_path / "victim") + result = _result(status) + handle = EgressHandle(network="c-net", sidecar="c-egress", plan=EgressPlan(("a.example:443",), grading_phase=True)) + await _graded_runner(tmp_path)._record_grading_denials(result, handle, task_json) + if after is None: + assert (result.final_status, result.environment_info) == (status, {}) + assert task_json.is_symlink() + return + assert result.final_status == after + assert result.environment_info["egress_denied_in_grading"] == ["deb.debian.org:80", "pypi.org:443"] + assert ("deb.debian.org:80, pypi.org:443" in (result.error_message or "")) is (after == FinalStatus.ERROR) + assert not task_json.is_symlink() + assert not (tmp_path / "victim").exists() + assert EvaluationResult.model_validate_json(task_json.read_text(encoding="utf-8")).final_status == after diff --git a/tests/test_docker_litellm_env.py b/tests/test_docker_litellm_env.py index b9f0307f5..c27f76c6e 100644 --- a/tests/test_docker_litellm_env.py +++ b/tests/test_docker_litellm_env.py @@ -14,10 +14,8 @@ import pytest -from coder_eval.isolation.docker_runner import ( - DockerRunner, - _rewrite_loopback_for_container, -) +from coder_eval.isolation.docker_runner import DockerRunner +from coder_eval.isolation.egress import rewrite_loopback_for_container from coder_eval.models import DockerDriverConfig, FileExistsCriterion, SandboxConfig, TaskDefinition @@ -28,19 +26,19 @@ class TestRewriteLoopbackForContainer: """localhost/127.0.0.1 -> host.docker.internal, preserving scheme/port/path.""" def test_localhost_with_port(self): - assert _rewrite_loopback_for_container("http://localhost:4000") == "http://host.docker.internal:4000" + assert rewrite_loopback_for_container("http://localhost:4000") == "http://host.docker.internal:4000" def test_127_0_0_1_with_port(self): - assert _rewrite_loopback_for_container("http://127.0.0.1:4000") == "http://host.docker.internal:4000" + assert rewrite_loopback_for_container("http://127.0.0.1:4000") == "http://host.docker.internal:4000" def test_preserves_scheme_and_path(self): - assert _rewrite_loopback_for_container("https://localhost:8443/v1") == "https://host.docker.internal:8443/v1" + assert rewrite_loopback_for_container("https://localhost:8443/v1") == "https://host.docker.internal:8443/v1" def test_no_port(self): - assert _rewrite_loopback_for_container("http://localhost") == "http://host.docker.internal" + assert rewrite_loopback_for_container("http://localhost") == "http://host.docker.internal" def test_non_loopback_returns_none(self): - assert _rewrite_loopback_for_container("http://litellm.internal:4000") is None + assert rewrite_loopback_for_container("http://litellm.internal:4000") is None class TestLitellmEnvForwarding: diff --git a/tests/test_egress_grading_phase.py b/tests/test_egress_grading_phase.py new file mode 100644 index 000000000..759a894ba --- /dev/null +++ b/tests/test_egress_grading_phase.py @@ -0,0 +1,95 @@ +"""The grading half of ``network: llm_only``: the in-container request and the orchestrator hook.""" + +from __future__ import annotations + +import asyncio +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from coder_eval.cli.run_task_internal_command import request_grading_egress +from coder_eval.config import settings +from coder_eval.models import ( + AgentKind, + ApiBackend, + DockerDriverConfig, + FileExistsCriterion, + FinalStatus, + RunCommandCriterion, + SandboxConfig, + TaskDefinition, + parse_agent_config, +) +from coder_eval.models.tasks import SimulationConfig +from coder_eval.orchestrator import Orchestrator +from coder_eval.path_utils import EGRESS_GRADING_ACK_FILENAME, EGRESS_GRADING_REQUEST_FILENAME + + +async def _answer(output_dir: Path, reply: str) -> None: + request = output_dir / EGRESS_GRADING_REQUEST_FILENAME + while not request.is_file(): + await asyncio.sleep(0.01) + (output_dir / EGRESS_GRADING_ACK_FILENAME).write_text(reply, encoding="utf-8") + + +async def test_request_returns_once_the_host_confirms(tmp_path: Path) -> None: + await asyncio.gather(request_grading_egress(tmp_path, timeout=5), _answer(tmp_path, "ok\n")) + assert (tmp_path / EGRESS_GRADING_REQUEST_FILENAME).read_text(encoding="utf-8") == "grading" + + +async def test_request_raises_on_a_refusal_or_no_answer(tmp_path: Path) -> None: + with pytest.raises(RuntimeError, match=r"could not switch.*error: probe"): + await asyncio.gather(request_grading_egress(tmp_path, timeout=5), _answer(tmp_path, "error: probe")) + (tmp_path / EGRESS_GRADING_ACK_FILENAME).unlink() + with pytest.raises(RuntimeError, match="did not switch"): + await request_grading_egress(tmp_path, timeout=0.2) + + +async def test_a_switch_that_only_marks_the_phase_does_not_stop_grading( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + await asyncio.gather(request_grading_egress(tmp_path, timeout=5, widens=False), _answer(tmp_path, "error: x")) + (tmp_path / EGRESS_GRADING_ACK_FILENAME).unlink() + await request_grading_egress(tmp_path, timeout=0.2, widens=False) + assert caplog.text.count("Refused grading hosts will not be reported") == 2 + + +def test_a_simulated_task_cannot_widen_grading() -> None: + with pytest.raises(ValidationError, match="between dialog turns"): + TaskDefinition( + task_id="t", + description="d", + agent=parse_agent_config(type="claude-code"), + sandbox=SandboxConfig( + driver="docker", docker=DockerDriverConfig(network="llm_only", grading_network="bridge") + ), + simulation=SimulationConfig(enabled=True, persona="a user", goal="get help"), + success_criteria=[FileExistsCriterion(description="c", path="x.txt")], + ) + + +async def test_the_hook_runs_once_before_the_criteria(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(settings, "api_backend", ApiBackend.DIRECT) + marker = tmp_path / "grading-started" + task = TaskDefinition( + task_id="noop", + description="d", + agent=parse_agent_config(type=AgentKind.NONE), + sandbox=SandboxConfig(driver="tempdir"), + success_criteria=[RunCommandCriterion(description="after the hook", command=f"test -f {marker}")], + ) + run_dir = tmp_path / "run" + run_dir.mkdir() + orch = Orchestrator(task=task, run_dir=run_dir, variant_id="v") + calls: list[int] = [] + + async def _hook() -> None: + calls.append(1) + marker.touch() + + orch.before_grading = _hook + await orch.run() + await orch._start_grading() + assert calls == [1] + assert orch.result is not None and orch.result.final_status == FinalStatus.SUCCESS diff --git a/tests/test_egress_proxy.py b/tests/test_egress_proxy.py new file mode 100644 index 000000000..33ca11cc3 --- /dev/null +++ b/tests/test_egress_proxy.py @@ -0,0 +1,395 @@ +"""The stdlib-only egress proxy: real sockets on 127.0.0.1, no docker.""" + +from __future__ import annotations + +import ast +import asyncio +import contextlib +import socket +import subprocess +import sys +import time +from collections.abc import AsyncIterator +from pathlib import Path + +import pytest + +from coder_eval import egress_proxy +from coder_eval.cli.run_task_internal_command import heartbeat_is_alive + + +PROXY_FILE = Path(egress_proxy.__file__) +ALLOWED_IMPORTS = {"argparse", "asyncio", "os", "sys", "time"} + + +async def _upstream(handler) -> tuple[asyncio.Server, int]: + server = await asyncio.start_server(handler, "127.0.0.1", 0) + return server, server.sockets[0].getsockname()[1] + + +async def _echo(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + while data := await reader.read(1024): + writer.write(data) + await writer.drain() + writer.close() + + +@contextlib.asynccontextmanager +async def _proxy(allow: set[str] | egress_proxy.Policy) -> AsyncIterator[int]: + policy = allow if isinstance(allow, egress_proxy.Policy) else egress_proxy.Policy(frozenset(allow)) + server = await egress_proxy.start_proxy(policy, "127.0.0.1", 0) + try: + yield server.sockets[0].getsockname()[1] + finally: + server.close() + server.abort_clients() + + +async def _request(port: int, head: bytes) -> tuple[asyncio.StreamReader, asyncio.StreamWriter, bytes]: + reader, writer = await asyncio.open_connection("127.0.0.1", port) + writer.write(head) + await writer.drain() + status = await asyncio.wait_for(reader.readline(), 5) + return reader, writer, status + + +async def _status(allow: set[str], head: bytes) -> bytes: + async with _proxy(allow) as port: + _reader, writer, status = await _request(port, head) + writer.close() + return status.split()[1] + + +def _free_port() -> int: + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + return sock.getsockname()[1] + + +async def test_connect_to_allowlisted_target_tunnels_both_ways(capsys): + echo, echo_port = await _upstream(_echo) + async with echo, _proxy({f"localhost:{echo_port}"}) as port: + reader, writer, status = await _request(port, f"CONNECT LocalHost:{echo_port} HTTP/1.1\r\n\r\n".encode()) + assert status.startswith(b"HTTP/1.1 200") + assert await reader.readline() == b"\r\n" + writer.write(b"ping") + await writer.drain() + assert await asyncio.wait_for(reader.readexactly(4), 5) == b"ping" + writer.close() + assert f"ALLOW localhost:{echo_port} CONNECT" in capsys.readouterr().out + + +async def test_connect_to_denied_target_is_403(capsys): + assert await _status(set(), b"CONNECT 127.0.0.1:9 HTTP/1.1\r\n\r\n") == b"403" + assert capsys.readouterr().out.splitlines() == ["DENY 127.0.0.1:9 CONNECT"] + + +async def test_connect_to_allowlisted_closed_port_is_502(capsys): + closed = _free_port() + assert await _status({f"127.0.0.1:{closed}"}, f"CONNECT 127.0.0.1:{closed} HTTP/1.1\r\n\r\n".encode()) == b"502" + assert f"FAIL 127.0.0.1:{closed}" in capsys.readouterr().out + + +@pytest.mark.parametrize( + ("line", "expected"), + [ + ("CONNECT 127.0.0.1 HTTP/1.1", b"400"), + ("CONNECT 127.0.0.1:abc HTTP/1.1", b"400"), + ("CONNECT 127.0.0.1:70000 HTTP/1.1", b"400"), + ("CONNECT [::1 HTTP/1.1", b"400"), + ("CONNECT [::1]:443 HTTP/1.1", b"403"), + ("GET http://example.com/ HTTP/1.1", b"403"), + ("GET https://127.0.0.1:443/ HTTP/1.1", b"400"), + ("GET http://127.0.0.1:443@evil.example/ HTTP/1.1", b"403"), + ("GET http://127.0.0.1:443:443/ HTTP/1.1", b"403"), + ("GET http://evil%2eexample/ HTTP/1.1", b"403"), + ("GET http://evil.example./ HTTP/1.1", b"403"), + ("GET http://[::1%25eth0]/ HTTP/1.1", b"403"), + ], +) +async def test_unallowed_or_unparseable_authorities_are_refused(line: str, expected: bytes): + assert await _status({"127.0.0.1:443"}, f"{line}\r\n\r\n".encode()) == expected + + +async def test_absolute_form_https_is_400_with_explanation(capsys): + assert await _status({"example.com:443"}, b"GET https://example.com/ HTTP/1.1\r\n\r\n") == b"400" + assert "BAD example.com:443 absolute-form https (use CONNECT)" in capsys.readouterr().out + + +async def test_absolute_form_get_is_rewritten_to_origin_form(capsys): + received: list[bytes] = [] + + async def _http(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + received.append(await reader.readuntil(b"\r\n\r\n")) + writer.write(b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\nConnection: close\r\n\r\nhello") + await writer.drain() + writer.close() + + upstream, up_port = await _upstream(_http) + head = ( + f"GET http://user:pw@127.0.0.1:{up_port}/path?q=1 HTTP/1.1\r\n" + f"Host: 127.0.0.1:{up_port}\r\nProxy-Connection: keep-alive\r\nProxy-Authorization: x\r\n" + "Connection: keep-alive\r\nAccept: */*\r\n\r\n" + ).encode() + async with upstream, _proxy({f"127.0.0.1:{up_port}"}) as port: + reader, writer, status = await _request(port, head) + body = await asyncio.wait_for(reader.read(), 5) + writer.close() + assert status.startswith(b"HTTP/1.1 200") + assert body.endswith(b"hello") + forwarded = received[0].decode() + assert forwarded.startswith("GET /path?q=1 HTTP/1.1\r\n") + assert "Proxy-" not in forwarded + assert "keep-alive" not in forwarded + assert "Connection: close\r\n" in forwarded + assert "Accept: */*" in forwarded + assert f"ALLOW 127.0.0.1:{up_port} GET" in capsys.readouterr().out + + +@pytest.mark.parametrize( + "line", + [ + b"garbage", + b"GET /relative HTTP/1.1", + b"GET http://u:SECRET@h:99999/?token=SECRET HTTP/1.1", + b"X\nALLOW\tevil.com:443\tCONNECT http://evil.com/ HTTP/1.1", + b"CONNECT evil.com:443 HTTP/1.1\x00", + b"GET http://evil.com/\xff HTTP/1.1", + b"GET http://evil.com/ HTTP/1.1\tX", + ], +) +async def test_a_bad_request_line_is_one_redacted_log_line(line: bytes, capsys): + assert await _status(set(), line + b"\r\n\r\n") == b"400" + out = capsys.readouterr().out + assert len(out.splitlines()) == 1 + assert out.startswith("BAD ") + assert "SECRET" not in out + + +async def test_connections_beyond_the_cap_get_503(monkeypatch, capsys): + monkeypatch.setattr(egress_proxy, "MAX_CONNECTIONS", 1) + async with _proxy(set()) as port: + _held_reader, held_writer = await asyncio.open_connection("127.0.0.1", port) + await asyncio.sleep(0.05) + _reader, writer, status = await _request(port, b"CONNECT a.example:443 HTTP/1.1\r\n\r\n") + writer.close() + held_writer.close() + assert status.startswith(b"HTTP/1.1 503") + assert "BAD too-many-connections" in capsys.readouterr().out + + +async def test_oversized_head_closes_the_connection(capsys): + async with _proxy(set()) as port: + reader, writer = await asyncio.open_connection("127.0.0.1", port) + writer.write(b"GET http://x/ HTTP/1.1\r\nX: " + b"a" * (egress_proxy.HEAD_LIMIT_BYTES + 10)) + with contextlib.suppress(ConnectionError): + await writer.drain() + assert await asyncio.wait_for(reader.read(), 5) == b"" + writer.close() + assert "BAD head-too-large" in capsys.readouterr().out + + +async def test_connect_half_close_still_delivers_the_response(): + async def _reply_after_eof(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + writer.write(b"got:" + await reader.read()) + await writer.drain() + writer.close() + + upstream, up_port = await _upstream(_reply_after_eof) + async with upstream, _proxy({f"127.0.0.1:{up_port}"}) as port: + reader, writer, status = await _request(port, f"CONNECT 127.0.0.1:{up_port} HTTP/1.1\r\n\r\n".encode()) + assert status.startswith(b"HTTP/1.1 200") + assert await reader.readline() == b"\r\n" + writer.write(b"question") + writer.write_eof() + assert await asyncio.wait_for(reader.read(), 5) == b"got:question" + writer.close() + + +async def test_upstream_closing_mid_stream_closes_the_client(): + async def _one_shot(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + writer.write(b"data: 1\n\n") + await writer.drain() + writer.close() + + upstream, up_port = await _upstream(_one_shot) + async with upstream, _proxy({f"127.0.0.1:{up_port}"}) as port: + reader, writer, status = await _request(port, f"CONNECT 127.0.0.1:{up_port} HTTP/1.1\r\n\r\n".encode()) + assert status.startswith(b"HTTP/1.1 200") + rest = await asyncio.wait_for(reader.read(), 5) + writer.close() + assert rest == b"\r\ndata: 1\n\n" + + +async def test_probe_succeeds_only_when_every_target_is_allowed(monkeypatch, capsys): + echo, echo_port = await _upstream(_echo) + async with echo, _proxy({f"127.0.0.1:{echo_port}"}) as port: + ok = await egress_proxy.probe(f"127.0.0.1:{port}", [f"127.0.0.1:{echo_port}"], 5) + mixed = await egress_proxy.probe(f"127.0.0.1:{port}", [f"127.0.0.1:{echo_port}", "denied.example:443"], 5) + monkeypatch.setattr(egress_proxy, "PROBE_STARTUP_RETRY_SECONDS", 0.3) + unreachable = await egress_proxy.probe(f"127.0.0.1:{_free_port()}", ["a.example:443"], 1) + out = capsys.readouterr().out + assert (ok, mixed, unreachable) == (0, 1, 1) + assert f"OK 127.0.0.1:{echo_port}" in out + assert "FAIL denied.example:443 HTTP/1.1 403 Forbidden" in out + assert "FAIL a.example:443 proxy unreachable" in out + + +async def test_watchdog_stops_serve_on_a_stale_or_missing_heartbeat(tmp_path, capsys): + heartbeat = tmp_path / "hb" + heartbeat.write_text("1", encoding="utf-8") + started = time.monotonic() + await asyncio.wait_for(egress_proxy.serve(egress_proxy.Policy(frozenset()), "127.0.0.1", 0, str(heartbeat), 1.0), 5) + assert time.monotonic() - started < 3.5 + await asyncio.wait_for(egress_proxy.watch_heartbeat(str(tmp_path / "absent"), 0.5), 5) + assert capsys.readouterr().out.count("STALE") == 2 + + +async def test_watchdog_stays_alive_while_the_counter_advances(tmp_path): + heartbeat = tmp_path / "hb" + heartbeat.write_text("0", encoding="utf-8") + watcher = asyncio.create_task(egress_proxy.watch_heartbeat(str(heartbeat), 1.0)) + for counter in range(1, 15): + await asyncio.to_thread(heartbeat.write_text, str(counter), encoding="utf-8") + await asyncio.sleep(0.2) + assert not watcher.done() + watcher.cancel() + await asyncio.gather(watcher, return_exceptions=True) + + +@pytest.mark.parametrize( + ("current", "last", "mtime", "last_mtime"), + [ + ("", "", 0.0, 0.0), + ("", "", 5.0, 0.0), + ("1", "", 0.0, 0.0), + ("1", "1", 5.0, 5.0), + ("2", "1", 5.0, 5.0), + ("1", "1", 6.0, 5.0), + ("", "1", 5.0, 5.0), + ("1", "1", 4.0, 5.0), + ], +) +def test_liveness_rule_matches_the_container_watchdog(current: str, last: str, mtime: float, last_mtime: float): + assert egress_proxy.heartbeat_alive(current, last, mtime, last_mtime) == heartbeat_is_alive( + current, last, mtime, last_mtime + ) + + +def test_module_imports_only_the_five_stdlib_modules(): + roots: set[str] = set() + for node in ast.walk(ast.parse(PROXY_FILE.read_text(encoding="utf-8"))): + if isinstance(node, ast.Import): + roots |= {alias.name.split(".")[0] for alias in node.names} + elif isinstance(node, ast.ImportFrom): + assert node.level == 0, "egress_proxy.py must not use a relative import" + assert node.module is not None + roots.add(node.module.split(".")[0]) + assert roots <= ALLOWED_IMPORTS + + +def test_runs_standalone_in_isolated_mode(): + result = subprocess.run( + [sys.executable, "-I", str(PROXY_FILE), "--help"], capture_output=True, text=True, encoding="utf-8", timeout=20 + ) + assert result.returncode == 0 + assert "serve" in result.stdout + assert "probe" in result.stdout + assert "grading" in result.stdout + + +@pytest.mark.parametrize( + ("extra", "named"), + [ + (["--listen", "nonsense"], "BAD --listen"), + *( + (["--allow", entry], "BAD --allow") + for entry in ["example.com", "[::1]:443", "a.example:443:1", "a.example:0"] + ), + (["--heartbeat", "hb"], "--stale"), + *((["--heartbeat", "hb", "--stale", stale], "--stale") for stale in ["0", "nan", "inf"]), + (["--grading-allow", "a.example:443"], "need --control"), + (["--grading-allow-any"], "need --control"), + (["--control", "0.0.0.0:3129"], "BAD --control"), + (["--control", "127.0.0.1:3129", "--grading-allow", "a.example"], "BAD --grading-allow"), + ], +) +def test_main_refuses_invalid_arguments(extra: list[str], named: str, capsys): + assert egress_proxy.main(["serve", "--listen", f"127.0.0.1:{_free_port()}", *extra]) == 2 + assert named in capsys.readouterr().out + + +def test_main_serves_until_the_heartbeat_is_stale(tmp_path, capsys): + heartbeat = tmp_path / "hb" + heartbeat.write_text("1", encoding="utf-8") + argv = ["serve", "--listen", f"127.0.0.1:{_free_port()}", "--heartbeat", str(heartbeat), "--stale", "0.5"] + assert egress_proxy.main([*argv, "--allow", "A.example:443"]) == 0 + out = capsys.readouterr().out + assert "allow=a.example:443" in out + assert "STALE" in out + + +async def _connect_status(port: int, target: str) -> bytes: + _reader, writer, status = await _request(port, f"CONNECT {target} HTTP/1.1\r\n\r\n".encode()) + writer.close() + return status.split()[1] + + +@pytest.mark.parametrize("grading_any", [False, True]) +async def test_grading_widens_the_allowlist_one_way(grading_any: bool, capsys): + echo, echo_port = await _upstream(_echo) + agent_target, grading_target = f"localhost:{echo_port}", f"127.0.0.1:{echo_port}" + policy = egress_proxy.Policy(frozenset({agent_target}), frozenset({grading_target}), grading_any) + control = await egress_proxy.start_control(policy, "127.0.0.1", 0) + control_address = f"127.0.0.1:{control.sockets[0].getsockname()[1]}" + async with echo, control, _proxy(policy) as port: + before = [await _connect_status(port, t) for t in (agent_target, grading_target)] + switched = await egress_proxy.request_grading(control_address, 5) + again = await egress_proxy.request_grading(control_address, 5) + after = [await _connect_status(port, t) for t in (agent_target, grading_target, "other.invalid:443")] + assert before == [b"200", b"403"] + assert (switched, again) == (0, 0) + assert after[:2] == [b"200", b"200"] + # Any host is forwarded, so an unresolvable one fails upstream (502) instead of being denied (403). + assert after[2] == (b"502" if grading_any else b"403") + out = capsys.readouterr().out + assert out.count("PHASE grading") == 1 + assert ("PHASE grading allow=*" in out) == grading_any + + +async def test_control_refuses_any_other_request(capsys): + policy = egress_proxy.Policy(frozenset(), frozenset({"a.example:443"})) + control = await egress_proxy.start_control(policy, "127.0.0.1", 0) + async with control: + reader, writer = await asyncio.open_connection("127.0.0.1", control.sockets[0].getsockname()[1]) + writer.write(b"grading\n") + await writer.drain() + reply = await asyncio.wait_for(reader.readline(), 5) + writer.close() + assert reply == b"BAD\n" + assert not policy.grading + assert "BAD control request" in capsys.readouterr().out + + +async def test_request_grading_fails_when_nothing_listens(capsys): + assert await egress_proxy.request_grading(f"127.0.0.1:{_free_port()}", 1) == 1 + assert "control unreachable" in capsys.readouterr().out + + +async def test_direct_probe_dials_each_target(capsys): + echo, echo_port = await _upstream(_echo) + async with echo: + ok = await egress_proxy.probe(None, [f"127.0.0.1:{echo_port}"], 5) + mixed = await egress_proxy.probe(None, [f"127.0.0.1:{echo_port}", f"127.0.0.1:{_free_port()}"], 1) + assert (ok, mixed) == (0, 1) + assert f"OK 127.0.0.1:{echo_port}" in capsys.readouterr().out + + +def test_main_serves_with_a_grading_policy(tmp_path, capsys): + heartbeat = tmp_path / "hb" + heartbeat.write_text("1", encoding="utf-8") + argv = ["serve", "--listen", f"127.0.0.1:{_free_port()}", "--heartbeat", str(heartbeat), "--stale", "0.5"] + control = ["--control", f"127.0.0.1:{_free_port()}"] + assert egress_proxy.main([*argv, *control, "--allow", "a.example:443", "--grading-allow", "B.example"]) == 2 + assert egress_proxy.main([*argv, *control, "--allow", "a.example:443", "--grading-allow", "B.example:443"]) == 0 + assert "allow=a.example:443 grading=+b.example:443" in capsys.readouterr().out diff --git a/tests/test_egress_targets.py b/tests/test_egress_targets.py new file mode 100644 index 000000000..7719b11e6 --- /dev/null +++ b/tests/test_egress_targets.py @@ -0,0 +1,199 @@ +"""The host-side allowlist derivation for ``network: llm_only`` and the agent-config hook.""" + +from __future__ import annotations + +import logging +import traceback +from typing import Any + +import pytest + +from coder_eval.agents.registry import AgentRegistry +from coder_eval.config import Settings +from coder_eval.isolation import docker_runner, errors +from coder_eval.isolation.egress import resolve_egress_targets +from coder_eval.models import ( + AgentJudgeCriterion, + ApiBackend, + ApiRouteContext, + CheckerContext, + DockerDriverConfig, + FileExistsCriterion, + LLMJudgeCriterion, + NoulQuestion, + SandboxConfig, + SimulationConfig, + SystemOneJudgeCriterion, + TaskDefinition, + normalize_egress_target, + parse_agent_config, +) +from coder_eval.plugins import ensure_plugins_loaded + + +DIRECT = ["api.anthropic.com:443", "platform.claude.com:443"] +BEDROCK = ["bedrock-runtime.eu-north-1.amazonaws.com:443", "bedrock.eu-north-1.amazonaws.com:443"] +OPENAI, GEMINI, OPENROUTER = "api.openai.com:443", "generativelanguage.googleapis.com:443", "openrouter.ai:443" +LITELLM = ApiBackend.LITELLM +BR = {"backend": ApiBackend.BEDROCK, "region": "eu-north-1"} +CC, CODEX, PI, OPENCODE = {"type": "claude-code"}, {"type": "codex"}, {"type": "pi"}, {"type": "opencode"} +LL_URL, CX_URL, REGION = "LITELLM_BASE_URL", "CODEX_BASE_URL", "AWS_REGION" +JUDGE = LLMJudgeCriterion(description="j", prompt="p") +QUESTIONS = {"q": NoulQuestion(instructions="i")} +SYSTEM_ONE = [ + SystemOneJudgeCriterion(description="j", questions=QUESTIONS), + SystemOneJudgeCriterion(description="k", questions=QUESTIONS, base_url="https://gw.example:9443/v1"), +] +DIRECT_ROUTE = ApiRouteContext(route=ApiBackend.DIRECT) +EGRESS_LOG = "coder_eval.isolation.egress" + + +def _targets( + agent: dict[str, Any] | None = CC, + *, + backend: ApiBackend = ApiBackend.DIRECT, + region: str | None = None, + env: dict[str, str] | None = None, + criteria: list[Any] | None = None, + simulation: bool | None = None, + route: ApiRouteContext | None = None, + bedrock_token: str | None = "tok", + **docker: Any, +) -> list[str]: + task = TaskDefinition( + task_id="t", + description="d", + initial_prompt="p", + agent=parse_agent_config(**(agent or CC)), + sandbox=SandboxConfig(driver="docker", docker=DockerDriverConfig(network="llm_only", **docker)), + success_criteria=criteria or [FileExistsCriterion(description="c", path="x.txt")], + ) + if agent is None: + task.agent = None + if simulation is not None: + task.simulation = SimulationConfig(enabled=simulation, persona="p", goal="g") + if route is not None: + task.checker_context = CheckerContext(api_route=route) + settings = Settings.model_construct( + api_backend=backend, + aws_region=region, + aws_bearer_token_bedrock=bedrock_token if region else None, + anthropic_api_key="key", + litellm_base_url="http://gateway.invalid", + litellm_auth_token="tok", + litellm_model="m", + ) + return resolve_egress_targets(task, env=env or {}, settings=settings) + + +HDI = ["host.docker.internal:4000"] +CASES: dict[str, tuple[dict[str, Any] | None, dict[str, Any], list[str]]] = { + "direct-claude-code": (CC, {}, DIRECT), + "bedrock-region-lowercased": (CC, {"backend": ApiBackend.BEDROCK, "region": "EU-North-1"}, BEDROCK), + "unresolved-agent": (None, {}, []), + "litellm-loopback": (CC, {"backend": LITELLM, "env": {LL_URL: "http://localhost:4000"}}, HDI), + "litellm-ipv6-loopback": (CC, {"backend": LITELLM, "env": {LL_URL: "http://[::1]:4000"}}, HDI), + "litellm-remote-port": (CC, {"backend": LITELLM, "env": {LL_URL: "https://l.corp:8443/v1"}}, ["l.corp:8443"]), + "codex-base-url": (CODEX, {"env": {CX_URL: "https://x.openai.azure.com/v1"}}, ["x.openai.azure.com:443"]), + "codex-plain-http": (CODEX, {"env": {CX_URL: "http://gw.example/v1"}}, ["gw.example:80"]), + "claude-code-ignores-codex-url": (CC, {"env": {CX_URL: "https://x.openai.azure.com"}}, DIRECT), + "replaced-passthrough": (CODEX, {"env": {CX_URL: "https://x.corp"}, "env_passthrough": []}, [OPENAI]), + "forwarded-url-vars": (CC, {"env": {"MY_URL": "https://s.example"}, "env_passthrough_extra": ["MY_URL"]}, DIRECT), + "user-allowlist": (CC, {"egress_allowlist": ["pypi.org", "API.anthropic.com"]}, sorted([*DIRECT, "pypi.org:443"])), + "codex-off-bedrock": (CODEX, BR, [OPENAI]), + "antigravity-off-bedrock": ({"type": "antigravity"}, BR, [GEMINI]), + "pi-off-bedrock": (PI, BR, [OPENROUTER]), + "delegate-off-bedrock": ({"type": "delegate"}, BR, []), + "pi-anthropic": ({**PI, "model": "anthropic/claude-haiku-4-5"}, {}, ["api.anthropic.com:443"]), + "pi-openrouter": ({**PI, "model": "openrouter/moonshotai/kimi-k3"}, {}, [OPENROUTER]), + "pi-openai": ({**PI, "model": "openai/gpt-5"}, {}, [OPENAI]), + "pi-google": ({**PI, "model": "google/gemini-3"}, {}, [GEMINI]), + "pi-unknown-prefix": ({**PI, "model": "unknown/x"}, {}, [OPENROUTER]), + "pi-no-prefix": ({**PI, "model": "no-prefix"}, {}, [OPENROUTER]), + "opencode-no-model": (OPENCODE, {"backend": LITELLM}, []), + "system-one-judge": (CC, {"backend": LITELLM, "criteria": SYSTEM_ONE}, ["api.typesafe.ai:443", "gw.example:9443"]), + "judge-adds-backend": ({"type": "antigravity"}, {**BR, "criteria": [JUDGE]}, sorted([*BEDROCK, GEMINI])), + "disabled-judge": (CODEX, {"criteria": [LLMJudgeCriterion(description="j", prompt="p", enabled=False)]}, [OPENAI]), + "judge-route-override": ( + CODEX, + {**BR, "criteria": [AgentJudgeCriterion(description="j", prompt="p")], "route": DIRECT_ROUTE}, + sorted([*DIRECT, OPENAI]), + ), + "simulation-enabled": (CODEX, {"simulation": True}, sorted([*DIRECT, OPENAI])), + "simulation-disabled": (CODEX, {"simulation": False}, [OPENAI]), + "litellm-simulator-pinned": (CODEX, {"backend": LITELLM, "simulation": True}, sorted([*DIRECT, OPENAI])), +} + + +@pytest.mark.parametrize(("agent", "kwargs", "expected"), list(CASES.values()), ids=list(CASES)) +def test_resolved_targets(agent: dict[str, Any] | None, kwargs: dict[str, Any], expected: list[str]): + assert _targets(agent, **kwargs) == expected + + +@pytest.mark.parametrize( + ("agent", "kwargs", "variable"), + [ + *( + (CODEX, {"env": {CX_URL: url}}, CX_URL) + for url in ["https://u:secret@[::2]", "http://[secret", "https://secret:70000"] + ), + (CC, {"backend": LITELLM, "env": {LL_URL: "http://localhost:secret"}}, LL_URL), + (CC, {"backend": ApiBackend.BEDROCK, "region": "secret region"}, REGION), + ], +) +def test_unallowlistable_value_raises_without_echoing_it(agent: dict[str, Any], kwargs: dict[str, Any], variable: str): + with pytest.raises(ValueError, match=variable) as excinfo: + _targets(agent, **kwargs) + assert "secret" not in "".join(traceback.format_exception(excinfo.value)) + + +@pytest.mark.parametrize( + ("agent", "kwargs", "logger", "named", "hidden"), + [ + (CC, {"backend": ApiBackend.BEDROCK}, EGRESS_LOG, REGION, None), + (CC, {"backend": LITELLM}, EGRESS_LOG, LL_URL, None), + (CC, {"bedrock_token": None, **BR}, EGRESS_LOG, "AWS_BEARER_TOKEN_BEDROCK", None), + (CODEX, {"env": {CX_URL: "http://127.0.0.1:8080"}}, "coder_eval.models.sandbox", CX_URL, "8080"), + ], + ids=["bedrock-no-region", "litellm-no-url", "unconfigured-backend", "non-litellm-loopback"], +) +def test_unusable_route_warns_and_adds_nothing(agent, kwargs, logger, named, hidden, caplog): + with caplog.at_level(logging.WARNING, logger=logger): + assert _targets(agent, **kwargs) == [] + assert named in caplog.text + assert hidden is None or hidden not in caplog.text + + +@pytest.mark.parametrize(("region", "expected"), [("eu-north-1", BEDROCK), (None, DIRECT)]) +def test_litellm_agent_judge_and_simulator_get_the_pinned_claude_backend(region: str | None, expected: list[str]): + env = {LL_URL: "https://gw.corp/v1"} + targets = _targets(backend=LITELLM, region=region, env=env, criteria=[JUDGE], simulation=True) + assert targets == sorted(["gw.corp:443", *expected]) + + +@pytest.mark.parametrize( + ("route", "env", "host"), + [ + ({"params": {"api_base": "https://judge.example/v1"}}, {}, "judge.example:443"), + ({"env_params": {"api_base": "JUDGE_BASE"}}, {"JUDGE_BASE": "https://j2.example"}, "j2.example:443"), + ], +) +def test_litellm_judge_route_uses_its_own_api_base(route: dict[str, Any], env: dict[str, str], host: str): + judge_route = ApiRouteContext(route=LITELLM, model="azure/x", **route) + targets = _targets(CODEX, env=env, criteria=[JUDGE], route=judge_route, env_passthrough_extra=["JUDGE_BASE"]) + assert targets == [OPENAI, host] + + +def test_every_registered_agent_config_answers_with_normalized_targets(): + ensure_plugins_loaded() + registrations = AgentRegistry.registrations() + assert registrations + for registration in registrations: + hosts = registration.config_class.model_construct().egress_hosts({}) + assert isinstance(hosts, tuple), registration.config_class.__name__ + assert all(normalize_egress_target(host) == host for host in hosts), registration.config_class.__name__ + + +def test_docker_run_error_is_one_class_in_both_modules(): + assert docker_runner.DockerRunError is errors.DockerRunError + assert issubclass(errors.EgressSetupError, errors.DockerRunError) diff --git a/tests/test_harbor_packager.py b/tests/test_harbor_packager.py index cb383aca4..553b8062f 100644 --- a/tests/test_harbor_packager.py +++ b/tests/test_harbor_packager.py @@ -55,6 +55,12 @@ def test_tempdir_driver_is_refused(self, tmp_path: Path) -> None: with pytest.raises(TaskNotExportableError): export_task(task_file, tmp_path / "out") + def test_llm_only_network_is_refused_not_widened_to_public(self, tmp_path: Path) -> None: + task_file = _write_task(tmp_path, {"sandbox": {"driver": "docker", "docker": {"network": "llm_only"}}}) + with pytest.raises(TaskNotExportableError, match="llm_only"): + export_task(task_file, tmp_path / "out") + assert not (tmp_path / "out").exists() + def test_unsupported_criteria_are_refused_before_any_file_is_written(self, tmp_path: Path) -> None: task_file = _write_task( tmp_path, diff --git a/tests/test_merge_characterization.py b/tests/test_merge_characterization.py index 74ac89249..9ba4d1633 100644 --- a/tests/test_merge_characterization.py +++ b/tests/test_merge_characterization.py @@ -167,6 +167,28 @@ def test_env_passthrough_extra_appends(self): # exp-defaults appended first, then task. assert resolved.sandbox.docker.env_passthrough_extra == ["EXP_VAR", "TASK_VAR"] + def test_egress_allowlist_appends_exp_defaults_then_task(self): + default_exp = _default_exp() + task = _make_task( + agent={"type": "claude-code"}, + sandbox=SandboxConfig(driver="docker", docker=DockerDriverConfig(egress_allowlist=["task.example.com"])), + ) + experiment = ExperimentDefinition( + experiment_id="test", + defaults=ExperimentDefaults( + sandbox=SandboxConfig(docker=DockerDriverConfig(egress_allowlist=["pypi.org"])) + ), + variants=[ExperimentVariant(variant_id="v")], + ) + resolved, _lineage, _ = resolve_task_for_variant(default_exp, task, experiment, experiment.variants[0]) + assert resolved.sandbox.docker.egress_allowlist == ["pypi.org:443", "task.example.com:443"] + apply_overrides(resolved, {"sandbox.docker.egress_allowlist": ["cli.example.com:80"]}) + assert resolved.sandbox.docker.egress_allowlist == [ + "pypi.org:443", + "task.example.com:443", + "cli.example.com:80", + ] + def test_template_sources_task_first_order(self): """template_sources resolve task-first: the task's base templates, then experiment-defaults and variant overlays appended after (the documented diff --git a/uv.lock b/uv.lock index f63fadc6d..eed5dc6dd 100644 --- a/uv.lock +++ b/uv.lock @@ -911,11 +911,11 @@ wheels = [ [[package]] name = "filelock" -version = "3.32.6" +version = "4.2.4" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/38/46/126b1831dca12060d4a8296bf9c4fe5c93c4f22197fa239cb0cc82042bba/filelock-3.32.6.tar.gz", hash = "sha256:a3f55a18af3652a94d8f47d6055df434f254ca1d02ef2524850c6d249ca2512c", size = 225172, upload-time = "2026-09-08T22:57:11.528Z" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/28/3893324bcadf8f89b1616637468d766af3f625008c0735bf704fdfa86617/filelock-4.2.4.tar.gz", hash = "sha256:c6d8663535e1650ee430507b4ffbac7b464dd64b7647ea0112a9f406499d163d", size = 652444, upload-time = "2026-10-19T01:57:51.001Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/cc/06/4f138f618dbea66803291274f228f01daf29f306fe8b96bc30dab765df75/filelock-3.32.6-py3-none-any.whl", hash = "sha256:3f16ecd0117feae0dfc147e8c62eb5daeccd8bd800378c3ddf416de9b4feb6b1", size = 100189, upload-time = "2026-09-08T22:57:10.182Z" }, + { url = "https://files.pythonhosted.org/packages/e0/1b/5e74289a1878e666bb4b65fe5805a04cafcac2aa5f519ef1ead67ef91c56/filelock-4.2.4-py3-none-any.whl", hash = "sha256:301241438c57936ef93f4243a94f2559022268bfcf9b375a9a72eb0f8db8358e", size = 162475, upload-time = "2026-10-19T01:57:49.419Z" }, ] [[package]] @@ -3336,7 +3336,7 @@ wheels = [ [[package]] name = "virtualenv" -version = "21.14.12" +version = "21.14.13" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "distlib" }, @@ -3345,9 +3345,9 @@ dependencies = [ { name = "platformdirs" }, { name = "python-discovery" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/c9/2c/1a69b3346931e9a7db9cc00cc0fa847c843a85fa7cf52a598f57261c68df/virtualenv-21.14.12.tar.gz", hash = "sha256:bbcfb78e36c22ad70c6917c4aecb5e63477d3c138e9f88c946f320e31859c76a", size = 5562800, upload-time = "2026-10-21T01:58:51.724Z" } +sdist = { url = "https://files.pythonhosted.org/packages/70/aa/bff50daee0488e16d611258a9b6c5a1d0c9e128f8ca424e9f4e1c0444ef2/virtualenv-21.14.13.tar.gz", hash = "sha256:de5307ba4b9a0001d9c8ea02832d38f63ca432880f416c1bface61c4323fd905", size = 5570301, upload-time = "2026-10-22T20:10:16.952Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a0/92/f3b8af2d36ddec4dc61dc25655dc010e419ff456cb4ca0e720c8112f1047/virtualenv-21.14.12-py3-none-any.whl", hash = "sha256:7820aeaef23f1f59a96458edfc7f9d3deac937155718de15f1c9a64471e15551", size = 5556824, upload-time = "2026-10-21T01:58:48.505Z" }, + { url = "https://files.pythonhosted.org/packages/bb/a9/c10d8a44a29fb94a18b43fdf11f38caca53ae7c9406f11a72b1cb8f624f1/virtualenv-21.14.13-py3-none-any.whl", hash = "sha256:5fbb04138a4861580b78ba9110d529a26bc48fcc93c8ec36bc0202eeed96a788", size = 5557094, upload-time = "2026-10-22T20:10:15.096Z" }, ] [[package]]