bundlebench: Task Bundle CLI

A portable command-line harness that evaluates an LLM's attempt to fix real code. It packages a coding task into a self-contained bundle, proves the task's baseline, runs a solver inside an isolated container, and grades the attempt by how its tests transition. Every invocation is recorded in a local audit database, queryable by run id.

The problem

Engineers who author SWE-bench style coding tasks need to know how a model performs on their task. Each task pins a repository to a known commit, describes the fix in plain prose, and ships the tests that decide whether an attempt is correct. Measuring a model against that by hand does not scale: every repository pulls in a different dependency stack, the scoring tests have to stay hidden from the model until it finishes, and a teammate who later asks what happened on a run should be able to find out without rerunning it.

Correctness here is defined by how tests transition against a baseline. Tests declared fail-to-pass must fail before the fix and pass after it. Tests declared pass-to-pass must pass on both sides. That mechanic drives three requirements for an honest harness:

1. Prove the baseline, never assume it. If a fail-to-pass test already passes before any solver runs, the task itself is broken. The harness must report that, not silently accept it.
2. Make isolation structural. A solver with shell access will find hidden tests, or the answer itself, if either is reachable from inside its container. Hiding by instruction is not hiding. The graded tests and the fix must be absent from the solver's world by construction.
3. Make every run reproducible and auditable. Pin the image digest, record the resolved configuration, capture the artifacts, and let anyone query the outcome later by a single id.

So I built task: one CLI that materializes a task bundle into a container, validates the baseline, runs a solver, grades the transitions, and writes the whole story to disk.

What it delivers

Each of these requirements maps to a concrete piece of the tool:

• A self-contained task format. A bundle is one directory: task.json (a strict, schema-validated manifest), description.md (the only task text the solver sees), patch.diff (the golden fix), and the declared fail-to-pass and pass-to-pass test ids. A bundle plus its image is everything a run needs.
• task init materializes the bundle's container image, either a prebuilt registry reference or a local Dockerfile build, and runs a build-verification smoke check. With --from-dataset it ingests an entry from SWE-bench Pro and writes a ready-to-run bundle.
• task validate executes the baseline before any solver touches the code and asserts both halves explicitly: every pass-to-pass test passes, every fail-to-pass test fails. --with-gold additionally proves the golden patch resolves the task, so a bundle is known solvable before a model ever sees it.
• task run lets a solver attempt the fix, evaluates the attempt, and reports per-test status in four buckets: fail_to_pass, pass_to_pass, fail_to_fail, pass_to_fail. The verdict is strict: a run is resolved only if every declared test lands in the first two buckets. There is no partial credit.
• Hidden-test isolation, enforced by architecture. task run uses two containers. The solve container holds a sanitized repository: git history is cut at the base commit and the graded tests are absent entirely. The solver produces a patch there. A second, fresh eval container then applies that patch, injects the hidden tests, and runs the suite. The solver's process never executes inside a container that could reveal the answer.
• Pluggable solvers. --solver null | golden | sabotage | agent. The three stubs are deterministic oracles that need no API key: null proves an empty attempt stays unresolved, golden proves the bundle is solvable, sabotage proves grading catches regressions. The agent solver is a Pydantic AI agent with three tools (bash, read_file, edit_file) that works inside the sandbox like an engineer would.
• A lightweight audit database. Every CLI invocation writes a row to a WAL-mode SQLite store. Each run also persists its pinned image digest, resolved config, per-test results, a phase timeline, and artifacts under runs/<run_id>/ (solver.diff, eval.log, report.json, a manifest of what was withheld from the solver). task history lists runs; task show <run_id> replays one in full. report.json is the structured evaluation artifact.
• Readable and scriptable output. Human output is a compact table. Every subcommand takes --json for a single machine-readable payload. Exit codes carry meaning: 0 for success or resolved, 1 for a contract not met (an unresolved run, an unknown run id), 2 for an error such as a broken bundle. Expected failures print a one-line cause, never a traceback.
• A runnable example. bundles/example-mini is a hand-authored micro task (a forum store that accepts blank post titles) that exercises the entire pipeline against real Docker in seconds.

Getting started

You need Python 3.11+, a running Docker daemon, and uv.

uv sync --all-extras

Prove the whole pipeline on the bundled example:

uv run task init bundles/example-mini            # build the image, smoke-check it
uv run task validate bundles/example-mini --with-gold
uv run task run bundles/example-mini --solver golden    # resolved, exit 0
uv run task run bundles/example-mini --solver null      # unresolved, exit 1
uv run task history                              # list runs, newest first
uv run task show <run_id>                        # full detail for one run

Then run against any SWE-bench Pro entry of your choosing (--offset picks the dataset row; images are amd64 and can be large):

uv run task init --from-dataset --offset 0 --dest bundles/swb-pro-0
uv run task validate bundles/swb-pro-0
uv run task run bundles/swb-pro-0 --solver golden

--from-dataset fetches that row over the Hugging Face datasets-server HTTP API, so this step needs no local dataset copy. If you would rather download the full dataset first, see datasets/README.md for the optional step.

The stub solvers need no credentials. The agent solver and tracing read their secrets from the environment (OPENAI_BASE_URL, OPENAI_API_KEY, and the Langfuse keys). I manage those with ssm, a self-hosted secrets manager whose CLI injects a project's secret set into any command. This repository already pins the right project and config in .ssm/config.json:

uv tool install git+https://github.com/bearlike/Simple-Secrets-Manager.git
ssm login
ssm run -- uv run task run bundles/swb-pro-0 --solver agent

Without ssm, copy the template, fill in your values, source it, and run the tool:

cp .env.example .env
set -a && source .env && set +a
uv run task run bundles/swb-pro-0 --solver agent

Every variable is documented inline in .env.example.

Architecture

The package is a set of single-purpose components around a typed kernel, and the data flows one way. core owns the shared vocabulary: the domain value objects plus two ports, Sandbox and Solver, defined as structural protocols. bundle owns the on-disk task format and validates it at load. dataset ingests a SWE-bench Pro row and normalizes its string fields into typed models before anything else touches them. sandbox is the only component that talks to Docker: it provisions disposable, resource-limited, non-root containers with an explicit network policy, and runs commands in them over docker exec. solver produces a candidate patch behind the Solver port. extractors normalize raw test output (JUnit XML, a bundle-supplied parser script, or a regex fallback) into one uniform per-test status list. grading is pure logic: it buckets observed results against the declared sets and computes the strict verdict. persistence writes the SQLite audit rows and the runs/<run_id>/ artifacts. observability is a best-effort tracing layer that no-ops without credentials.

engine is the orchestration layer: the one component that sequences a phase (create container, sanitize git state, apply a patch through a fallback chain, inject hidden tests, run the suite, extract, grade) and the only one that composes the others. cli is the composition root: it parses input, builds the concrete adapters, and injects them into one Engine per invocation through the core ports. Every seam between components is a strict Pydantic model, so a malformed payload fails at construction rather than three calls later. The import direction is enforced in CI by import-linter, not by convention.

flowchart LR
    cli["cli (composition root)"] --> engine["engine (orchestrator)"]
    engine --> solve["solve container: sanitized repo, graded tests absent"]
    solve -- "captured patch" --> eval["eval container: apply patch, inject hidden tests, run suite"]
    eval -- "raw test output" --> extractors
    extractors -- "per-test status" --> grading
    grading -- "report" --> store[("SQLite audit store and run artifacts")]
    engine -. "best-effort traces" .-> langfuse[("Langfuse")]

Loading

The two-container split in the middle is the load-bearing decision. The solve container starts from the bundle's image with git history truncated at the base commit and the graded tests removed, and it defaults to --network=none, so the answer is unreachable rather than merely off-limits. The patch is captured as a plain git diff inside that container. The eval container is created fresh only after the patch exists. It applies the patch, overlays the hidden tests after the patch so the attempt cannot rewrite what grades it, and runs the test command. Determinism comes from pinned image digests, a pinned timezone, hash seed, and locale inside every container, and solver temperature=0. All of it is recorded per run, so a past run is reproducible from its database row alone.

Observability and LLM-as-judge

Traces and structured logs are intentionally collected into a self-hosted Langfuse server; ssm supplies the endpoint and keys. Each trace shares its run id with the SQLite row and the runs/<run_id>/ artifacts, so the solver's full trajectory (every tool call, the captured patch, the graded outcome) joins up with the audit record. This is what enables automated LLM-as-judge scoring workflows: a judge model reads a completed evaluation's trajectory and attaches scores back onto the same trace. Tracing is strictly best-effort. Without credentials every observability call is a no-op and the CLI behaves identically.

Development

make install   # uv sync --all-extras + pre-commit hooks
make check     # ruff lint + mypy strict + import-linter + pytest

The test suite is hermetic: orchestration is covered with in-memory fakes for the Sandbox and Solver ports, the agent solver is driven by scripted fake models, and no test needs an API key. Integration tests drive bundles/example-mini against real Docker.

Further reference lives in documentation/: the design notes on the key tradeoffs, the CLI reference, and the bundle format.