CodingBench

CodingBench evaluates coding agents on repo-scale, from-scratch tasks.

The agent receives a product spec and an empty workspace, implements the project in a real shell environment, and is scored by hidden, deterministic acceptance tests (binary reward: 1.0 / 0.0). No LLM-as-judge.

This repository ships:

• 26 released cases under benchmark/cases/
• The cbrun harness (recommended)
• An optional Harbor adapter

---

Quick start

Requirements

• Python 3.10+
• Docker (for full agent trials)
• An agent CLI API key (Codex / OpenCode / Claude Code)

Install

git clone https://github.com/OpenEdgeHQ/CodingBench.git
cd CodingBench

# vendor dependency used by cost reporting
git submodule update --init --recursive 2>/dev/null || true

cd benchmark
pip install -e .

Configure an agent

cd benchmark/local_agents
cp codex.env.example codex.env          # set OPENAI_API_KEY / MODEL
# or: cp opencode.env.example opencode.env
# or: cp claude-code.env.example claude-code.env

Smoke-test the container wiring

cd benchmark
./local_agents/run_smoke.sh codex

Run one case

cd benchmark
set -a && source local_agents/codex.env && set +a
cbrun --case case010 --backend codex --model "$MODEL"

Results land under benchmark/output/ (reward, logs, trajectories).

---

What the agent sees

During the solve phase the container looks like:

/environment/prd/Full_PRD.md
/environment/Interface_Contract.md
/environment/Hardware_Requirements.md   # when present
/app/                                   # empty workspace — implement here

Hidden acceptance tests are not present while the agent works. After the agent exits, the harness injects them and runs the judge.

Scoring is black-box: only the hidden suite decides the reward.

---

Case layout

Each case under benchmark/cases/<case_id>/ contains:

Path	Role
public/Full_PRD.md	Product requirements (visible)
public/Interface_Contract.md	Public API / CLI contract (visible)
public/Hardware_Requirements.md	Optional hardware constraints (visible)
source/manifest.json	Runner metadata for the harness
milestones/final/tests/	Hidden acceptance tests
milestones/final/test_manifest.json	Test inventory / command template
milestones/final/run_acceptance.sh	Authoritative judge entrypoint

List released cases:

ls benchmark/cases

---

Harbor adapter (optional)

cd benchmark
pip install -e ".[harbor]"

# Build Harbor tasks for the released suite
coding-bench-harbor --all --force

# Example agent run (edit model/env in the YAML first)
harbor run --config configs/agents/codex.yaml

---

Docs

Doc	Contents
benchmark/README.md	Harness details, Harbor mapping, offline judge
benchmark/cbrun/README.md	cbrun flags, images, limits
benchmark/local_agents/README.md	Agent env templates and smoke tests
doc/CodingBench.md	Product contract: assets, visibility, scoring

---

License

See repository license terms. Bring your own model provider credentials; this repo does not bundle API keys or third-party proxies.