This document is the participant-facing contract: what each case contains, what the agent may see, and how scoring works.
CodingBench is a repo-scale, from-scratch coding-agent benchmark. The agent
works in a real shell sandbox with only the public specification, implements the
project under /app, and receives a deterministic binary reward from hidden
acceptance tests.
This repository is the evaluation package: released cases + harness. It does not include case-authoring tooling or internal production notes.
| Piece | Role |
|---|---|
| Public specs | Full_PRD.md, Interface_Contract.md, optional Hardware_Requirements.md |
| Hidden acceptance | milestones/final/tests/, test_manifest.json, run_acceptance.sh |
| Runner metadata | source/manifest.json (harness-only) |
| Harness | cbrun and optional Harbor adapter |
| Shared judge | final_judge.py |
Upstream source snapshots, reference implementations, authoring trajectories, environment build recipes, and internal QA artifacts are out of scope for this package and must not appear in the agent context.
/environment/prd/Full_PRD.md
/environment/Interface_Contract.md
/environment/Hardware_Requirements.md # if present
/app/ # empty; agent implements here
Hidden tests are absent during solve. The agent may write and run its own tests; those do not affect the final score.
After the agent exits (or times out), the harness injects hidden acceptance and
runs the judge. Only the hidden suite determines reward.
| Outcome | Reward |
|---|---|
| Hidden acceptance passes | 1.0 |
| Otherwise | 0.0 |
No LLM-as-judge. Prefer run_acceptance.sh when present; otherwise run the
command from test_manifest.json.
Public PRD / Contract must not reveal:
- Upstream project identity or repository URLs
- Hidden-test assertion details
- Implementation hints that short-circuit the task
The harness may scan public text against sensitive_terms in
source/manifest.json and refuse to build a task on hits.
See the root README.md and benchmark/README.md.
Released suite size: 26 cases under benchmark/cases/.