ARC-Bench

arc-bench is a benchmark for requirement-to-application generation. It evaluates whether a generation system can transform multi-modal web application requirements into a runnable implementation whose behavior is validated by end-to-end Playwright tests.

The benchmark is organized as a set of web application tasks. Each task pairs a requirement package with an executable test suite, so different generators can be compared against the same inputs and behavioral checks. This repository contains:

arc-bench/webapp/<app>/requirements/: structured requirements for the benchmark web apps;
arc-bench/webapp/<app>/tests/: Playwright tests for those apps;
arc-bench/webapp/<app>/project/: optional reference implementation for an app;
scripts/run-playwright.js and playwright.config.ts: the benchmark test runner;
Dockerfile: an optional containerized benchmark execution environment.

The benchmark itself is generator-agnostic. Any implementation, whether it is produced by a generator or written as a reference implementation, is responsible for starting successfully, initializing the data required by the requirements, and exposing one entry URL. The benchmark runner is responsible only for running the selected app's tests against that URL.

📊 Benchmark Applications

Requirement counts are the number of atomic requirement nodes in requirements.yaml. Test counts are the number of Playwright test(...) cases.

App	# Requirements	# Test cases	# Domain
`keep`	32	32	Google Keep, https://keep.google.com/
`bookstack`	34	34	BookStack, https://demo.bookstackapp.com/
`stackoverflow`	66	66	Stack Overflow, https://stackoverflow.com/
`prestashop`	86	86	PrestaShop, https://demo.prestashop.com/
`12306`	117	117	China Railway 12306, https://www.12306.cn/en
`ctrip`	125	125	Ctrip, https://www.ctrip.com/

🚀 Benchmark Basic Usage

The benchmark usage is independent of any particular generation method:

arc-bench/webapp/<app>/requirements/
  -> generate a runnable web application with a chosen method
  -> the implementation starts, initializes required data, and exposes an entry URL
  -> run arc-bench/webapp/<app>/tests/ against that URL

Install the local test runner when running tests directly on the host:

npm install
npm run test:install

Run one benchmark application's tests against a running application:

npm run test -- --app bookstack --target-url http://127.0.0.1:3301

If the application is already deployed and you only have an entry URL, pass the URL with --target-url. The runner uses that URL as Playwright's baseURL. No environment variables are required for this path.

npm run test -- --app bookstack --target-url https://your-app.example.com

To run the same tests inside the Docker benchmark environment, use:

npm run test:docker -- --app bookstack --target-url https://your-app.example.com

When the application is running on the host machine, a localhost URL must be reachable from inside Docker. The npm wrapper rewrites localhost, 127.0.0.1, and 0.0.0.0 to host.docker.internal for the container:

npm run test:docker -- --app bookstack --target-url http://127.0.0.1:3301

Docker test outputs are exported to:

docker-output/test/<app>/
|-- test-results/
`-- playwright-report/

The Docker image in this repository provides a benchmark execution environment: Node.js, Playwright browsers, the benchmark runner, and benchmark files. For an already-running implementation, the Docker test command only needs the selected app and the entry URL.

🧪 Reference Implementation Testing

Reference implementations can be placed under:

arc-bench/webapp/<app>/project/

The reference app must listen on PORT and expose a health endpoint at /api/health. It must initialize the database or other seed data required by the requirements during startup. It does not own or execute the benchmark E2E tests.

Run Reference Implementation

Build or rebuild the benchmark image after changing Docker scripts or the test runner:

npm run docker:build

The image prepares the benchmark test environment and the Playwright Chromium browser cache used by the benchmark runner. It does not include reference implementation source code or reference node_modules.

If a run reports a missing path such as /ms-playwright/chromium_headless_shell-xxxx/..., rebuild the image. That error means the image's browser cache does not match the benchmark runner's @playwright/test version.

Run the 12306 reference implementation in Docker, then execute the benchmark tests against the URL exposed inside the container:

npm run reference -- --app 12306

Equivalent shorthand:

npm run reference:12306

This command does not require manually setting PORT, TARGET_URL, PLAYWRIGHT_BASE_URL, or ARC_TEST_DATE. The wrapper chooses the container port, starts the reference implementation, and calls the benchmark runner with the resulting URL.

The reference flow performs the following steps in one fresh container:

mount arc-bench/webapp/12306/project/
mount arc-bench/webapp/12306/tests/
  -> copy source to /workspaces/reference/12306/project
  -> install dependencies inside the Linux container
  -> build the frontend if package.json defines `build`
  -> start the reference backend on PORT=3301
  -> wait for http://127.0.0.1:3301/api/health
  -> run `npm run test -- --app 12306 --target-url http://127.0.0.1:3301`

🧩 ARC Baseline Reproduction Flow

This section is an application example of the benchmark using ARC (Agentic Requirement Compiler) as the generation method. This repository is not a standalone implementation of the ARC compiler; the compiler and its application templates are provided through Git submodules.

ARC Agent: https://github.com/code-philia/agentic-requirement-compiler

Prerequisites

Install the following for the ARC baseline example:

Docker Desktop on Windows/macOS or Docker Engine on Linux;
access to an OpenAI-compatible model API.

Clone the Repository and Update Submodules

Clone the benchmark repository and initialize the ARC compiler submodule and its nested submodules:

git submodule sync --recursive
git submodule update --init --remote --recursive

Configure ARC Environment

The environment file belongs to the ARC compiler submodule. Read the configuration instructions in:

agentic-requirement-compiler/README.md

Create the compiler environment file from the template.

Linux/macOS:

cp agentic-requirement-compiler/.env_example \
   agentic-requirement-compiler/.env

Windows PowerShell:

Copy-Item `
  agentic-requirement-compiler\.env_example `
  agentic-requirement-compiler\.env

Edit agentic-requirement-compiler/.env according to the ARC compiler README. At minimum, configure the model API credentials and model name. The file is passed to Docker at runtime and is excluded from the Docker image.

For one application, the complete flow is:

arc-bench/webapp/<app>/
|-- requirements/   input requirements and reference assets
`-- tests/          Playwright tests for the generated app

ARC compiles `arc-bench/webapp/<app>/requirements/`
  -> generated backend starts on port 3301
  -> /api/health becomes available
  -> Playwright tests run from `arc-bench/webapp/<app>/tests/`
  -> application, logs, raw results, and HTML report are exported

The recommended ARC baseline command performs all steps in one isolated container. The examples below use bookstack; replace it with another benchmark app name as needed.

If you add or modify arc-bench/, apps.config.json, scripts/, or docker/entrypoint.sh, rebuild the image before running the container again. The image copies those files at build time.

Linux:

mkdir -p docker-output

docker run --rm \
  --env-file agentic-requirement-compiler/.env \
  --mount "type=bind,source=$PWD/docker-output,target=/export" \
  arc-reproduction:latest bookstack

macOS:

mkdir -p docker-output

docker run --rm \
  --env-file agentic-requirement-compiler/.env \
  --mount "type=bind,source=$PWD/docker-output,target=/export" \
  arc-reproduction:latest bookstack

Windows PowerShell:

New-Item -ItemType Directory -Force docker-output | Out-Null

docker run --rm `
  --env-file agentic-requirement-compiler\.env `
  --mount "type=bind,source=$((Get-Location).Path)\docker-output,target=/export" `
  arc-reproduction:latest bookstack

Replace bookstack with one of:

keep
bookstack
stackoverflow
prestashop
12306
ctrip

The container entrypoint performs:

arc compile
  -> PORT=3301 npm run start
  -> wait for http://127.0.0.1:3301/api/health
  -> npm run test -- --app <app-name>

The container exits with:

0 when compilation, startup, and tests succeed;
a non-zero status when compilation fails, the health check fails, or tests fail.

Step-by-Step Commands

The one-container command above is recommended for complete reproduction. The following commands are useful when debugging or running one stage separately.

Step 1: Build the Docker Image

Linux/macOS:

docker build --progress=plain -t arc-reproduction:latest .

Windows PowerShell:

docker build --progress=plain -t arc-reproduction:latest .

Parameter meanings:

docker build: builds an image from the Dockerfile;
--progress=plain: prints complete build logs;
-t arc-reproduction:latest: assigns the image name and tag;
.: uses the current repository as the Docker build context.

Step 2: Compile an Application Only

This command runs ARC compilation without starting the generated application or running Playwright.

Linux/macOS:

docker run --rm \
  --env-file agentic-requirement-compiler/.env \
  --mount "type=bind,source=$PWD/docker-output,target=/export" \
  --entrypoint arc \
  arc-reproduction:latest \
  compile /opt/arc/arc-bench/webapp/bookstack/requirements \
  -o /export/bookstack/application \
  --type web \
  --clean

Windows PowerShell:

docker run --rm `
  --env-file agentic-requirement-compiler\.env `
  --mount "type=bind,source=$((Get-Location).Path)\docker-output,target=/export" `
  --entrypoint arc `
  arc-reproduction:latest `
  compile /opt/arc/arc-bench/webapp/bookstack/requirements `
  -o /export/bookstack/application `
  --type web `
  --clean

Parameter meanings:

--entrypoint arc: bypasses the default Docker entrypoint and calls the installed ARC CLI directly;
compile: compiles a requirement directory into an application;
/opt/arc/arc-bench/webapp/bookstack/requirements: the requirement directory inside the image;
-o /export/bookstack/application: the generated application output directory;
--type web: selects web application generation;
--clean: removes an existing output directory before compilation;
--mount ...:/export: persists the container output under docker-output/bookstack/application on the host.

Step 3: Start an Existing Application and Run Its Tests

The default entrypoint already starts and tests a newly generated application. To test an existing generated application, mount it into a fresh container, start its backend, wait for the health endpoint, and run Playwright.

Linux/macOS:

docker run --rm \
  --mount "type=bind,source=$PWD/docker-output/bookstack/application,target=/workspaces/bookstack" \
  --entrypoint /bin/bash \
  arc-reproduction:latest \
  -lc 'set -e
       (cd /workspaces/bookstack/backend && PORT=3301 npm run start >/tmp/arc-app.log 2>&1) &
       server_pid=$!
       trap "kill $server_pid 2>/dev/null || true" EXIT
       until curl --fail --silent http://127.0.0.1:3301/api/health >/dev/null; do sleep 1; done
       cd /opt/arc
       TARGET_URL=http://127.0.0.1:3301 npm run test -- --app bookstack'

Windows PowerShell:

docker run --rm `
  --mount "type=bind,source=$((Get-Location).Path)\docker-output\bookstack\application,target=/workspaces/bookstack" `
  --entrypoint /bin/bash `
  arc-reproduction:latest `
  -lc 'set -e; (cd /workspaces/bookstack/backend && PORT=3301 npm run start >/tmp/arc-app.log 2>&1) & server_pid=$!; trap "kill $server_pid 2>/dev/null || true" EXIT; until curl --fail --silent http://127.0.0.1:3301/api/health >/dev/null; do sleep 1; done; cd /opt/arc; TARGET_URL=http://127.0.0.1:3301 npm run test -- --app bookstack'

This form assumes the generated application already contains its dependencies and frontend build output. Otherwise, use the complete one-container command.

Step 4: Run All Applications

Linux/macOS:

for app in keep bookstack stackoverflow prestashop 12306 ctrip; do
  docker run --rm \
    --env-file agentic-requirement-compiler/.env \
    --mount "type=bind,source=$PWD/docker-output,target=/export" \
    arc-reproduction:latest "$app"
done

Windows PowerShell:

$apps = @("keep", "bookstack", "stackoverflow", "prestashop", "12306", "ctrip")
New-Item -ItemType Directory -Force docker-output | Out-Null

foreach ($app in $apps) {
  docker run --rm `
    --env-file agentic-requirement-compiler\.env `
    --mount "type=bind,source=$((Get-Location).Path)\docker-output,target=/export" `
    arc-reproduction:latest $app
}

Each application runs in its own container and does not reuse another application's backend process or workspace.

Outputs and Test Runner

Results are written to:

docker-output/<app>/
|-- application/       generated ARC application
|-- logs/
|   |-- compile.log    ARC compilation log
|   |-- app.log        application startup log
|   `-- test.log       Playwright log
|-- test-results/      screenshots, videos, traces, and raw results
|-- playwright-report/ Playwright HTML report
`-- summary.txt        compilation, runtime, and test statuses

Open the HTML report at:

docker-output/bookstack/playwright-report/index.html

Navigation Menu

Uh oh!

Repository files navigation

ARC-Bench

📊 Benchmark Applications

🚀 Benchmark Basic Usage

🧪 Reference Implementation Testing

Run Reference Implementation

🧩 ARC Baseline Reproduction Flow

Prerequisites

Clone the Repository and Update Submodules

Configure ARC Environment

Step-by-Step Commands

Step 1: Build the Docker Image

Step 2: Compile an Application Only

Step 3: Start an Existing Application and Run Its Tests

Step 4: Run All Applications

Outputs and Test Runner

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
agentic-requirement-compiler @ 2feea34		agentic-requirement-compiler @ 2feea34
arc-bench/webapp		arc-bench/webapp
docker		docker
scripts		scripts
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitmodules		.gitmodules
Dockerfile		Dockerfile
README.md		README.md
apps.config.json		apps.config.json
package-lock.json		package-lock.json
package.json		package.json
playwright.config.ts		playwright.config.ts

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

ARC-Bench

📊 Benchmark Applications

🚀 Benchmark Basic Usage

🧪 Reference Implementation Testing

Run Reference Implementation

🧩 ARC Baseline Reproduction Flow

Prerequisites

Clone the Repository and Update Submodules

Configure ARC Environment

Step-by-Step Commands

Step 1: Build the Docker Image

Step 2: Compile an Application Only

Step 3: Start an Existing Application and Run Its Tests

Step 4: Run All Applications

Outputs and Test Runner

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages