feat(evals): port HardBenchmark suite onto the harness wave

[miguelg719]

Stacked on #2812 (top of the harness wave). Part of the harness×model campaign.

What

Ports the 46-task HardBenchmark suite — the residual hard core of WebTailBench + Online-Mind2Web (tasks both frontier models failed, audited to genuine model faults, date-rotted tasks dropped) — onto the wave head.

• datasets/hardbenchmark/HardBenchmark_data.jsonl (verbatim precomputed_rubric per row + audited provenance: source_suite, failure_mode, capability_axis)
• suites/hardbenchmark.ts builder; registered as agent/hardbenchmark in discovery, planner, external-harness plan, TUI (b:hardbenchmark), config
• Provenance fields added to testcase metadata types so regressions attribute to a capability, not just a task

Notes

• Rubric-carrying: no LLM-generated rubric divergence between harnesses.
• The validity audit (quarantine flags, review list) lands in the follow-up PR on feat/facade-batch-surface.

---

Summary by cubic

Adds the 46-task HardBenchmark residual suite to the eval harness to focus model evaluation on audited failures from WebTailBench and Online-Mind2Web. Unlike the parent suites, it runs all 46 cases by default and preserves each task’s precomputed rubric.

Included

• Registers agent/hardbenchmark across discovery, planning, external harnesses, config, and the b:hardbenchmark TUI shorthand.
• Supports task ID selection, sampling, failure-mode filtering, and suite-specific limits.
• Adds source suite, failure mode, and capability axis to testcase metadata for regression analysis.

^{Written for commit 2c93dcf. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2c93dcf

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

4 issues found across 10 files

Confidence score: 3/5

• packages/evals/suites/hardbenchmark.ts labels all rows as agent/hardbenchmark, dropping OnlineMind2Web site-scoping metadata; external harnesses may apply generic guidance and run those tasks incorrectly. Preserve the dataset-specific contract in the generated rows.
• packages/evals/evals.config.json exposes benchmarks.hardbenchmark.limit, but the runner does not consume it, so configured limits have no effect. Wire the value into suite limit resolution or remove the setting.
• packages/evals/framework/benchPlanner.ts has no focused coverage for the new agent/hardbenchmark route, dataset routing, or generated testcase shape, leaving regressions able to disable the suite silently. Add planner tests for this route.
• packages/evals/framework/externalHarnessPlan.ts omits agent/hardbenchmark from the unsupported-task fallback, producing incomplete diagnostics when that suite is requested. Update the fallback message.

Not reviewed (too large): packages/evals/datasets/hardbenchmark/HardBenchmark_data.jsonl (~46 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/evals.config.json">

<violation number="1" location="packages/evals/evals.config.json:22">
P2: `benchmarks.hardbenchmark.limit` is not consumed by the runner, so this setting cannot control the suite and creates a false configuration surface. Wire the config value into the suite's limit resolution, or remove this entry until benchmark limits are read from config.</violation>
</file>

<file name="packages/evals/framework/benchPlanner.ts">

<violation number="1" location="packages/evals/framework/benchPlanner.ts:303">
P2: Add a focused planner test for `agent/hardbenchmark`; this new route currently has no coverage for dataset routing or generated testcase shape, so regressions can silently disable the suite.

(Based on your team's feedback about unit tests for new behavior.)</violation>
</file>

<file name="packages/evals/framework/externalHarnessPlan.ts">

<violation number="1" location="packages/evals/framework/externalHarnessPlan.ts:51">
P3: When an unsupported external task is requested, the error now omits the newly supported `agent/hardbenchmark` suite. Add `agent/hardbenchmark` to the fallback message so the diagnostic matches this branch.</violation>
</file>

<file name="packages/evals/suites/hardbenchmark.ts">

<violation number="1" location="packages/evals/suites/hardbenchmark.ts:116">
P2: OnlineMind2Web tasks lose their site-scoping contract when run through this suite. Because every row is labeled `agent/hardbenchmark`, external harnesses use generic guidance and cannot distinguish the 13 OnlineMind2Web rows; preserve the source suite in the plan/input or add source-aware guidance, and retain the source URL when it is available.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant TUI as TUI/CLI
    participant Discovery as Discovery
    participant Planner as Bench Planner
    participant Builder as HardBenchmark Builder
    participant Dataset as HardBenchmark_data.jsonl
    participant Config as Config/TaskConfig
    participant Harness as External Harness Plan
    participant Eval as Eval Runner

    Note over TUI,Eval: HardBenchmark Suite Registration Flow

    TUI->>Discovery: b:hardbenchmark shorthand
    Discovery->>Discovery: Register agent/hardbenchmark as external_agent_benchmark
    Discovery-->>TUI: Suite discovered

    TUI->>Planner: Run agent/hardbenchmark
    Planner->>Planner: Generate suite testcases

    Planner->>Builder: buildHardBenchmarkTestcases(models)
    Builder->>Dataset: Read HardBenchmark_data.jsonl
    Dataset-->>Builder: 46 rows with precomputed_rubric + provenance

    alt EVAL_HARDBENCHMARK_IDS set
        Builder->>Builder: Select tasks by explicit IDs
    else EVAL_HARDBENCHMARK_MODE set
        Builder->>Dataset: Filter by audited failure_mode
        Builder->>Builder: Apply sampling/limit
    else Default (all 46)
        Builder->>Builder: Use all tasks (EVAL_MAX_K or EVAL_HARDBENCHMARK_LIMIT)
    end

    Builder->>Config: Look up task categories
    Config-->>Builder: categories
    Builder->>Builder: Build Testcase with provenance metadata (source_suite, failure_mode, capability_axis)
    Builder-->>Planner: 46 testcases

    Planner->>Harness: Build external harness plan for each task
    Harness->>Harness: Map agent/hardbenchmark -> dataset: "hardbenchmark"
    Harness->>Eval: Execute with startUrl, instruction, precomputed_rubric

    alt Task runs successfully
        Eval-->>Harness: Result with rubric match
        Harness-->>Planner: Testcase outcome
    else Task fails (expected at ~0% pass rate)
        Eval-->>Harness: Failure with trajectory
        Harness-->>Planner: Failure attributed to capability_axis
    end

    Planner-->>TUI: Aggregate results with provenance-aware reporting

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: benchmarks.hardbenchmark.limit is not consumed by the runner, so this setting cannot control the suite and creates a false configuration surface. Wire the config value into the suite's limit resolution, or remove this entry until benchmark limits are read from config.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/evals.config.json, line 22:

<comment>`benchmarks.hardbenchmark.limit` is not consumed by the runner, so this setting cannot control the suite and creates a false configuration surface. Wire the config value into the suite's limit resolution, or remove this entry until benchmark limits are read from config.</comment>

<file context>
@@ -18,6 +18,9 @@
       "limit": 25
     },
+    "hardbenchmark": {
+      "limit": 46
+    },
     "odysseysbench": {
</file context>

[cubic-dev-ai]

P2: Add a focused planner test for agent/hardbenchmark; this new route currently has no coverage for dataset routing or generated testcase shape, so regressions can silently disable the suite.

(Based on your team's feedback about unit tests for new behavior.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/benchPlanner.ts, line 303:

<comment>Add a focused planner test for `agent/hardbenchmark`; this new route currently has no coverage for dataset routing or generated testcase shape, so regressions can silently disable the suite.

(Based on your team's feedback about unit tests for new behavior.) </comment>

<file context>
@@ -299,6 +300,7 @@ export function generateSuiteTestcases(
     "agent/webvoyager": (models) => buildWebVoyagerTestcases(models),
     "agent/onlineMind2Web": (models) => buildOnlineMind2WebTestcases(models),
     "agent/webtailbench": (models) => buildWebTailBenchTestcases(models),
+    "agent/hardbenchmark": (models) => buildHardBenchmarkTestcases(models),
     "agent/odysseysbench": (models) => buildOdysseysBenchTestcases(models),
   };
</file context>

[cubic-dev-ai]

P2: OnlineMind2Web tasks lose their site-scoping contract when run through this suite. Because every row is labeled agent/hardbenchmark, external harnesses use generic guidance and cannot distinguish the 13 OnlineMind2Web rows; preserve the source suite in the plan/input or add source-aware guidance, and retain the source URL when it is available.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/suites/hardbenchmark.ts, line 116:

<comment>OnlineMind2Web tasks lose their site-scoping contract when run through this suite. Because every row is labeled `agent/hardbenchmark`, external harnesses use generic guidance and cannot distinguish the 13 OnlineMind2Web rows; preserve the source suite in the plan/input or add source-aware guidance, and retain the source URL when it is available.</comment>

<file context>
@@ -0,0 +1,154 @@
+  for (const modelEntry of normalizeAgentModelEntries(models)) {
+    for (const row of rows) {
+      const input: EvalInput = {
+        name: "agent/hardbenchmark",
+        modelName: modelEntry.modelName as AvailableModel,
+        agentMode: modelEntry.mode,
</file context>

[cubic-dev-ai]

P3: When an unsupported external task is requested, the error now omits the newly supported agent/hardbenchmark suite. Add agent/hardbenchmark to the fallback message so the diagnostic matches this branch.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/externalHarnessPlan.ts, line 51:

<comment>When an unsupported external task is requested, the error now omits the newly supported `agent/hardbenchmark` suite. Add `agent/hardbenchmark` to the fallback message so the diagnostic matches this branch.</comment>

<file context>
@@ -48,13 +48,13 @@ export function buildExternalHarnessTaskPlan(input: EvalInput): ExternalHarnessT
   }
 
-  if (input.name === "agent/webtailbench") {
+  if (input.name === "agent/webtailbench" || input.name === "agent/hardbenchmark") {
     const instruction = readString(params, "ques");
     if (!instruction) {
</file context>

[miguelg719]

This work is consolidated in #2891, which adds HardBench and rubric v1.2 on the current evaluator. The replacement retains 19 active tasks from this PR; 27 previously retired or quarantined tasks are intentionally omitted. Its full corpus contains 122 tasks across core, extended and holdout, without the corpus manifest layer.