Add HardBench and rubric v1.2 on the current evaluator

[miguelg719]

Adds 102 HardBench tasks with precomputed rubric v1.2 through the existing suite and evaluator: core (38) and extended (64). Core is the default; extended selects all 102 shipped tasks. Selection uses the dataset directly, and CLI help lists only the supported tiers.

Keeps the benchmark data, suite selection tests, and rubric policy documentation. Offline authoring scripts, the V3 compatibility campaign, and their dedicated tests/fixtures are outside this PR; they are preserved separately for reference ahead of the evaluator V4 migration.

Validation: 18 HardBench suite and help-import tests pass; eval typecheck and formatting pass.

Stack position: 1/17. Base: main.

Reviewer entry points:

• packages/evals/suites/hardbenchmark.ts
• packages/evals/tests/suites/hardbenchmark.test.ts
• packages/evals/datasets/hardbenchmark/RUBRIC-CLARIFICATIONS.md

[changeset-bot]

⚠️ No Changeset found

Latest commit: 405d108

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 26 files

Not reviewed (too large): packages/evals/datasets/hardbenchmark/HardBenchmark_data.jsonl (~158 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

Architecture diagram

sequenceDiagram
    participant TUI as TUI/CLI
    participant Planner as Bench Planner
    participant Discovery as Task Discovery
    participant Suite as HardBench Suite
    participant Corpus as HardBench Dataset (158 rows)
    participant Plan as External Harness Plan
    participant V3 as V3 Evaluator/Verifier
    participant Gate as Compatibility Gate
    participant Audit as Audit Tooling

    Note over TUI,Audit: HardBench Evaluation Flow

    TUI->>Discovery: b:hardbenchmark shorthand
    Discovery->>Discovery: Register virtual external agent suite
    Discovery-->>TUI: agent/hardbenchmark task

    TUI->>Planner: Run suite with harness
    Planner->>Suite: generateSuiteTestcases()
    
    Note over Suite,Corpus: Selection Logic
    Suite->>Corpus: Load JSONL + read env config
    Corpus-->>Suite: 158 rows (valid + retired/quarantined)
    
    alt EVAL_HARDBENCHMARK_SET = core (default)
        Suite->>Suite: Filter valid non-holdout, set == core, 38 tasks
    else extended/all
        Suite->>Suite: Filter valid non-holdout, 102 tasks
    else holdout
        Suite->>Suite: Filter set == holdout, 20 tasks
    end
    
    alt Explicit EVAL_HARDBENCHMARK_IDS
        Suite->>Suite: Resolve IDs/slugs, preserve order
    else Sampling/cap
        Suite->>Suite: Apply EVAL_MAX_K/LIMIT/SAMPLE
    end

    Suite-->>Planner: Testcases with precomputed rubric + metadata
    Planner->>Plan: buildExternalHarnessTaskPlan()
    Plan->>Plan: Map to dataset "hardbenchmark"
    Plan-->>Planner: Task plan (startUrl or Google fallback)
    Planner-->>TUI: Ready for execution

    Note over TUI,V3: Runtime Execution
    TUI->>V3: Run agent on task
    V3->>V3: normalizeRubric(v1.2)
    V3->>V3: Execute via pinned stagehand-v3@3.7.1
    V3-->>TUI: Evaluation result with process/outcome scores

    Note over Gate,Audit: Offline Validation (Dev/CI)
    Gate->>Gate: runCompatibilityGate(repo, dataset, fixtures, out)
    Gate->>V3: Import real V3 evaluator/adapter
    V3->>V3: Full rubric verification (offline)
    alt Offline mode
        Gate->>Gate: Intercept all fetch calls, block network
        Gate->>Gate: Inject provider/schema faults for negative tests
    else Live mode (--live --judge-model)
        Gate->>Gate: createLiveVerifierFetch (sanitize + redact)
        Gate->>V3: Forward real requests to google/gemini-2.5-flash
        V3-->>Gate: Real provider responses
    end
    
    alt Transport fault injected
        Gate->>Gate: Reject case, record faultRejected=true
    else No fault
        Gate->>Gate: Verify rubric source, criterion identity, point arithmetic
        Gate->>Gate: Separate outcome from process credit
        alt Process within expected interval
            Gate-->>Gate: Accept case
        else Process outside bounds
            Gate-->>Gate: Reject case
        end
    end

    Gate->>Gate: Persist sanitized request evidence
    Gate-->>TUI: Pass/fail report

    Note over Audit,Corpus: Authoring Checks
    Audit->>Corpus: audit-hardbenchmark.ts (offline, no credentials)
    Audit->>Corpus: rubric-clarify.py (idempotent v1.2 transform)
    Audit-->>Corpus: Validated rows, retired/quarantined stay attributable

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 15 files (changes from recent commits).

Not reviewed (too large): packages/evals/datasets/hardbenchmark/HardBenchmark_data.jsonl (~280 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

Not reviewed (too large): packages/evals/datasets/hardbenchmark/HardBenchmark_data.jsonl (~20 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>