[feat]: evals welcome — animated onboarding on the agent benchmarks

[miguelg719]

why

the first-run experience for the evals CLI is a static block of text. the public framing (stagehand.dev/evals) is accuracy · speed · cost on the agent benchmarks — the first minute with the CLI should show that on a real task, then hand the user a real run or tell them exactly what unlocks one

what changed

• added evals welcome, one animated flow on a real WebVoyager case (never the core tier):
  • intro: the Stagehand mark carves in → the statement → accuracy / speed / cost land big in their colors and collapse into lines → top of the board (92.1%) → the EVALS mark assembles and stays as the header
  • the run: three models (Claude Fable 5.1, Claude Opus 5, GPT-5.6-Sol) run the same task in lanes; the fastest fails on a wrong condition filter; podium ranked by accuracy → speed → cost
  • "want to see how the winner got there?" → the agent narrates the winning run chat-style (thoughts as bubbles, tool calls as chips with the clock, browser observations from the right, the judge's verdict)
  • hand-off: run b:webvoyager -l 3 --harness claude_code|codex -e local|browserbase (harness from the detected key, env explicit) when a key + browser exist, otherwise the new evals setup
• added evals setup, a guided flow: detects what's present, asks only for what's missing (provider → harness, masked key paste; Browserbase creds or install Chrome), writes packages/evals/.env (mode 600), re-detects, and offers to run the first real case; prints a checklist off-TTY
• EVALS_WELCOME_WIZARD=1 auto-runs the flow on the first REPL launch; EVALS_NO_WELCOME=1 still suppresses the first-run welcome; any key advances the intro, Esc skips ahead, Ctrl+C cancels without burning the first-run marker
• new animation layer (tui/fx.ts, tui/wizardAnim.ts): 24fps frame loop with easing, per-cell truecolor canvas (fg + bg → half-block pixels), seeded particles, reading-paced reveals, in-place repaint that degrades to static frames off-TTY or under 80 cols
• vendored the figlet ANSI Shadow face (what the EVALS banner already used) + generator, so every headline shares one voice; the Stagehand mark is rasterized from the SVG's contour snapped to the pixel grid so its slits stay crisp at any size
• stdin discipline: raw-byte Esc/Ctrl+C listener (readline's keypress decoder can leak a held Esc into clack); the REPL detaches its keypress listeners around the flow (a paused interface still echoes into rl.line); argv releases its Esc-exits-CLI handler; an accepted hand-off is dispatched through the command tree in both argv and REPL modes (from the root context); the CLI also loads packages/evals/.env (after the cwd .env) so a repo-root launch sees the keys evals setup saved
• adds @clack/prompts; pnpm-lock.yaml regenerated with the pinned pnpm 11.10 (the supports-color peer suffixes / @types/node bump are its re-resolution of main's lock, not intentional)

test plan

• unit: tests/tui/welcomeFramework.test.ts (opt-in parsing, plan derivation real vs scripted incl. harness choice, setup hand-off), tests/tui/welcomeRender.test.ts (block-font typesetting, the mark keeps both slits at 24/28/32px and stays binary, fx helpers, particle z-order, canvas fg/bg)
• tests/cli.test.ts warms the tsx cache in beforeAll — its first spawn ("shows help") sat at ~10.0s on CI against the 10s per-test budget and flaked twice on this PR
• tsc --noEmit, oxlint, oxfmt --check clean
• PTY harness: Ctrl+C at 2s cancels + exits with no first-run marker; Esc fast-forwards; non-TTY completes without prompting; full flow task → run → podium → prompt → inside → judge → hand-off → clean exit; hand-off ignores ordinary keys and Enter runs the recommended command; Ctrl+C mid-judge cancels cleanly
• evals setup PTY-driven: provider → key → saved → re-detected → offer; key never echoed; .env upsert unit-tested; off-TTY checklist
• REPL: first-run auto-launch, cancel leaves the marker unset, EVALS_NO_WELCOME still burns it, REPL responsive after an in-REPL welcome
• the recommended real command plans (--preview) for both harnesses; agent suites reject the default stagehand harness, which is why the harness is chosen from the detected key
• all 28 cubic threads from the first pass triaged and resolved on-thread (21 fixed, 7 dismissed with reasons)

notes: best viewed ≥ 96 cols wide. the first-run marker still lives in the tracked evals.config.json (unchanged from main); running the CLI in tsx source mode dirties it, dist mode writes the dist copy.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 7b455cf

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​@​clack/​prompts@​1.7.0

View full report

[cubic-dev-ai]

All reported issues were addressed across 25 files

Architecture diagram

sequenceDiagram
    participant CLI as CLI Entry
    participant REPL as REPL Runtime
    participant Tree as Command Tree
    participant Welcome as Welcome Engine
    participant Fx as FX/Anim Engine
    participant TTY as Terminal/TTY
    participant Env as Env/Detect
    participant Script as Scripted Agent
    participant Handoff as Hand-off Dispatcher

    Note over CLI,Handoff: evals welcome onboarding architecture

    CLI->>CLI: Parse args and env
    alt Command: "welcome"
        CLI->>Tree: Dispatch welcome command
        Tree->>Welcome: handleWelcome()
    else REPL first-run with EVALS_WELCOME_WIZARD
        REPL->>Welcome: runWelcome() pre-readline
    else REPL explicit "welcome"
        REPL->>Tree: Dispatch through command tree
        Tree->>Welcome: handleWelcome()
    end

    Note over Welcome,Script: Welcome flow owns stdin exclusively
    Welcome->>Welcome: Suspend input handlers
    Welcome->>Welcome: Acquire raw-byte Esc/Ctrl+C listener

    Welcome->>Env: detectMachine()
    Env->>Env: Check provider keys (OPENAI/ANTHROPIC/GOOGLE)
    Env->>Env: Check browser (Chrome/Browserbase)
    Env-->>Welcome: Plan: real vs scripted

    Welcome->>Script: loadScriptedCases()
    Script-->>Welcome: Real WebVoyager case (Amazon--4)

    Welcome->>Fx: Initialize animation engine
    Fx->>TTY: Check TTY/canAnimateInPlace
    TTY-->>Fx: Animation capability

    alt Scripted mode (no key/browser)
        Welcome->>Fx: Render static frames
        Fx-->>TTY: Static output (no animation)
    else Animated mode (≥80 cols TTY)
        Welcome->>Fx: Run arena race animation
        Fx->>Fx: 24fps frame loop with easing
        Fx->>TTY: Paint frames via LiveBlock
        TTY-->>Fx: Frame complete
    end

    Note over Welcome,Script: Race scenario
    Fx->>Fx: Three model lanes race
    Fx->>Fx: Fastest model fails (wrong condition)
    Fx->>TTY: Show podium (accuracy/speed/cost)

    Welcome->>Welcome: Prompt: "See how winner got there?"
    alt User says yes
        Welcome->>Script: Replay winning trajectory
        Script-->>Welcome: Scripted steps (observe/act/answer)
        Welcome->>Fx: Render chat-style narration
        Fx->>TTY: Bubbles, tool chips, judge verdict
    end

    Welcome->>Handoff: Prepare recommendation
    alt Real run possible
        Handoff->>Handoff: Command: "run b:webvoyager -l 3"
    else Missing requirements
        Handoff->>Handoff: Command: "list bench" + what's missing
    end

    Welcome->>TTY: Show handoff chip (Enter to run)
    alt User presses Enter
        Welcome-->>Handoff: runNext command
        alt CLI context
            Handoff->>Tree: Dispatch recommended command
        else REPL context
            Handoff->>REPL: prefillInput() next prompt
            REPL->>REPL: Pre-fill buffer
        end
    else User presses Esc
        Welcome-->>Handoff: null (drop to prompt)
    else Ctrl+C
        Welcome-->>Welcome: cancelled (no first-run marker)
    end

    Welcome->>Welcome: Restore input handlers
    Welcome->>Welcome: Restore cursor visibility

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[miguelg719]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 28 files

Architecture diagram

sequenceDiagram
    participant USR as User
    participant CLI as CLI Entry
    participant CT as Command Tree
    participant WZ as Welcome Wizard
    participant SET as Setup Command
    participant DET as Detection
    participant ENV as .env Store
    participant REPL as REPL Interface

    Note over USR,ENV: Welcome Flow - Full Onboarding Experience

    USR->>CLI: evals welcome
    CLI->>CT: Dispatch welcome command
    CT->>WZ: handleWelcome()
    WZ->>WZ: suspendInput() - own stdin
    WZ->>WZ: Run intro screens
    WZ->>DET: detectMachine()
    DET->>ENV: Check provider keys
    
    alt Provider key + browser available
        DET-->>WZ: plan: real
        WZ->>WZ: Run real benchmark task
        WZ->>WZ: Show lanes + podium
        WZ-->>USR: Display winner
        WZ->>USR: Handoff chip (Enter/Esc)
        
        alt User presses Enter
            WZ-->>CT: runNext command
            CT->>CT: Dispatch run command
            CLI->>REPL: Pre-fill or execute
        else User presses Esc
            WZ-->>CLI: Drop to prompt
        end
    
    else No key or browser
        DET-->>WZ: plan: scripted
        WZ->>USR: Show scripted replay
        WZ-->>CT: setup command
    end

    CLI->>SET: evals setup
    SET->>DET: detectMachine()
    
    alt Off-TTY (no terminal)
        SET-->>USR: Print static checklist
    else TTY interactive
        SET->>SET: suspendInput()
        
        alt No provider key
            SET->>USR: Select provider
            USR->>SET: Provider type
            SET->>USR: Paste masked key
            USR-->>SET: Key entry
            SET->>ENV: Write key entry
        end
        
        alt No browser detected
            SET->>USR: Browser choice prompt
            alt Browserbase chosen
                USR->>SET: Browserbase creds
                SET->>ENV: Write API key + project ID
            end
        end
        
        SET->>ENV: writeEnv(mode 600)
        ENV-->>SET: Confirm write
        SET->>DET: Re-detect machine
        DET-->>SET: Updated status
        
        alt Real plan now possible
            SET->>USR: Offer first run
            alt User confirms
                SET->>CT: Dispatch run command
                CT->>REPL: Pre-fill input buffer
                REPL-->>USR: Press Enter to run
            else User declines
                SET-->>USR: Later command hint
            end
        end
    end

    Note over WZ,SET: Ctrl+C handling in both flows
    USR->>WZ: Ctrl+C
    WZ->>WZ: Cancel without marking first-use
    
    Note over REPL,WZ: REPL First-Line Wizard
    USR->>REPL: Start REPL
    REPL->>WZ: EVALS_WELCOME_WIZARD=1 check
    alt Wizard enabled + first run
        REPL->>WZ: Auto-launch runWelcome()
        WZ-->>REPL: pendingHandoff (on completion)
        REPL->>REPL: Pre-fill readline buffer
    else Wizard disabled
        REPL->>REPL: Show static banner
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[miguelg719]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 28 files

Architecture diagram

sequenceDiagram
    participant CLI as CLI Entry
    participant Repl as REPL
    participant Tree as Command Tree
    participant Welcome as Welcome Flow
    participant Setup as Setup Flow
    participant Detect as Detector
    participant Env as .env File

    Note over CLI,Repl: First-run detection and opt-in
    CLI->>Repl: Start (EVALS_WELCOME_WIZARD=1)
    Repl->>Repl: Check first-run + wizard enabled
    alt Wizard enabled + first run
        Repl->>Welcome: runWelcome()
        Welcome-->>Repl: completed with runNext
    else Normal flow
        Repl->>Repl: Print banner/tip
        Repl->>Repl: Mark first-run complete
    end

    Note over Welcome,Detect: Machine detection for real vs scripted
    Welcome->>Detect: detectMachine()
    Detect->>Env: Check package .env
    Detect->>Detect: Promote keys to process.env
    Detect-->>Welcome: Plan (real or scripted)

    alt Real plan (key + browser)
        Welcome->>Welcome: Run simulated race (3 models, podium)
        Welcome->>Welcome: Show winning run chat-style
        Welcome->>Welcome: Hand-off recommended real command
        Welcome-->>CLI: Dispatch run b:webvoyager command
        CLI->>Tree: Execute real benchmark
    else Scripted plan (missing deps)
        Welcome->>Welcome: Run scripted animation only
        Welcome->>Setup: Hand-off to guided setup
        Note over Setup: Interactive prompts
        Setup->>Detect: Check current state
        Setup->>Setup: Ask for provider key (masked)
        Setup->>Env: Write/update .env (mode 600)
        Env-->>Setup: Confirm saved
        Setup->>Detect: Re-detect machine
        alt Now real
            Setup-->>CLI: Offer first run
            CLI->>Tree: Dispatch run command
        else Still missing
            Setup-->>CLI: Print checklist/outro
        end
    end

    Note over Welcome,Setup: Input handling
    Welcome->>Welcome: Raw listener (Esc skip, Ctrl+C cancel)
    Welcome->>Welcome: Suspend REPL/argv input handlers
    Welcome-->>Welcome: Restore input handlers
    Setup->>Setup: Clack prompts handle own input
    Setup->>Setup: Restore input on finish/cancel

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[miguelg719]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 29 files

Architecture diagram

sequenceDiagram
    participant User as User
    participant CLI as CLI (argv/REPL)
    participant CmdTree as Command Tree
    participant Welcome as Welcome Flow
    participant Setup as Setup Flow
    participant Env as Packages/evals/.env
    participant Detect as detectMachine()
    participant Anim as Animation Layer
    participant Keys as Stdin Listener
    participant Run as Run Framework

    Note over User,Run: First-run welcome with animated onboarding

    alt EVALS_WELCOME_WIZARD=1 and first run (REPL)
        CLI->>Welcome: Auto-launch runWelcome()
        Welcome->>Anim: Render intro (mark, metrics, leaderboard)
        Anim-->>Welcome: Animated frames
    else Manual `evals welcome` or `evals setup`
        User->>CLI: Run welcome command
        CLI->>CmdTree: Dispatch command
        CmdTree->>Welcome: handleWelcome()
    end

    Note over Welcome: Flow starts with stdin ownership
    Welcome->>Keys: suspendInput() - detach REPL/argv key listeners
    Welcome->>Welcome: Own raw-byte Esc/Ctrl+C listener

    Welcome->>Detect: Check key + browser presence
    Detect->>Env: Resolve credentials
    Env-->>Detect: Key/browser status
    Detect-->>Welcome: Plan (real vs scripted)

    alt Has provider key + browser (real plan)
        Welcome->>Anim: Show 3-model race on WebVoyager task
        Anim-->>Welcome: Animated race lanes
        Welcome->>Welcome: Podium by accuracy/speed/cost
        Welcome->>Anim: Chat-style talk-through of winning run
        Welcome->>User: Display hand-off command chip
        User->>Welcome: Press Enter (accept hand-off)
        Note over Welcome: Esc skips, Ctrl+C cancels
        Welcome->>CmdTree: Dispatch accepted hand-off command
        CmdTree->>Run: Execute `run b:webvoyager -l 3 --harness ...`
        Run-->>CLI: Real benchmark results
    else Missing key or browser (scripted plan)
        Welcome->>Anim: Replay scripted trajectory
        Welcome->>User: Offer `evals setup` as next step
    end

    Note over Welcome: Cleanup
    Welcome->>Keys: Restore previous key listeners
    Welcome-->>CLI: Return outcome (completed/cancelled)

    alt User picks setup (no key/browser)
        CLI->>CmdTree: Dispatch setup command
        CmdTree->>Setup: handleSetup()
        Setup->>Detect: Detect what's missing
        Setup->>User: Ask only for missing items
        User->>Setup: Provide masked key / browser choice
        Setup->>Env: Write .env (mode 600)
        Env-->>Setup: Persisted
        Setup->>Detect: Re-detect machine state
        alt All requirements met
            Setup->>User: Offer first real run
            User->>Setup: Confirm
            Setup->>CmdTree: Dispatch run command
        else Still missing
            Setup->>User: Show what's needed
        end
    end

    Note over Welcome,Run: Non-TTY degradation
    alt Off-TTY or <80 cols
        Welcome->>Anim: Static frames only, no prompts
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[miguelg719]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 29 files

Architecture diagram

sequenceDiagram
    participant User as User Terminal
    participant CLI as evals CLI (argv/REPL)
    participant CmdTree as Command Tree
    participant Welcome as Welcome Flow
    participant Setup as evals Setup
    participant Detect as Machine Detection
    participant Env as .env Files
    participant Anim as Animation Layer
    participant Browser as Chrome/Browserbase

    Note over User,CLI: Entry Points
    User->>CLI: evals welcome / evals setup
    User->>CLI: REPL first launch (EVALS_WELCOME_WIZARD=1)
    
    CLI->>CLI: Load cwd .env then packages/evals/.env
    CLI->>CmdTree: Dispatch command

    Note over CmdTree,Welcome: Welcome Flow (Arena)
    CmdTree->>Welcome: handleWelcome()
    Welcome->>Welcome: suspendInput() - detach keypress listeners
    
    Welcome->>Detect: detectMachine()
    Detect->>Env: Check env keys
    Detect->>Browser: Detect Chrome/Browserbase
    
    alt Key + Browser Available
        Welcome->>Welcome: Real plan (claude_code/codex)
        Welcome->>Anim: Run 3-model race animation
        Anim-->>Welcome: Podium + judge verdict
        Welcome->>User: Hand-off chip: run b:webvoyager...
    else Missing Key or Browser
        Welcome->>Welcome: Scripted plan (replay)
        Welcome->>Anim: Run scripted trajectories
        Welcome->>Setup: Hand-off to evals setup
    end

    User->>Welcome: Esc (skip) / Ctrl+C (cancel)
    Welcome->>Welcome: Cancel without marking first-run

    Note over Setup,Env: Setup Flow
    Setup->>Detect: detectMachine()
    Detect-->>Setup: Current state
    Setup->>User: Ask only for missing items (clack prompts)
    
    opt No Provider Key
        User->>Setup: Select provider (Anthropic/OpenAI)
        User->>Setup: Paste masked API key
    end
    
    opt No Browser
        User->>Setup: Select Browserbase or Chrome
        opt Browserbase Selected
            Setup-->>Setup: Prompt for API key/project ID
        end
    end

    Setup->>Env: Write packages/evals/.env (mode 600)
    Setup->>Detect: Re-detect
    Detect-->>Setup: Updated state
    
    alt Setup Complete
        Setup->>User: Offer first real run
        User->>Setup: Confirm (Enter)
        Setup->>CmdTree: Dispatch run command
    else Off-TTY
        Setup->>User: Print static checklist
    end

    Note over CLI,Anim: REPL Wizard Auto-run
    CLI->>Welcome: runWelcome() (EVALS_WELCOME_WIZARD=1)
    Welcome->>Anim: Run intro + race + handoff
    Welcome-->>CLI: result (completed/runNext)
    
    alt Hand-off Accepted
        Welcome->>CmdTree: Dispatch firstRunCommand
        CmdTree->>Browser: Run real WebVoyager cases
    else Cancelled
        Welcome-->>CLI: Print tip line, no first-run marker
    end

    Welcome->>Welcome: restoreInput() - reattach listeners
    CLI->>CLI: markFirstRunComplete() (on natural completion)

Loading

<sub>You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic</sub>

[miguelg719]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@miguelg719 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

4 issues found across 29 files

Confidence score: 3/5

• In packages/evals/tui/repl.ts, a failed accepted welcome hand-off prints an arbitrary dispatch error verbatim, which could expose unsanitized details to users; sanitize and normalize the error before displaying it.
• In packages/evals/tui/commands/setup.ts, partial Browserbase credentials are reported as if neither credential exists, and the non-TTY checklist requests both variables, potentially sending users toward incorrect setup; detect and describe missing credentials individually.
• In packages/evals/tui/commands/setup.ts, the new interactive branches lack focused coverage for supported-key gating, partial credentials, cancellation after saving, and accepted runs, leaving observable setup regressions easier to introduce; add targeted tests.
• In packages/evals/tui/commands/setup.ts, Windows non-interactive output displays the .env path with backslashes instead of the CLI’s forward-slash convention, causing a minor cross-platform presentation inconsistency; normalize only the displayed path while retaining the native path internally.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/tui/commands/setup.ts">

<violation number="1" location="packages/evals/tui/commands/setup.ts:84">
P2: When only one Browserbase credential is configured, setup reports “no local Chrome, no Browserbase credentials” and the non-TTY checklist asks for both variables. Use `m.keys.browserbase.apiKey` and `.projectId` to report and list only missing fields; the interactive prompts already do this.</violation>

<violation number="2" location="packages/evals/tui/commands/setup.ts:98">
P3: On Windows, the non-interactive checklist prints the `.env` path with backslashes, unlike the CLI's forward-slash display convention. Normalize this display-only path before printing it while retaining the native path for filesystem calls.

(Based on your team's feedback about user-visible path formatting).</violation>

<violation number="3" location="packages/evals/tui/commands/setup.ts:114">
P3: `handleSetup` adds several interactive branches without tests for their observable behavior. Add focused tests for supported-key gating, partial Browserbase credentials, cancellation after saving, and the accepted-run handoff.

(Based on your team's feedback about unit tests for new behavior).</violation>
</file>

<file name="packages/evals/tui/repl.ts">

<violation number="1" location="packages/evals/tui/repl.ts:204">
P1: Custom agent: **Exception and error message sanitization**

When the accepted welcome hand-off fails, this catch prints the arbitrary dispatch error message verbatim. Sanitize the message and normalize it through the typed user-facing error path before writing it to the console, so provider or child-process failures cannot expose secrets, project IDs, or connection URLs.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant User as User
    participant CLI as CLI Entry (cli.ts)
    participant REPL as REPL (repl.ts)
    participant CmdTree as Command Tree
    participant Welcome as Welcome Wizard
    participant Setup as Setup Flow
    participant Env as .env Package
    participant Detect as Machine Detection
    participant Anim as Animation Layer (fx)
    participant Input as Raw Stdin Listener

    Note over User,Input: New User Onboarding Flow
    User->>CLI: Launch evals (first run)
    CLI->>CLI: Load .env files (cwd first, then packages/evals/.env)
    
    alt EVALS_WELCOME_WIZARD=1 and first run
        CLI->>REPL: Enter REPL
        REPL->>Welcome: Auto-run wizard
        Note over Welcome,Anim: Animated intro (fx engine 24fps)
        Welcome->>Anim: Initialize canvas, particles, easing
        Anim->>Welcome: Render frames
        Welcome->>Detect: Check machine state
        Detect->>Env: Read keys from packages/evals/.env
        Env-->>Detect: Key status
        Detect-->>Welcome: Machine profile (provider, browser)
        
        alt Has provider key + browser
            Welcome->>Welcome: Scripted race replay (3 models)
            Welcome->>Input: Listen for raw keys
            User->>Input: Press Enter
            Input-->>Welcome: Run command accepted
            Welcome-->>REPL: Hand-off command
            REPL->>CmdTree: Dispatch run command
            CmdTree-->>User: Execute real benchmark
        else Missing key or browser
            Welcome->>Setup: Hand-off to setup
        end
    else First run (no wizard env)
        REPL->>REPL: Print static banner + tip
    end

    Note over User,Setup: Guided Setup Flow
    User->>CmdTree: evals setup
    CmdTree->>Setup: handleSetup()
    Setup->>Detect: detectMachine()
    Detect->>Env: Check keys, chrome, browserbase
    Env-->>Detect: Current state
    Detect-->>Setup: Machine status
    
    alt TTY interactive
        Setup->>Setup: Show status note
        alt Missing provider key
            Setup->>User: Select provider (Anthropic/OpenAI)
            User->>Setup: Select provider
            Setup->>User: Masked key input
            User->>Setup: Paste key (masked)
            Setup->>Env: Prepare env entries
        end
        
        alt Missing browser
            Setup->>User: Choose browser (Browserbase/Chrome)
            User->>Setup: Choice
            alt Browserbase
                Setup->>User: API key + project ID
                User->>Setup: Credentials
            end
        end
        
        Setup->>Env: Write .env (mode 600)
        Env->>Setup: Confirm save
        Setup->>Detect: Re-detect
        Detect-->>Setup: Updated status
        
        alt Machine ready
            Setup->>User: Confirm first run
            User->>Setup: Accept
            Setup->>CmdTree: Dispatch run command
            CmdTree-->>User: Execute benchmark
        end
    else Non-TTY
        Setup->>User: Print checklist (no prompts)
    end
    
    Note over Welcome,Input: Input Discipline
    Welcome->>CLI: suspendInput()
    CLI->>Input: Detach keypress listeners
    Input->>CLI: Confirm raw stdin mode
    CLI->>Input: Release Esc/Ctrl+C handlers
    Welcome->>Input: Raw byte listener
    
    alt Any key
        Input-->>Welcome: Advance intro
    else Esc
        Input-->>Welcome: Skip ahead
    else Ctrl+C
        Input-->>Welcome: Cancel (no marker)
        Welcome-->>REPL: Return cancelled
        REPL-->>User: Clean exit
    end
    
    Note over Welcome,REPL: Completion & State
    Welcome->>REPL: WizardOutcome (completed/cancelled)
    alt Completed
        REPL->>REPL: Mark first-run complete
        Welcome->>CmdTree: Dispatch accepted hand-off
    else Cancelled
        REPL->>REPL: Skip marker (re-show next time)
    end

Loading

<sub>You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Custom agent: Exception and error message sanitization

When the accepted welcome hand-off fails, this catch prints the arbitrary dispatch error message verbatim. Sanitize the message and normalize it through the typed user-facing error path before writing it to the console, so provider or child-process failures cannot expose secrets, project IDs, or connection URLs.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tui/repl.ts, line 204:

<comment>When the accepted welcome hand-off fails, this catch prints the arbitrary dispatch error message verbatim. Sanitize the message and normalize it through the typed user-facing error path before writing it to the console, so provider or child-process failures cannot expose secrets, project IDs, or connection URLs.</comment>

<file context>
@@ -148,6 +196,15 @@ export async function startRepl(entryDir: string, options: ReplOptions = {}): Pr
+    try {
+      await dispatch(tree, tokenize(pendingHandoff), ctx);
+    } catch (err) {
+      console.error(red(`  Error: ${(err as Error).message}`));
+    }
+    rl.setPrompt(renderPrompt(contextPath));
</file context>

[cubic-dev-ai]

P2: When only one Browserbase credential is configured, setup reports “no local Chrome, no Browserbase credentials” and the non-TTY checklist asks for both variables. Use m.keys.browserbase.apiKey and .projectId to report and list only missing fields; the interactive prompts already do this.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tui/commands/setup.ts, line 84:

<comment>When only one Browserbase credential is configured, setup reports “no local Chrome, no Browserbase credentials” and the non-TTY checklist asks for both variables. Use `m.keys.browserbase.apiKey` and `.projectId` to report and list only missing fields; the interactive prompts already do this.</comment>

<file context>
@@ -0,0 +1,239 @@
+      ? ok(`browser: local Chrome ${dim(`(${m.chrome})`)}`)
+      : m.browserbase
+        ? ok("browser: Browserbase (hosted)")
+        : todo("browser: no local Chrome, no Browserbase credentials"),
+  );
+  return rows;
</file context>

[cubic-dev-ai]

P3: handleSetup adds several interactive branches without tests for their observable behavior. Add focused tests for supported-key gating, partial Browserbase credentials, cancellation after saving, and the accepted-run handoff.

(Based on your team's feedback about unit tests for new behavior).

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tui/commands/setup.ts, line 114:

<comment>`handleSetup` adds several interactive branches without tests for their observable behavior. Add focused tests for supported-key gating, partial Browserbase credentials, cancellation after saving, and the accepted-run handoff.

(Based on your team's feedback about unit tests for new behavior). </comment>

<file context>
@@ -0,0 +1,239 @@
+  console.log(lines.join("\n"));
+}
+
+export async function handleSetup(args: string[], ctx?: CommandContext): Promise<void> {
+  if (args[0] === "--help" || args[0] === "-h" || args[0] === "help") {
+    printSetupHelp();
</file context>

[cubic-dev-ai]

P3: On Windows, the non-interactive checklist prints the .env path with backslashes, unlike the CLI's forward-slash display convention. Normalize this display-only path before printing it while retaining the native path for filesystem calls.

(Based on your team's feedback about user-visible path formatting).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tui/commands/setup.ts, line 98:

<comment>On Windows, the non-interactive checklist prints the `.env` path with backslashes, unlike the CLI's forward-slash display convention. Normalize this display-only path before printing it while retaining the native path for filesystem calls.

(Based on your team's feedback about user-visible path formatting). </comment>

<file context>
@@ -0,0 +1,239 @@
+      `      ${cyan(`evals ${firstRunCommand(m.plan)}`)}`,
+    );
+  } else {
+    lines.push(`  ${dim("Add to")} ${cyan(envPath)}${dim(":")}`);
+    if (!m.providers.includes("anthropic") && !m.providers.includes("openai"))
+      lines.push(`      ${cyan("ANTHROPIC_API_KEY")}=…  ${dim("or")}  ${cyan("OPENAI_API_KEY")}=…`);
</file context>