Keep verifier outcomes, uncertainty, and execution failures auditable

[miguelg719]

Separates verified task completion from execution health. Verification errors fail closed, preserve the agent report separately, and retain captured evidence and raw uncertainty for audit. A supported completion can still pass after a late browser disconnect.

Preserves judge verdicts beside deterministic evidence gates; missing criterion scores remain in the denominator. Blocker wording and execution error alone do not erase a completed rubric requirement. Numeric grounding remains advisory by default. It includes captured step and terminal accessibility text, records terminal matches separately, and treats unknown-page text as untrusted. Batch summaries expose explicit zero-browser passes; the configured verifiability gate rejects them. Producer-dependent usage, cost, facade and lifecycle fields are documented as conditional.

Offline verification uses the same model factory as live grading: CLI override, then EVAL_VERIFIER_MODEL, then the shared default. Thrown verification errors and uncertain judgments are preserved as explicit ungraded artifacts without top-level outcome/process scores, including live persistence and tracing; the offline command exits unsuccessfully. Trace labels are restricted to filename components, and failed writes emit sanitized warnings.

Judge policy still needs confirmation: this proposed change selects google/gemini-3.5-flash by default, while current main uses 2.5 Flash. The completed 108-case compatibility gate explicitly used 2.5 Flash and does not validate the proposed 3.5 default.

Validation: 104 focused verifier/CLI/span/persistence/summary tests at this boundary, eval typecheck and scoped formatting; 13 assertions reproduced the new review gaps before fixes. No frozen 108-case compatibility inputs changed. No evaluator-v4 implementation.

Stack position: 11/17. Base: evals/consolidation-10-usage-provenance. Each PR contains the change relative to its immediate predecessor.

Reviewer entry points:

• packages/evals/framework/verifierAdapter.ts
• packages/evals/framework/verifierGates.ts
• packages/evals/docs/verifier-gates.md

Readiness: local validation is described above; GitHub CI and automated review feedback are pending. This PR is open as non-draft so automated reviewers can run.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2136ac4

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

1 issue found across 17 files

Confidence score: 3/5

• packages/evals/framework/verifierAdapter.ts makes normal verifier runs use an unvalidated judge model, while the compatibility gate only exercises 2.5 Flash; rerun the compatibility campaign with 3.5 before switching the default, or retain the validated 2.5 model to avoid unreliable benchmark results.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/verifierAdapter.ts">

<violation number="1" location="packages/evals/framework/verifierAdapter.ts:78">
P2: This makes normal verifier runs use an unvalidated judge model even though the compatibility gate exercised 2.5 Flash. Keep the validated 2.5 default until the compatibility campaign is rerun with 3.5, otherwise benchmark outcomes can change without the shipped default being covered.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Harness as External Harness
    participant Adapter as verifierAdapter.ts
    participant Evaluator as V3Evaluator
    participant Gates as verifierGates.ts
    participant Persist as persistTrajectory.ts
    participant Logger as EvalLogger
    participant CLI as verify CLI

    Note over Harness,CLI: Verifier grading and audit flow

    Harness->>Adapter: gradeExternalTrajectory({buildTrajectory, verifier config})
    Adapter->>Adapter: createVerifierEvaluator(v3, modelOverride)
    
    alt modelOverride provided
        Adapter->>Adapter: Use CLI model override
    else EVAL_VERIFIER_MODEL set
        Adapter->>Adapter: Use env model
    else default
        Adapter->>Adapter: Use google/gemini-3.5-flash
    end

    Adapter->>Evaluator: verify(hydratedTrajectory)
    Evaluator->>Evaluator: LLM judge (rubric evaluation)
    Evaluator-->>Adapter: raw EvaluationResult

    alt verifier_uncertainty finding present
        Adapter->>Adapter: getUngradedVerifierResult() - mark ungraded
        Adapter->>Logger: log span (graded=false, verifierError)
        Adapter-->>Harness: Return failed-closed result with verifierError
    else judge verdict available
        Adapter->>Gates: applyVerdictGates({evaluation, trajectory, isFacadeTool})
        
        alt judge passed
            Gates->>Gates: Check final answer empty?
            alt empty final answer
                Gates->>Gates: outcomeGates += no_final_answer
            end
            
            opt isFacadeTool matcher present
                Gates->>Gates: Check facade tool calls in trajectory
                alt zero facade calls
                    Gates->>Gates: outcomeGates += no_browser_use
                end
            end
            
            opt EVAL_REQUIRE_GROUNDING=1
                Gates->>Gates: Check grounding datums (currency/percent/time)
                alt all numeric only in search results
                    Gates->>Gates: outcomeGates += ungrounded_answer
                end
            end
        end

        Gates->>Gates: strictProcessScore() - zero evidenceInsufficient criteria
        Gates-->>Adapter: VerdictGates (outcomeSuccess, processScoreStrict)
        
        Adapter->>Adapter: buildPersistedEvaluationResult (judge verdict + gates)
        Adapter->>Persist: persistAdapterTrajectory(trajectory, evaluationResult)
        Persist->>Persist: Write trajectory.json + metadata.json (terminationReason)
        Persist->>Persist: Write scores/result.json (top-level gated verdict, raw judge under "judge")
        Persist->>Persist: Write scores/gates.json + verifier-error.json
        Persist-->>Adapter: directory + persisted flag
        
        opt EVAL_VERIFIER_TRACE=1
            Adapter->>Logger: getLogs({maxLevel: 2})
            Adapter->>Persist: Write scores/verifier-trace.jsonl
        end
        
        Adapter-->>Harness: Return TaskResult (with gate metrics, judge verdict, strict score)
    end

    Note over CLI,Persist: Offline verification command

    CLI->>CLI: handleVerify(args)
    CLI->>Adapter: createVerifierEvaluator(v3, parsed.model)
    
    opt EVAL_VERIFIER_TRACE=1
        CLI->>CLI: V3 verbose=2, capture level-2 log lines
    end
    
    CLI->>Evaluator: verify(trajectory)
    Evaluator-->>CLI: EvaluationResult
    
    alt uncertainty result
        CLI->>CLI: getUngradedVerifierResult()
        CLI->>CLI: Write { graded: false, judge: raw, verifierError }
        CLI->>CLI: process.exitCode = 1
    else trustworthy judge verdict
        CLI->>Gates: applyVerdictGates()
        Gates-->>CLI: VerdictGates
        CLI->>CLI: buildPersistedEvaluationResult()
        CLI->>Persist: Write scores/result_<label>.json
        CLI->>CLI: formatVerdictLine() - gated verdict + judge verdict
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This makes normal verifier runs use an unvalidated judge model even though the compatibility gate exercised 2.5 Flash. Keep the validated 2.5 default until the compatibility campaign is rerun with 3.5, otherwise benchmark outcomes can change without the shipped default being covered.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/verifierAdapter.ts, line 78:

<comment>This makes normal verifier runs use an unvalidated judge model even though the compatibility gate exercised 2.5 Flash. Keep the validated 2.5 default until the compatibility campaign is rerun with 3.5, otherwise benchmark outcomes can change without the shipped default being covered.</comment>

<file context>
@@ -10,31 +10,90 @@ import {
 const VERIFIER_MODEL_ENV = "EVAL_VERIFIER_MODEL";
 const KEYLESS_VERIFIER_PROVIDERS = new Set(["bedrock", "ollama"]);
+/** Campaign-validated default; callers can pin the judge independently. */
+export const DEFAULT_VERIFIER_MODEL = "google/gemini-3.5-flash";
 
 /**
</file context>

Suggested change

	export const DEFAULT_VERIFIER_MODEL = "google/gemini-3.5-flash";
	export const DEFAULT_VERIFIER_MODEL = "google/gemini-2.5-flash";

[miguelg719]

This decision remains open. Current main uses google/gemini-2.5-flash, and the completed 108-case live gate explicitly used that model. It does not validate the proposed google/gemini-3.5-flash default. The unsupported campaign-validation claim has been removed and the PR description records this limitation. Miguel has been asked whether to preserve the current default or include the new default with its own live validation; this thread stays unresolved until that policy decision is made.