Skip to content

fix(headless): require validated verifier grades - #1644

Merged
Astro-Han merged 3 commits into
apache:mainfrom
me2seeks:fix/1591-validated-verifier-authority
Aug 3, 2026
Merged

fix(headless): require validated verifier grades#1644
Astro-Han merged 3 commits into
apache:mainfrom
me2seeks:fix/1591-validated-verifier-authority

Conversation

@me2seeks

Copy link
Copy Markdown
Contributor

Summary

  • make rejected verifier payloads authority-neutral instead of treating field presence as a grade
  • validate every verifier attempt index so sparse arrays fail closed
  • cover malformed, sparse, and internally inconsistent runner output while preserving provider infrastructure classification

Follow-up to #1591.

Verification

  • npm run build:test
  • npm run lint
  • npm run format:check
  • npm --workspace @maka/headless run typecheck
  • npm --workspace @maka/headless run test:dist — 1,429 passed, 4 environment skips, 0 failed
  • git diff --check

Review focus

This changes only verifier-derived scoring authority. Existing completed-cell, benchmark-deadline, and tool-step eligibility behavior remains unchanged; a verifier influences classification only when structuredVerifierGrade() accepts the full outcome/reward/attempt contract.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reproject stale fallback scores when the stored verifier is rejected

readFixedPromptWal() sends persisted task_completed events through projectStructuredVerifierOutcome(). If structuredVerifierGrade() rejects the stored verifier, the event is returned with its old score flags unchanged.

I reproduced this with a legacy failed max_tokens event written by the previous verifier-presence fallback. scored: true and eligible: true survive resume, the runner is not called, and the event still counts toward the acceptance denominator. The malformed-WAL test currently starts with both flags set to false, so it misses this case.

Please recalculate the identifiable failed/no-deadline/invalid-grade legacy records with the same rules used for fresh output. The existing authority for tool_step_cap_reached, completed cells, and deadlines should stay intact.

Approving with this P2. I would hold the merge until it is addressed.

@Astro-Han

Copy link
Copy Markdown
Contributor

Btw the E2E test has been fixed on main. Rebase to main would solve the Red CI.

Validate every attempt index explicitly so array holes cannot bypass sequence and infrastructure-prefix checks.
Remove the verifier-presence fallback so malformed runner output cannot mark max-token failures scored or eligible.
@me2seeks
me2seeks force-pushed the fix/1591-validated-verifier-authority branch from ecf7a42 to 0835442 Compare August 3, 2026 03:12
@me2seeks

me2seeks commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the replay P2 and rebased this branch onto current main.

Fresh output and stored-WAL replay now use the same scoring projection. A structured verifier contributes scoring authority only after structuredVerifierGrade() validates it. For rejected legacy verifier data, replay only repairs failed, no-deadline rows that are identifiable as output from the old verifier-presence fallback. This keeps completed cells and benchmark deadlines authoritative, while tool_step_cap_reached remains eligible but unscored without a valid grade.

The regression coverage now includes missing, non-array, sparse, and reward-disagreeing attempts; the stale passed: true shape written by the old fallback; policy_denied; and preservation of tool-step-cap, completed, deadline, and no-verifier records.

Validation on the updated head:

  • npm --workspace @maka/headless test — 1,304 passed, 4 skipped
  • node --test packages/headless/dist/__tests__/fixed-prompt-controller.test.js — 89 passed
  • npm --workspace @maka/headless run typecheck
  • Biome check and git diff --check

The earlier E2E conflict is also gone after the rebase.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: MERGE-READY ✅ (2 independent reviewer passes)

DeepSeek review of the full diff including the incremental commits after the earlier approval (head 08354425):

  • Prior P2 (reproject rejected stored verifier grades) is properly addressed: the read-time reprojection uses the same taskCompletedScoringProjection as fresh writes, and idempotency is verified (728-combination corpus: 0 non-idempotent).
  • Legacy-reprojection discriminator is precise: exhaustive 728-combination audit (status × 13 errorClasses × reward × verifier shapes × deadline) shows exactly the 27 fallback-produced rows change (scored→unscored), zero collateral damage to completed/deadline/valid-grade/no-verifier rows; every presence-fallback row is caught.
  • Sparse-array validation closed: the new index-explicit loop rejects holes anywhere (including trailing holes via at(-1)), closing the old .some()/.slice().some() bypass. Zero effect on main-era stored data (JSON serializes holes to null) — purely defensive.
  • Verified state: typecheck clean, full headless suite 1308 pass / 1 env-skip / 0 fail on merge-tree with current main, biome clean, git diff --check clean. CI green.

Non-blocking P3 (optional):

  1. The discriminator keeps {scored:true, max_tokens, no verifier} rows as-is (differs from fresh path) — unreachable from main-era writers, conservative choice, worth a comment.
  2. Fresh-path policy_denied/tool_step_cap_reached + invalid-verifier variants and setup_failed/verification_error replay remap lack direct tests (same code path, low value).
  3. PR branch typechecks red against its own stale fork base (provider-request-trace.ts — fixed upstream by #1679, not this PR's fault); verify against current main instead.

None block merging.

@Astro-Han
Astro-Han merged commit 34e7ddb into apache:main Aug 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants