fix(headless): require validated verifier grades - #1644
Conversation
Astro-Han
left a comment
There was a problem hiding this comment.
[P2] Reproject stale fallback scores when the stored verifier is rejected
readFixedPromptWal() sends persisted task_completed events through projectStructuredVerifierOutcome(). If structuredVerifierGrade() rejects the stored verifier, the event is returned with its old score flags unchanged.
I reproduced this with a legacy failed max_tokens event written by the previous verifier-presence fallback. scored: true and eligible: true survive resume, the runner is not called, and the event still counts toward the acceptance denominator. The malformed-WAL test currently starts with both flags set to false, so it misses this case.
Please recalculate the identifiable failed/no-deadline/invalid-grade legacy records with the same rules used for fresh output. The existing authority for tool_step_cap_reached, completed cells, and deadlines should stay intact.
Approving with this P2. I would hold the merge until it is addressed.
|
Btw the E2E test has been fixed on main. Rebase to main would solve the Red CI. |
Validate every attempt index explicitly so array holes cannot bypass sequence and infrastructure-prefix checks.
Remove the verifier-presence fallback so malformed runner output cannot mark max-token failures scored or eligible.
ecf7a42 to
0835442
Compare
|
Addressed the replay P2 and rebased this branch onto current Fresh output and stored-WAL replay now use the same scoring projection. A structured verifier contributes scoring authority only after The regression coverage now includes missing, non-array, sparse, and reward-disagreeing attempts; the stale Validation on the updated head:
The earlier E2E conflict is also gone after the rebase. |
Astro-Han
left a comment
There was a problem hiding this comment.
Review: MERGE-READY ✅ (2 independent reviewer passes)
DeepSeek review of the full diff including the incremental commits after the earlier approval (head 08354425):
- Prior P2 (reproject rejected stored verifier grades) is properly addressed: the read-time reprojection uses the same
taskCompletedScoringProjectionas fresh writes, and idempotency is verified (728-combination corpus: 0 non-idempotent). - Legacy-reprojection discriminator is precise: exhaustive 728-combination audit (status × 13 errorClasses × reward × verifier shapes × deadline) shows exactly the 27 fallback-produced rows change (scored→unscored), zero collateral damage to completed/deadline/valid-grade/no-verifier rows; every presence-fallback row is caught.
- Sparse-array validation closed: the new index-explicit loop rejects holes anywhere (including trailing holes via
at(-1)), closing the old.some()/.slice().some()bypass. Zero effect on main-era stored data (JSON serializes holes to null) — purely defensive. - Verified state: typecheck clean, full headless suite 1308 pass / 1 env-skip / 0 fail on merge-tree with current main, biome clean,
git diff --checkclean. CI green.
Non-blocking P3 (optional):
- The discriminator keeps
{scored:true, max_tokens, no verifier}rows as-is (differs from fresh path) — unreachable from main-era writers, conservative choice, worth a comment. - Fresh-path
policy_denied/tool_step_cap_reached+ invalid-verifier variants andsetup_failed/verification_errorreplay remap lack direct tests (same code path, low value). - PR branch typechecks red against its own stale fork base (
provider-request-trace.ts— fixed upstream by #1679, not this PR's fault); verify against current main instead.
None block merging.
Summary
Follow-up to #1591.
Verification
npm run build:testnpm run lintnpm run format:checknpm --workspace @maka/headless run typechecknpm --workspace @maka/headless run test:dist— 1,429 passed, 4 environment skips, 0 failedgit diff --checkReview focus
This changes only verifier-derived scoring authority. Existing completed-cell, benchmark-deadline, and tool-step eligibility behavior remains unchanged; a verifier influences classification only when
structuredVerifierGrade()accepts the full outcome/reward/attempt contract.