fix(headless): lock valid structured verifier failures as scored results - #1591
Conversation
Astro-Han
left a comment
There was a problem hiding this comment.
Approving current head 5765174 with a non-blocking P2 note.
Malformed or older verifier WAL data with a missing/non-array attempts field can throw a raw TypeError during projection and block resume. The shared verifier classification boundary should validate the runtime shape and map malformed data to the existing ungraded/infra path. The required test check is still pending.
Treat valid structured pass and failure outcomes as authoritative when the agent cell exits unsuccessfully. Validate reward and final-attempt agreement, and project stored failures as scored so they remain locked on resume. Refs apache#1585
Treat verifier data from runners and persisted WAL as untrusted runtime input. Missing or non-array attempts now stay on the existing ungraded path instead of throwing during resume.
5765174 to
3a8026f
Compare
|
Addressed the P2 in 3a8026f. The shared verifier classifier now treats runner and WAL payloads as untrusted runtime input, validates the harbor/verifier/attempt shapes before reading them, and returns the existing ungraded result for malformed data. The regression test covers both a missing |
Astro-Han
left a comment
There was a problem hiding this comment.
P2: taskCompletedEvent still treats any present harbor.verifier as scoring authority, even when structuredVerifierGrade() rejects it. On this head, { outcome: 'failed' } and non-array attempts both turn a max_tokens failure into task_completed with scored and eligible set to true. Sparse arrays also pass because Array#some skips holes. Please make the validated grade the only scoring input and cover malformed and sparse runner output.
|
Followed up in #1644. I removed the verifier-presence fallback, so runner data only affects scoring after The runner regressions cover missing and non-array attempts, a sparse array, and an internally inconsistent failed grade with a positive aggregate reward. Invalid verifier data remains ungraded, and a provider network failure still stays on the infrastructure path. The branch is rebased onto current main; build, lint, format, typecheck, and the full Headless suite pass locally. |
Summary
Closes #1585.
Verification
npm run build:testnpm --workspace @maka/headless test— 1,427 passed, 4 skipped because Harbor/Pier Python is unavailable, 0 failednpm run typechecknpm run lintnpm run format:checkgit diff --checkReview focus
#1261 is a mechanical WAL-type extraction that also touches
fixed-prompt-controller.ts. This PR keeps the scoring change and its regression coverage separate; I will rebase it onto the extracted type seam after #1261 lands.