Skip to content

fix(headless): lock valid structured verifier failures as scored results - #1591

Merged
Astro-Han merged 2 commits into
apache:mainfrom
me2seeks:fix/1585-structured-verifier-failure
Jul 30, 2026
Merged

fix(headless): lock valid structured verifier failures as scored results#1591
Astro-Han merged 2 commits into
apache:mainfrom
me2seeks:fix/1585-structured-verifier-failure

Conversation

@me2seeks

Copy link
Copy Markdown
Contributor

Summary

  • treat valid structured verifier pass and failure outcomes as authoritative even when the agent cell exits unsuccessfully
  • require the structured outcome, aggregate reward, and final verifier attempt to agree before overriding an infrastructure classification
  • project stored structured failures as scored results so resumed runs do not resample them
  • cover Harbor/Pier parity, infrastructure-only verifier outcomes, and WAL resume behavior

Closes #1585.

Verification

  • npm run build:test
  • npm --workspace @maka/headless test — 1,427 passed, 4 skipped because Harbor/Pier Python is unavailable, 0 failed
  • npm run typecheck
  • npm run lint
  • npm run format:check
  • git diff --check

Review focus

#1261 is a mechanical WAL-type extraction that also touches fixed-prompt-controller.ts. This PR keeps the scoring change and its regression coverage separate; I will rebase it onto the extracted type seam after #1261 lands.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving current head 5765174 with a non-blocking P2 note.

Malformed or older verifier WAL data with a missing/non-array attempts field can throw a raw TypeError during projection and block resume. The shared verifier classification boundary should validate the runtime shape and map malformed data to the existing ungraded/infra path. The required test check is still pending.

me2seeks added 2 commits July 30, 2026 09:43
Treat valid structured pass and failure outcomes as authoritative when the agent cell exits unsuccessfully. Validate reward and final-attempt agreement, and project stored failures as scored so they remain locked on resume.

Refs apache#1585
Treat verifier data from runners and persisted WAL as untrusted runtime input. Missing or non-array attempts now stay on the existing ungraded path instead of throwing during resume.
@me2seeks
me2seeks force-pushed the fix/1585-structured-verifier-failure branch from 5765174 to 3a8026f Compare July 30, 2026 01:47
@me2seeks

Copy link
Copy Markdown
Contributor Author

Addressed the P2 in 3a8026f. The shared verifier classifier now treats runner and WAL payloads as untrusted runtime input, validates the harbor/verifier/attempt shapes before reading them, and returns the existing ungraded result for malformed data. The regression test covers both a missing attempts field and a non-array value; each resumes without resampling and remains scored=false / eligible=false.\n\nThe branch is rebased onto current main. Headless typecheck, lint, format, the focused 76-test controller suite, and the full Headless suite (1435 tests, 0 failures, 4 environment skips) pass locally.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: taskCompletedEvent still treats any present harbor.verifier as scoring authority, even when structuredVerifierGrade() rejects it. On this head, { outcome: 'failed' } and non-array attempts both turn a max_tokens failure into task_completed with scored and eligible set to true. Sparse arrays also pass because Array#some skips holes. Please make the validated grade the only scoring input and cover malformed and sparse runner output.

@Astro-Han
Astro-Han merged commit 73b629f into apache:main Jul 30, 2026
3 checks passed
@me2seeks

Copy link
Copy Markdown
Contributor Author

Followed up in #1644. I removed the verifier-presence fallback, so runner data only affects scoring after structuredVerifierGrade() accepts the complete outcome/reward/attempt contract. The classifier now checks every array index explicitly, which rejects sparse attempts instead of letting Array#some skip holes.

The runner regressions cover missing and non-array attempts, a sparse array, and an internally inconsistent failed grade with a positive aggregate reward. Invalid verifier data remains ungraded, and a provider network failure still stays on the infrastructure path. The branch is rebased onto current main; build, lint, format, typecheck, and the full Headless suite pass locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(headless): lock valid structured verifier failures as scored results

2 participants