Skip to content

fix(eval): classify PrefEval judge answers like the reference benchmark - #2397

Open
jiangwu25 wants to merge 1 commit into
MemTensor:mainfrom
jiangwu25:fix/prefeval-error-type-classifier
Open

jiangwu25 wants to merge 1 commit into
MemTensor:mainfrom
jiangwu25:fix/prefeval-error-type-classifier

Conversation

@jiangwu25

@jiangwu25 jiangwu25 commented Sep 19, 2026

Copy link
Copy Markdown

Summary

PrefEval can treat a failed judge answer as Personalized Response.

This PR fixes the classifier to match the reference behavior:

  • invalid or missing answers become Judge Failure
  • Yes and No checks ignore case and extra punctuation
  • Unhelpful Response takes priority
  • all 16 standard answer patterns match the reference logic

Tests

  • 35 focused tests pass
  • Ruff, format checks, pre-commit, AST checks, and git diff --check pass

Fixes #2396

classify_error_type compared judge answers with exact 'Yes'/'No' strings and sent every unmatched case to the 'Personalized Response' bucket, including empty answers from failed judge calls and responses that were both violating and unhelpful. Move the classifier to evaluation/scripts/utils/pref_classify.py, mirror the amazon-science/PrefEval semantics, report unrecognized answers as 'Judge Failure', and add unit tests.
@Memtensor-AI Memtensor-AI added area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core MOS 编排层 / 框架底座 / 跨模块问题 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: PrefEval classifier counts failed or lowercase judge answers as "Personalized Response"

3 participants