feat(judge): 增加只判可数四问的判官与 shadow 闸口 - #324
Conversation
四问(主体数/凭空物体/动作类别/是否被裁)各有唯一答案,出参刻意不含 score; 闸口默认关,启用后为 shadow 档只记不拦,判官故障一律不拦截。
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![fennoai[bot]](https://avatars.githubusercontent.com/in/1924301?s=60&v=4){kind=small}
There was a problem hiding this comment.
Reviewed the quality-judge contract, provider parsing/wiring, shadow/enforce behavior, task-result persistence, and the SufyImageProvider refactor across the fixed range. The main flow is covered by the added tests and the defaults preserve shadow-off behavior, but custom actions currently lose their required action description before judging; see the inline finding. Focused tests were not runnable in this workspace because neither uv nor pytest is installed.
| "frames": frames, | ||
| } | ||
| decision = quality_gate.review( | ||
| self._get_judge(), checked, master, input.action_type.value |
There was a problem hiding this comment.
[P1] Pass the custom action description to the judge
When action_type is CUSTOM, the API requires and the generator uses input.custom_prompt (for example, wave hello), but this call passes only the enum value "custom" into the judge prompt. The model therefore cannot evaluate whether the frame matches the requested custom action; with QUALITY_GATE_ENFORCE=true it will commonly report action_matches=false and block otherwise valid custom-action tasks, while shadow metrics for all custom actions are meaningless. Pass the custom prompt for custom actions and retain the enum label for the built-in actions.
1 participant
Refs #323
改了什么
在出口处加一道判官,并把它接进动作任务的组装流程。
common/models/quality.pyJudgeVerdict—— 四个读数,没有 score 字段ai_engine/ports/__init__.pyJudgePort协议framework/providers/judge.pySufyJudgeProvider+ 解析,读不出结论抛JudgeResponseErrorframework/providers/sufy.pyChatCompletionsFace,出图与判官共用建连/重试/400-404 提示framework/config/quality_gate.pyQUALITY_GATE_ENABLED/QUALITY_GATE_ENFORCE,均默认 falseframework/config/provider.pyjudge_modelapp/server/orchestrator/quality_gate.pyreview()→GateDecision | Noneapp/server/orchestrator/executor.pyresult["judge"]tests/test_quality_judge.py为什么
出口处此前没有任何自动检查:帧上传完就组 payload、无条件置 COMPLETED。一段把角色分裂成两个人的 walk,与一段干净的 walk,在任务状态里长得完全一样。
判官只问四个有唯一答案的问题(主体数 / 凭空多出的物体 / 姿态属不属于该动作类别 / 有没有被画面边缘裁到),刻意不出分数:主观评分的噪声能盖过我们想比较的真实差异,而出参里一旦有分数,迟早有人拿它卡阈值——那时卡掉的是噪声,误杀的是用户已付费、退不回来的产物。理由展开在 Issue 正文。
三条对应的实现红线:
review()返回GateDecision | None。None= 没判,与"判了没问题"必须可分辨——只有后者能支持"这批产物是干净的"这句话。error并放行。因为我们自己的仪器坏了而扣住用户已付费的产物,是最糟的失败方式。同理 provider 不得兜底成"通过"。怎么验证的
cd backend:uv run ruff check .→ All checks passeduv run lint-imports→ Contracts: 2 kept, 0 brokenuv run pytest -q→ 721 passed, 14 skipped(全仓;判官相关 40 条)测试里两条是先验仪器:桩不是写死的一条回答,而是真去数提交上来的图里有几个主体的假模型;另有一条单独断言合成图本身数得出单人/双人的差别——否则那两条控制样本就是同一个断言跑了两遍。写死的桩发现不了请求形状错、base64 错、参考图顺序错。
其余覆盖:11 种读不出的回答一律抛错(含
subject_count: true——bool 是 int 的子类,不排掉会被读成 1 个主体这么一个凭空捏造的读数);闸口关时一次调用都不发且result里没有读数;判官抛错时任务照常 COMPLETED 并带error;enforce开时任务 FAILED 且原因带问题标签;JudgeVerdict字段名里没有score/rating/grade;共用管道后出图那条放大过的超时没被改掉。没有跑任何付费生成。
与在飞 PR 的关系
result["judge"],不是result["quality"]。#311(成色落库)那条要往同一个 payload 写引擎本地算出的ActionQuality,共用一个键会让先写的那份被后写的悄悄盖掉;两份数据的来源与代价都不同,分开落。sufy.py抽ChatCompletionsFace是本 PR 唯一动到既有代码的地方,SufyImageProvider的对外形状与超时倍数不变,有测试盯着。