feat(evals): tell a judge block which call it is serving

[TonsOfFun]

Judge calls its host-supplied block with only instructions: and prompt:, but it serves three different calls — scoring an answer, recommending a fix for a failed scenario, and writing the run's verdict. A host that wants to trace them apart, budget or rate-limit them, or grade with a cheaper model than it writes the verdict with has no way to tell which one it is in.

The only signal was the instructions string, so hosts matched against the gem's own constants:

when ActiveAgent::Evals::Judge::RECOMMEND_INSTRUCTIONS then 'recommend'

That couples a host to prose the gem is free to reword — and rewording it does not break anything loudly. Every such host falls through to its else branch and mislabels traces, which is worse than an error.

Changes

• A block that accepts kind: now receives :score, :recommend or :verdict. The four ask call sites already knew which they were; threading a symbol through was mechanical.
• The keyword reaches only a block that names it or collects **, decided by Proc#parameters and memoized per judge.

Why the arity check is not optional

The block signature is public API, and a block taking exactly instructions: and prompt: raises ArgumentError on a third keyword. That would not even surface as an error: ask rescues StandardError, so the raise would be swallowed by warn_failure and counted as a judge failure — silently degrading every run to rule scoring. Passing it unconditionally is a breaking change wearing a disguise, so it waits for a major if it is ever wanted.

The existing judge suite is itself the backward-compatibility proof: its fake_judge helper builds a two-keyword block, and all of it still passes untouched.

UAT Steps

1. BUNDLE_GEMFILE=gemfiles/rails8.gemfile bin/test test/evals/judge_call_kind_test.rb test/evals/judge_test.rb — 10 runs, 0 failures.
2. All evals tests: ... bin/test test/evals/*_test.rb — 133 runs, 0 failures.
3. bin/lint — 552 files, no offenses.
4. Regression check: revert lib/active_agent/evals/judge.rb and re-run (1) — 4 of the 5 new tests fail. The fifth is the legacy two-keyword case, which is expected to pass either way.

Screenshots

N/A — no visual surface.

Closes

Closes #462

Special Handling

• This PR requires user testing
• Include this PR in the changelog

🤖 Generated with Claude Code

https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V

---

Generated by Claude Code