Sometimes, LLMs disagree with their own thoughts

[derekmisler]

Summary

Adds structured output schemas and decision rules to the PR review agent to ensure consistent verdicts across multiple runs. The agent now returns JSON-formatted findings and applies mechanical rules to determine whether to APPROVE, COMMENT, or REQUEST_CHANGES based on severity and verification status.

Changes

• review-pr/agents/pr-review.yaml: Added structured output schemas for drafter and verifier agents, enforcing JSON responses with required fields (file, line, severity, verdict, etc.). Introduced mandatory decision rules that map findings to review verdicts mechanically, preventing LLMs from overriding severity assessments. Added console output mode detection to support local testing without posting to GitHub.
• review-pr/agents/evals/: Added 6 eval files (3 for a security vulnerability, 3 for a clean PR) to test consistency. Each eval runs the same PR through the agent and verifies the verdict matches expectations.
• review-pr/README.md: Documented how to run the agent locally, how to run evals, and eval naming conventions (success-* for clean PRs, security-* for PRs with issues).
• .gitignore: Excluded eval results directory from version control.

Breaking Changes

None. The agent still posts reviews to GitHub when running in CI and outputs to console when running locally.

How to Test

• Run the evals: cagent eval review-pr/agents/pr-review.yaml review-pr/agents/evals/ -e GITHUB_TOKEN -e GH_TOKEN
• Verify that all 3 runs of security-redirect-uri-*.json produce REQUEST_CHANGES verdicts
• Verify that all 3 runs of success-*.json produce APPROVE verdicts

Closes: https://github.com/docker/gordon/issues/153

[derekmisler]

/describe

[docker-agent]

✅ PR description has been generated and updated!

[docker-agent]

Review Summary

No issues found! This PR adds comprehensive eval testing infrastructure and documentation.

Changes Reviewed

• ✅ .gitignore entry for eval results
• ✅ Documentation for running evals locally
• ✅ Six eval test files (3 security, 3 success) with proper JSON structure
• ✅ All eval files have valid UUIDs, proper relevance assertions, and test fixtures

The added code is structurally sound, well-documented, and ready to merge.