Sync agenteye-evaluator skill (agenteye@3ddd9de)#9
Conversation
NiveditJain
added
the
skill-sync-agenteye-evaluator
📝 WalkthroughWalkthroughThe evaluator skill now documents separate plan and build modes, adds a detailed brainstorm reference for producing evaluation plans from logs, and clarifies SDK installation guidance. ChangesEvaluator skill workflows
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
16b3f6b to
78407ff
Compare
Automated mirror of evaluator-sdk/skill/. Source of truth: FailproofAI/agenteye — do not hand-edit.
78407ff to
89dc7a9
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/agenteye-evaluator/references/brainstorm.md`:
- Around line 130-138: Update the SQL example in the population-confirmation
section to explicitly compare good and bad session cohorts, using reviewed
session-ID lists or equivalent verdict-based cohort membership. Ensure the
output reports the metric by cohort so it can demonstrate separation, or clearly
relabel the query as feature extraction rather than discrimination evidence.
- Around line 163-171: Update the Filter A guidance and example around the
agenteye events query to measure payload-field coverage across the full
candidate population rather than a single session. Add a population-level query
that reports both sessions with the field populated and the total
eligible-session denominator, and require that coverage result before retaining
the dimension.
- Around line 90-94: Update the session inspection guidance around the `agenteye
--json events` command to explicitly page through results beyond the `--limit
1000` cap using the documented cursor mechanism. Require verifying that the
complete event stream has been retrieved before analyzing or cataloging a
session, while preserving the payload-free-first and selective `--full`
workflow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: bc74dde9-6a53-437d-9cc9-81d9da3d47d5
📒 Files selected for processing (3)
skills/agenteye-evaluator/SKILL.mdskills/agenteye-evaluator/references/brainstorm.mdskills/agenteye-evaluator/references/scaffold.md
| Pull mechanics are in [`session-data.md`](session-data.md): the export endpoint gives | ||
| a byte-identical fixture, or `agenteye --json events --full --session-id <id> --order | ||
| asc --all --limit 1000`. Skim on the payload-free feed first (its one-line server | ||
| `summary` per event is ideal for reading a session's *shape*), then `--full` only on | ||
| the ones you're reading line by line. **Confirm the corpus with the user** before you |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Page selected sessions until the complete event stream is retrieved.
This workflow hard-caps session reads at --limit 1000, while the later CLI guidance says larger results require cursor paging. A session with more than 1,000 events can therefore be misclassified from incomplete evidence. Explicitly instruct the agent to continue paging and verify that the session is complete before cataloging it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/agenteye-evaluator/references/brainstorm.md` around lines 90 - 94,
Update the session inspection guidance around the `agenteye --json events`
command to explicitly page through results beyond the `--limit 1000` cap using
the documented cursor mechanism. Require verifying that the complete event
stream has been retrieved before analyzing or cataloging a session, while
preserving the payload-free-first and selective `--full` workflow.
| Then don't stop at the sample — **confirm it holds across the population.** The metric | ||
| that looked decisive in two sessions may wash out over two hundred. Where the metric | ||
| is expressible in SQL, compute it per session and compare the cohorts: | ||
|
|
||
| ```bash | ||
| # does "how it ended" actually track the user's good/bad call? | ||
| agenteye --json query run --sql \ | ||
| "SELECT session_id, countIf(event_type='agent_end') AS ended, count() AS events | ||
| FROM events GROUP BY session_id ORDER BY ended ASC, events DESC LIMIT 30" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make the discrimination query compare good and bad cohorts.
The example query only reports agent_end and event counts per session; it never groups sessions by the user’s good/bad verdict. As written, it can show a distribution but cannot establish the required good-vs-bad separation. Add explicit cohort membership—such as reviewed session-ID lists—or label the query as a feature-extraction step rather than discrimination evidence.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/agenteye-evaluator/references/brainstorm.md` around lines 130 - 138,
Update the SQL example in the population-confirmation section to explicitly
compare good and bad session cohorts, using reviewed session-ID lists or
equivalent verdict-based cohort membership. Ensure the output reports the metric
by cohort so it can demonstrate separation, or clearly relabel the query as
feature extraction rather than discrimination evidence.
| **Filter A — is the signal in the transcript, and actually populated?** Name the exact | ||
| event types and payload fields that feed the dimension. Then *prove the field is there | ||
| in their data* — the schema allowing a field doesn't mean their instrumentation emits | ||
| it: | ||
|
|
||
| ```bash | ||
| agenteye --json events --full --session-id <id> --all --limit 1000 \ | ||
| | jq '[.events[] | select(.event_type=="tool_use") | .payload.tool_name] | unique' | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validate field coverage across the population, not one session.
The Filter A example checks payload presence for a single <id>, despite requiring that the signal be “actually populated.” A candidate can pass this check while the field is absent from most sessions. Add a population-level coverage query and report the populated-session denominator before retaining the dimension.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/agenteye-evaluator/references/brainstorm.md` around lines 163 - 171,
Update the Filter A guidance and example around the agenteye events query to
measure payload-field coverage across the full candidate population rather than
a single session. Add a population-level query that reports both sessions with
the field populated and the total eligible-session denominator, and require that
coverage result before retaining the dimension.
2 participants
Automated mirror — do not hand-edit this folder.
FailproofAI/agenteye→evaluator-sdk/skill/skills/agenteye-evaluator/main@3ddd9deThe skills repo's
validate-skills.pypassed. To change the skill, editevaluator-sdk/skill/in agenteye and re-run the Sync agenteye-evaluator skill workflow — this same PR refreshes.Summary by CodeRabbit