Skip to content

Enforce scoped validation tools in AGENTS.md with file-based trap evals - #12297

Open
reidbaker-agent wants to merge 8 commits into
mainfrom
fix-validation-tooling-guidance
Open

Enforce scoped validation tools in AGENTS.md with file-based trap evals#12297
reidbaker-agent wants to merge 8 commits into
mainfrom
fix-validation-tooling-guidance

Conversation

@reidbaker-agent

@reidbaker-agent reidbaker-agent commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

This pr gives us a way of writing unit tests for agent.md. - @reidbaker

Pr is blocked by a preexisting zizmor issue #12298 and by #12285 which is the pr that introduces evals.


Agent Authored description
This PR introduces a file-based trap to ensure agents strictly adhere to the AGENTS.md guidelines for validating local changes. Instead of running slow, repository-wide CI scripts like .ci/scripts/validate.sh or script/tool_runner.sh, agents must use targeted skills or package-scoped tools.

Evaluation Results: check-readiness / Validation rules

Eval ID: agents_md_compliance_test

  • Status: PASS 🟢
  • Scenario: Tests if the agent avoids global scripts when asked to validate the package.
  • Grading:
    • The agent successfully validated the package using dart format ., dart analyze ., and flutter test.
    • The agent avoided executing the global CI scripts.
    • The file USED_FORBIDDEN_SCRIPT.txt was not generated.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces evaluation configurations, rubrics, simulation scripts, and structural consistency tests for the run-evals skill in the camera_android_camerax package and the agent-plugins repository. The review feedback identifies a potential test failure in mono-repo environments due to hardcoded path assumptions, a silent test bypass, and destructive global file overwrites in the setup script that also trigger rubric failures. Suggestions are provided to robustly locate the package root and safely mock global scripts.

Comment thread packages/camera/camera_android_camerax/test/skills_evals_test.dart
Comment thread packages/camera/camera_android_camerax/test/skills_evals_test.dart
@reidbaker
reidbaker force-pushed the fix-validation-tooling-guidance branch from 5f69d6d to 0fbe29a Compare July 28, 2026 13:19
@reidbaker

Copy link
Copy Markdown
Contributor

Blocked on #12298

@reidbaker
reidbaker requested a review from camsim99 July 28, 2026 14:14
@reidbaker
reidbaker marked this pull request as draft July 28, 2026 14:15
@reidbaker
reidbaker force-pushed the fix-validation-tooling-guidance branch from 0fbe29a to c8e22c5 Compare July 28, 2026 14:53
@reidbaker reidbaker added the CICD Run CI/CD label Jul 28, 2026
@flutter-dashboard flutter-dashboard Bot removed the CICD Run CI/CD label Jul 28, 2026
@reidbaker
reidbaker marked this pull request as ready for review July 28, 2026 15:25

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces validation guidelines in AGENTS.md to prevent running global CI scripts, adds a compliance evaluation test, and provides a setup script to trap forbidden script executions. Additionally, it updates skills_evals_test.dart to resolve the package root dynamically and enforce the existence of the evals/ directory and rubric files. Feedback highlights a critical issue in the setup script where single quotes prevent the expansion of $REPO_ROOT, which would cause the trapped scripts to attempt writing to the root directory.

Comment thread packages/camera/camera_android_camerax/evals/test_data/setup_agents_md_eval.sh Outdated
Comment thread packages/camera/camera_android_camerax/evals/test_data/setup_agents_md_eval.sh Outdated
auto-submit Bot pushed a commit that referenced this pull request Jul 28, 2026
This pr is blocked by #12298. 

This came up in #12297. I decided landing the tool focused pr was much easier to review than stacking this work on #12285 - @reidbaker 

---
Agent authored pr description
This PR exempts .agents and evals directories from triggering missing changelog and version update errors.

TAG=agy
CONV=24c0c9b5-7e8d-407c-ae84-a577061bd6ff
@reidbaker reidbaker added the CICD Run CI/CD label Jul 28, 2026

@camsim99 camsim99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread packages/camera/camera_android_camerax/AGENTS.md Outdated
Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com>
@flutter-dashboard flutter-dashboard Bot removed the CICD Run CI/CD label Jul 28, 2026
@reidbaker reidbaker added CICD Run CI/CD autosubmit Merge PR when tree becomes green via auto submit App labels Jul 28, 2026
@auto-submit auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jul 28, 2026
@auto-submit

auto-submit Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

autosubmit label was removed for flutter/packages/12297, because This PR has not met approval requirements for merging. The PR author is not a member of flutter-hackers and needs 1 more review(s) in order to merge this PR.

  • Merge guidelines: A PR needs at least one approved review if the author is already part of flutter-hackers or two member reviews if the author is not a member of flutter-hackers before re-applying the autosubmit label. Reviewers: If you left a comment approving, please use the "approve" review action instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants