feat: automated ai powered qa agent for pr reviews and commits on prs#830
feat: automated ai powered qa agent for pr reviews and commits on prs#830amaan-bhati wants to merge 3 commits intomainfrom
Conversation
Signed-off-by: amaan-bhati <amaanbhati49@gmail.com>
Signed-off-by: amaan-bhati <amaanbhati49@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds an automated “QA Review Agent” to run on PR events and post an AI-generated QA review comment, using a repo-specific QA guidelines document and a prebuilt dependency map to flag second-order risks.
Changes:
- Introduces a Python-based QA review script that fetches PR diffs, builds a structured prompt, calls Anthropic, and comments the results back on the PR.
- Adds a GitHub Actions workflow to run the agent on PR events, manual dispatch, and
/qa-reviewissue comments. - Adds repo-derived QA guidelines and a committed
codebase_map.jsonto support rule checking and downstream risk analysis.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
qa-agent/scripts/qa_review.py |
Implements the PR diff fetch → prompt build → Anthropic review → PR comment flow, plus “second-order” dependency hints. |
qa-agent/requirements.txt |
Adds Python dependencies for Anthropic + PyGithub. |
qa-agent/codebase_map.json |
Provides a static dependency map used for downstream risk reporting. |
qa-agent/QA_GUIDELINES.md |
Adds the rule/checklist framework the agent enforces in its output. |
.github/workflows/qa-review.yml |
Defines CI triggers and job steps to run the agent and post results on PRs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| github.event.issue.pull_request != null && | ||
| ( | ||
| contains(github.event.comment.body, '/qa-review') | ||
| ) |
There was a problem hiding this comment.
The issue_comment trigger allows anyone who can comment on a PR to run this workflow with repository secrets (including ANTHROPIC_API_KEY) and pull-requests: write, which is an easy vector for cost/abuse. Restrict execution to trusted users (e.g., author_association in {OWNER,MEMBER,COLLABORATOR} or a repo permission check) and/or require a label/maintainer-only command before running.
| ) | |
| ) && | |
| contains(fromJson('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association) |
| - name: Run QA review agent | ||
| env: | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| PR_NUMBER: ${{ steps.context.outputs.pr_number }} | ||
| GITHUB_REPOSITORY: ${{ github.repository }} | ||
| REVIEW_MODE: ${{ steps.context.outputs.mode }} | ||
| run: python qa-agent/scripts/qa_review.py |
There was a problem hiding this comment.
On pull_request events from forks, secrets.ANTHROPIC_API_KEY won’t be available, so this step will fail (and may block external contributors). Add a job/step guard to skip gracefully when required secrets are missing (e.g., if: secrets.ANTHROPIC_API_KEY != '') and optionally post a neutral comment explaining it was skipped.
| anthropic>=0.40.0 | ||
| PyGithub>=2.1.1 |
There was a problem hiding this comment.
Using open-ended version ranges (>=) for anthropic/PyGithub can make the QA workflow non-deterministic and break unexpectedly when upstream releases introduce breaking changes. Pin to known-good versions (or at least cap major versions) to keep CI stable and make updates intentional.
| anthropic>=0.40.0 | |
| PyGithub>=2.1.1 | |
| anthropic==0.40.0 | |
| PyGithub==2.1.1 |
|
|
||
| Triggered by GitHub Actions on: | ||
| - pull_request (opened, synchronize, reopened) | ||
| - push to any open PR |
There was a problem hiding this comment.
The module docstring lists “push to any open PR” as a trigger, but the current workflow/job conditions don’t run the agent on push events. Update the docstring (or the workflow) so the documented triggers match actual behavior; otherwise it will mislead maintainers debugging CI behavior.
| - push to any open PR |
| def run_review(prompt: str) -> str: | ||
| client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY) | ||
| message = client.messages.create( | ||
| model="claude-3-5-sonnet-20241022", | ||
| max_tokens=4000, | ||
| messages=[{"role": "user", "content": prompt}] | ||
| ) | ||
| return message.content[0].text |
There was a problem hiding this comment.
This workflow sends PR title/body + diff content to Anthropic. That’s a potential data-exfiltration vector if a PR contains secrets or sensitive code. Consider adding explicit opt-in controls (e.g., only run on labeled PRs / trusted authors), redacting common secret patterns before sending, and documenting this behavior for contributors to avoid unintentionally sharing sensitive data with a third party.
| elif [ "${{ github.event_name }}" = "issue_comment" ]; then | ||
| echo "pr_number=${{ github.event.issue.number }}" >> $GITHUB_OUTPUT | ||
| # check if comment says /qa-review fast | ||
| if echo "${{ github.event.comment.body }}" | grep -q "fast"; then |
There was a problem hiding this comment.
The fast mode detection matches any occurrence of the substring "fast" in the comment body (e.g., "breakfast"), which can unintentionally switch to fast mode. Consider parsing the command more strictly (e.g., match /qa-review fast as a token or anchor the regex) to avoid accidental mode changes.
| if echo "${{ github.event.comment.body }}" | grep -q "fast"; then | |
| if echo "${{ github.event.comment.body }}" | grep -Eq '(^|[[:space:]])/qa-review[[:space:]]+fast([[:space:]]|$)'; then |
| body = ( | ||
| f"## 🔍 QA Review\n\n" | ||
| f"{review_text}" | ||
| f"{skipped_note}\n\n" | ||
| f"---\n" | ||
| f"*qa-agent · [Guidelines](./qa-agent/QA_GUIDELINES.md) · " |
There was a problem hiding this comment.
The comment footer link [Guidelines](./qa-agent/QA_GUIDELINES.md) is a relative URL; in PR/issue comments this typically resolves relative to the PR page URL, not the repository root, so it won’t point to the file. Use an absolute GitHub URL (e.g., https://github.com/<owner>/<repo>/blob/<ref>/qa-agent/QA_GUIDELINES.md) or a repo-root absolute path (/owner/repo/blob/...) so the link works reliably.
| body = ( | |
| f"## 🔍 QA Review\n\n" | |
| f"{review_text}" | |
| f"{skipped_note}\n\n" | |
| f"---\n" | |
| f"*qa-agent · [Guidelines](./qa-agent/QA_GUIDELINES.md) · " | |
| guidelines_url = f"https://github.com/{repo.full_name}/blob/{pr.base.ref}/qa-agent/QA_GUIDELINES.md" | |
| body = ( | |
| f"## 🔍 QA Review\n\n" | |
| f"{review_text}" | |
| f"{skipped_note}\n\n" | |
| f"---\n" | |
| f"*qa-agent · [Guidelines]({guidelines_url}) · " |
| ) | ||
|
|
||
| permissions: | ||
| pull-requests: write |
There was a problem hiding this comment.
qa_review.py posts via pr.create_issue_comment(...), which uses the Issues comments API. This job’s permissions block grants pull-requests: write but not issues: write; with explicit permissions set, missing scopes are none, so the comment call may 403. Add issues: write (as done in .github/workflows/greetings.yml) or switch to a PR review comment API that matches the granted permissions.
| pull-requests: write | |
| pull-requests: write | |
| issues: write |
| Load the pre-built codebase dependency map. | ||
| This tells the agent which files depend on which, so it can do second-order analysis. | ||
| Built by build_codebase_map.py (run during agent setup or on schedule). |
There was a problem hiding this comment.
load_codebase_map() references a build_codebase_map.py generator, but that script isn’t included anywhere in qa-agent/ in this PR. Either add the generator (and document how/when to run it) or adjust the docstring to reflect the actual process for updating codebase_map.json, otherwise the map will drift and the “second-order analysis” will become misleading.
| Load the pre-built codebase dependency map. | |
| This tells the agent which files depend on which, so it can do second-order analysis. | |
| Built by build_codebase_map.py (run during agent setup or on schedule). | |
| Load the checked-in codebase dependency map. | |
| This tells the agent which files depend on which, so it can do second-order analysis. | |
| Keep codebase_map.json updated whenever dependency relationships change so the analysis stays accurate. |
| # --- configuration --- | ||
| ANTHROPIC_API_KEY = os.environ["ANTHROPIC_API_KEY"] | ||
| GITHUB_TOKEN = os.environ["GITHUB_TOKEN"] | ||
| PR_NUMBER = int(os.environ["PR_NUMBER"]) | ||
| REPO_NAME = os.environ["GITHUB_REPOSITORY"] | ||
| REVIEW_MODE = os.environ.get("REVIEW_MODE", "full") |
There was a problem hiding this comment.
Environment variables are read at import time via os.environ[...] (and int(os.environ["PR_NUMBER"])). In GitHub Actions (especially pull_request from forks or when the secret isn’t set), this will raise a KeyError before main() runs, causing the workflow to fail without a helpful message. Move env parsing into main() and use os.getenv with explicit validation; if required secrets are missing, exit cleanly (or skip the job) with a clear next step (e.g., set ANTHROPIC_API_KEY secret).
Signed-off-by: amaan-bhati <amaanbhati49@gmail.com>
amaan-bhati
changed the title
Implemented a fully autonomous QA Review Agent integrated directly into our GitHub Actions. The agent acts as an elite, automated repository maintainer that validates every Pull Request, especially those from Open Source contributors against a massive, highly specific rulebook derived from our Docusaurus/React architecture.
It prevents Docusaurus SSR crashes, eliminates styling fragmentations (enforcing Tailwind), and ensures all markdown files maintain strict SEO/MDX standards without requiring humans to catch every missing alt-tag or bad
window.locationcall.How It Validates Changes (The Workflow)
When an OSS contributor opens or pushes to a PR, the agent executes the following validation pipeline:
1. Diff & Context Gathering
The workflow (
qa-review.yml) triggers our Python agent. It securely fetches the contributor's diff via the GitHub API and loads our newly constructed 900-lineQA_GUIDELINES.mdrulebook into memory.2. Second-Order Impact Mapping
Before reading the code, the agent cross-references the changed files against our
codebase_map.jsondependency graph.Example: If a contributor modifies
QuickStartFilter.js, the agent automatically maps out every.mdxfile relying on that component to ensure no downstream props or layouts were broken by the isolated change.3. The 4-Pass AI Validation
The agent packages the diff, dependency graph, and guidelines, sending them to Claude (Anthropic). Claude executes 4 strict passes:
console.log, correct Frontmatter presence).ExecutionEnvironment.canUseDOMis used before running browser APIs).4. Automated Feedback Delivery
The agent logs back into the GitHub PR and posts its findings natively as a comment, organizing feedback by severity:
Setup
Requires
ANTHROPIC_API_KEYto be set within the Repository Action Secrets.(Note:
GITHUB_TOKENis natively injected).Architecture Origins & Verified Sources
This document serves as an exhaustive bibliography of the concepts, architectural patterns, and verified engineering sources used to design the Autonomous QA Review Agent.
1. The Idea: LLM Code Review & Dependency Graph Context
The Challenge: Out-of-the-box, Large Language Models (LLMs) struggle to effectively review code if they only see a single isolated file or git diff. If a developer edits
<QuickStartFilter />, the LLM has no idea what other files break. Injecting the entire repository into the LLM context limits performance and rapidly exceeds cost ceilings.The Solution: We combined LLM prompts with a Static Dependency Graph. Instead of blindly sending code, the Python script builds a map of the repository's imports. It performs a "topological lookup" to trace the blast radius of a change before compiling the final context window for Claude. This is an emerging industry standard for LLM code agents.
Verified Industry Sources & Research:
2. Designing the Python Automation (
qa_review.py)The Challenge: We needed a Python environment that runs securely inside a CI pipeline, gracefully intercepts GitHub webhooks, parses the PR diff, and structures a request.
The Solution: We utilized
PyGithubto interface securely via automated Action Tokens, specifically pulling the raw textual diff without needing to perform riskygit checkoutcommands on untrusted contributor forks.anthropicPython client to query Claude 3.5 via the Messages API securely.git clonerisks by directly scraping the.diff_urlpayload via GitHub's API.pathlib,re) for traversing imports without untrusted execution.pathlibReference3. Creating the GitHub Automation (
.github/workflows/qa-review.yml)The Challenge: The agent needed to run autonomously whenever Open Source (OSS) contributors push code, but it had to operate safely so malicious actors couldn't steal the environment secrets during execution.
The Solution: We bound the execution directly to the
.github/workflowssystem relying on thepull_requesttrigger. This native trigger deliberately restricts the scope of the pre-injected$GITHUB_TOKENto "read/comment-only" bounds specifically tied to the PR context.Verified Sources of Truth:
pull_requestoverpull_request_targetto safely sandbox untrusted fork executions. GitHub Actions: Security Guidelines for Fork PRs4. Designing the Ruleset (
QA_GUIDELINES.md)The Challenge: Knowing exactly what rules an LLM should aggressively hunt for within a Docusaurus/React architecture.
The Solution: We used anitgravity and codex to craft a massive Markdown file that acts as the "Brain" of the LLM. It focuses intensely on Static Site Generator (SSG) strictness, specifically the fact that Docusaurus is pre-rendered via Node.js before shipping to the client browser.
Verified Sources of Truth:
window.location) to be isolated behindExecutionEnvironment.canUseDOMor ReactuseEffecthooks. Docusaurus SSR Advanced GuideQA_GUIDELINES.md) from the Data (The Pull Request Diff) using XML-tags to prevent prompt injection or hallucination. Anthropic Prompt Engineering Interactive Tutorial