feat: automated ai powered qa agent for pr reviews and commits on prs

.github/workflows/qa-review.yml

@@ -0,0 +1,89 @@
+name: QA Review Agent
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
+
+  push:
+    branches-ignore:
+      - main
+      - master
+
+  issue_comment:
+    types: [created]
+
+  workflow_dispatch:
+    inputs:
+      pr-number:
+        description: "PR number to review"
+        required: true
+      review-mode:
+        description: "full (with second-order analysis) or fast"
+        required: false
+        default: "full"
+
+jobs:
+  # NOTE: update-codebase-map job which automatically commits back changes has been omitted 
+  # to satisfy constraints regarding git commits to GitHub. We expect manual codebase map rebuilds.
+
+  # ---------------------------------------------------------------
+  # the actual PR review
+  # ---------------------------------------------------------------
+  qa-review:
+    name: QA Review
+    runs-on: ubuntu-latest
+
+    if: |
+      github.event_name == 'pull_request' ||
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event_name == 'issue_comment' &&
+        github.event.issue.pull_request != null &&
+        (
+          contains(github.event.comment.body, '/qa-review')
+        )
+      )
+
+    permissions:
+      pull-requests: write
+      contents: read
+
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install -r qa-agent/requirements.txt
+
+      - name: Resolve PR number and review mode
+        id: context
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            echo "pr_number=${{ github.event.inputs.pr-number }}" >> $GITHUB_OUTPUT
+            echo "mode=${{ github.event.inputs.review-mode }}" >> $GITHUB_OUTPUT
+          elif [ "${{ github.event_name }}" = "issue_comment" ]; then
+            echo "pr_number=${{ github.event.issue.number }}" >> $GITHUB_OUTPUT
+            # check if comment says /qa-review fast
+            if echo "${{ github.event.comment.body }}" | grep -q "fast"; then
+              echo "mode=fast" >> $GITHUB_OUTPUT
+            else
+              echo "mode=full" >> $GITHUB_OUTPUT
+            fi
+          else
+            echo "pr_number=${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
+            echo "mode=full" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Run QA review agent
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUMBER: ${{ steps.context.outputs.pr_number }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+          REVIEW_MODE: ${{ steps.context.outputs.mode }}
+        run: python qa-agent/scripts/qa_review.py

qa-agent/README.md

@@ -0,0 +1,66 @@
+# QA Agent
+
+Automated QA review system. Reviews every PR and commit against guidelines derived from this codebase's own patterns.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `QA_GUIDELINES.md` | The rules. Derived from codebase analysis. Edit via PR. |
+| `codebase_map.json` | Dependency graph. Updated manually. |
+| `scripts/qa_review.py` | Review agent — runs on every PR |
+| `scripts/build_codebase_map.py` | Builds the dependency graph |
+| `requirements.txt` | Python dependencies |
+
+## How it works
+
+1. PR opened or commit pushed → GitHub Actions triggers
+2. Agent loads `QA_GUIDELINES.md`
+3. Agent loads `codebase_map.json` (pre-built dependency graph)
+4. Agent fetches the PR diff
+5. Agent finds all files that depend on changed files (second-order analysis)
+6. Agent sends everything to Claude with a structured review prompt
+7. Claude runs four passes: checklist, rule check, breakage analysis, test coverage
+8. Review posted as PR comment
+
+## Triggers
+
+| Event | What happens |
+|---|---|
+| PR opened | Full review runs automatically |
+| Commit pushed to open PR | Full review runs automatically |
+| Comment `/qa-review` on PR | Re-runs full review |
+| Comment `/qa-review fast` | Re-runs without second-order analysis |
+| `workflow_dispatch` | Manual — enter PR number in Actions tab |
+
+## Updating the guidelines
+
+Edit `QA_GUIDELINES.md` via a PR. The agent will review its own guidelines update for consistency. Guidelines changes should be reviewed by a human before merging.
+
+## Environment Variables & Secrets Required
+
+The QA Agent relies on specific environment variables to authenticate with the language model and the GitHub API. 
+
+### 1. `ANTHROPIC_API_KEY` (Required Secret)
+The API key used to query the Claude model for QA checks.
+- **Where to configure**: In your GitHub repository, go to `Settings` -> `Secrets and variables` -> `Actions` -> `New repository secret`.
+- **Name**: `ANTHROPIC_API_KEY`
+- **Example Value**: `sk-ant-api03-XXXXXXXXXXXXXXXXXXXXXXX`
+
+### 2. `GITHUB_TOKEN` (Automatically Provided)
+Used by the `PyGithub` client to read PR diffs and post comments.
+- **Where to configure**: None needed! GitHub Actions injects this automatically.
+- **Example Value**: `${{ secrets.GITHUB_TOKEN }}` (Visible in `.github/workflows/qa-review.yml`).
+
+### 3. `REVIEW_MODE` (Optional Environment Variable)
+Determines if the agent runs a full topological check or skips it for speed.
+- **Where to configure**: Passed natively in the Workflow file or triggered via issue comments.
+- **Example Value**: `full` or `fast`
+- **Usage**: Comment `/qa-review fast` on a PR to bypass second-order risk analysis.
+
+## Running the map builder locally
+
+```bash
+pip install -r qa-agent/requirements.txt
+python qa-agent/scripts/build_codebase_map.py
+```

qa-agent/codebase_map.json

@@ -0,0 +1,71 @@
+{
+  "src/components/responsive-player/ResponsivePlayer.js": [
+    "src/components/index.js"
+  ],
+  "src/components/shared/Button.js": [
+    "src/components/index.js"
+  ],
+  "src/components/KeployCloud.js": [
+    "src/components/index.js"
+  ],
+  "src/components/Community.js": [
+    "src/components/index.js"
+  ],
+  "src/components/Intro.js": [
+    "src/components/index.js"
+  ],
+  "src/components/Resources.js": [
+    "src/components/index.js"
+  ],
+  "src/components/QuickStart.js": [
+    "src/components/index.js"
+  ],
+  "src/components/Hacktoberfest.js": [
+    "src/components/index.js"
+  ],
+  "src/components/GetStartedPaths.js": [
+    "src/components/index.js"
+  ],
+  "src/components/TestingCapabilities.js": [
+    "src/components/index.js"
+  ],
+  "src/components/QuickStartTabs.js": [
+    "src/components/index.js"
+  ],
+  "src/components/WhatIsKeploy.js": [
+    "src/components/index.js"
+  ],
+  "src/components/EcosystemSupport.js": [
+    "src/components/index.js"
+  ],
+  "src/components/DocHeaderChips.js": [
+    "src/components/index.js"
+  ],
+  "src/components/SidebarBadge.js": [
+    "src/components/index.js"
+  ],
+  "src/components/SidebarCategoryIcon.js": [
+    "src/components/index.js"
+  ],
+  "src/components/TierCallout.js": [
+    "src/components/index.js"
+  ],
+  "src/components/QuickStartList.js": [
+    "src/components/QuickStartFilter.js"
+  ],
+  "versioned_docs/version-2.0.0/server/sdk-installation/index.md": [
+    "versioned_docs/version-2.0.0/server/sdk-installation/go.md",
+    "versioned_docs/version-2.0.0/server/sdk-installation/java.md",
+    "versioned_docs/version-2.0.0/server/sdk-installation/javascript.md",
+    "versioned_docs/version-2.0.0/server/sdk-installation/python.md"
+  ],
+  "versioned_docs/version-1.0.0/go/supported-frameworks.md": [
+    "versioned_docs/version-1.0.0/go/integration.md"
+  ],
+  "versioned_docs/version-1.0.0/go/integration-with-go-test.md": [
+    "versioned_docs/version-1.0.0/go/replay.md"
+  ],
+  "versioned_docs/version-1.0.0/java/integration-with-junit.md": [
+    "versioned_docs/version-1.0.0/java/replay.md"
+  ]
+}

qa-agent/requirements.txt

@@ -0,0 +1,2 @@
+anthropic>=0.40.0
+PyGithub>=2.1.1

qa-agent/scripts/build_codebase_map.py

@@ -0,0 +1,131 @@
+"""
+Codebase Map Builder
+Scans the repository and builds a reverse dependency map:
+  { "src/utils/auth.js": ["src/pages/login.js", "src/components/guard.js", ...] }
+
+This tells the QA agent: "if auth.js changes, these files might be affected."
+
+This is supposed to run when:
+  - Once during initial setup
+  - In CI on merges to main (to keep the map current) (omitted per user instructions)
+  - Output: qa-agent/codebase_map.json
+"""
+
+import os
+import re
+import json
+from pathlib import Path
+from collections import defaultdict
+
+ROOT = Path(".")
+OUTPUT = Path("qa-agent/codebase_map.json")
+
+# adjust these patterns for your language/framework
+IMPORT_PATTERNS = [
+    # TypeScript/JavaScript/MDX: import ... from '...'
+    r"""from\s+['"]([^'"]+)['"]""",
+    # TypeScript/JavaScript: require('...')
+    r"""require\s*\(\s*['"]([^'"]+)['"]\s*\)""",
+]
+
+SKIP_DIRS = {
+    "node_modules", ".git", "build", ".docusaurus", "dist",
+    "coverage", ".turbo"
+}
+
+SOURCE_EXTENSIONS = {
+    ".ts", ".tsx", ".js", ".jsx", ".mdx", ".md",
+}
+
+
+def get_all_source_files() -> list[Path]:
+    files = []
+    for path in ROOT.rglob("*"):
+        if any(part in SKIP_DIRS for part in path.parts):
+            continue
+        if path.suffix in SOURCE_EXTENSIONS and path.is_file():
+            files.append(path)
+    return files
+
+
+def resolve_import(importer: Path, import_path: str, all_files: set[str]) -> str | None:
+    """
+    Try to resolve an import string to an actual file path.
+    Handles relative imports and path aliases.
+    Returns None if the import cannot be resolved to a local file.
+    """
+    # skip node_modules and external packages natively
+    if not import_path.startswith(".") and not import_path.startswith("@theme/") and not import_path.startswith("@site/"):
+        return None
+
+    # handle path aliases for Docusaurus
+    if import_path.startswith("@theme/"):
+        import_path = "src/theme/" + import_path[len("@theme/"):]
+    elif import_path.startswith("@site/"):
+        import_path = import_path[len("@site/"):]
+
+    base = importer.parent / import_path
+
+    # try exact match (already has extension)
+    candidate = str(base)
+    if candidate in all_files:
+        return candidate
+
+    # try with extensions
+    for ext in SOURCE_EXTENSIONS:
+        candidate = str(base) + ext
+        if candidate in all_files:
+            return candidate
+
+    # try as directory index
+    for ext in SOURCE_EXTENSIONS:
+        candidate = str(base / f"index{ext}")
+        if candidate in all_files:
+            return candidate
+
+    return None
+
+
+def build_map() -> dict:
+    print("Scanning source files...")
+    all_files = get_all_source_files()
+    all_file_paths = {str(f.as_posix()) for f in all_files}
+    print(f"Found {len(all_files)} source files.")
+
+    # reverse dependency map: file -> list of files that import it
+    reverse_deps: dict[str, list[str]] = defaultdict(list)
+
+    for source_file in all_files:
+        try:
+            content = source_file.read_text(encoding="utf-8", errors="ignore")
+        except Exception:
+            continue
+
+        for pattern in IMPORT_PATTERNS:
+            matches = re.findall(pattern, content, re.MULTILINE)
+            for match in matches:
+                resolved = resolve_import(source_file, match, all_file_paths)
+                if resolved and resolved != source_file.as_posix():
+                    reverse_deps[resolved].append(source_file.as_posix())
+
+    # clean up: sort and deduplicate
+    return {k: sorted(set(v)) for k, v in reverse_deps.items()}
+
+
+def main():
+    codebase_map = build_map()
+
+    OUTPUT.parent.mkdir(parents=True, exist_ok=True)
+    OUTPUT.write_text(json.dumps(codebase_map, indent=2))
+
+    print(f"\nCodebase map built:")
+    print(f"  {len(codebase_map)} files have dependents")
+    print(f"  Top 10 most depended-upon files:")
+    top = sorted(codebase_map.items(), key=lambda x: len(x[1]), reverse=True)[:10]
+    for path, dependents in top:
+        print(f"    {path}: {len(dependents)} dependents")
+    print(f"\nWritten to {OUTPUT}")
+
+
+if __name__ == "__main__":
+    main()