feat(ci): validate embedded Mergify config examples against the schema

.claude/skills/validate-config-examples/SKILL.md

@@ -2,82 +2,105 @@
 name: validate-config-examples
 description: >-
   Validate Mergify YAML configuration examples embedded in the docs against the
-  real Mergify schema, so broken config snippets never ship. Use after editing
-  docs pages that contain .mergify.yml / YAML examples, when reviewing a docs PR
-  with config snippets, or when asked to check/validate config examples in the
-  docs. Extracts YAML code fences, classifies them, and validates full Mergify
-  configs with `mergify config validate` (optionally via the mergify:mergify-config
-  skill when that plugin is installed).
+  real Mergify config JSON Schema, so broken config snippets never ship. Use
+  after editing docs pages that contain .mergify.yml / YAML examples, when
+  reviewing a docs PR with config snippets, or when asked to check/validate
+  config examples in the docs. Runs `pnpm check:config-examples`, which extracts
+  every YAML fence, classifies it, and validates the complete Mergify configs
+  against public/mergify-configuration-schema.json.
 ---
 
 # Validate Config Examples
 
 Mergify config snippets in the docs are the most-copied content on the site. A
-broken example erodes trust instantly. This skill finds every YAML example and
-validates the ones that are real Mergify configs.
-
-## Workflow
-
-1. **Pick the scope.**
-   - Reviewing a change: run against the changed MDX files only.
-   - Auditing: run against the whole `src/content/docs/` tree.
-
-2. **Extract and classify** with the bundled script:
-
-   ```bash
-   .claude/skills/validate-config-examples/scripts/extract_yaml.py <files-or-dir>
-   ```
-
-   It prints a JSON array of every YAML fence with `file`, `line`,
-   `classification`, and `code`. Classifications:
-
-   | Classification | Meaning | Action |
-   | --- | --- | --- |
-   | `mergify-config` | A real Mergify config (has top-level keys like `queue_rules`, `pull_request_rules`, `merge_protections`) | **Validate it** |
-   | `github-actions` | A CI workflow YAML, not Mergify config | Skip |
-   | `partial` | Marked `# partial` or a fragment (no top-level key) | Skip validation |
-   | `unknown` | YAML that fits none of the above | Eyeball manually |
-
-3. **Validate each `mergify-config` snippet** with `mergify config validate` —
-   the authoritative validator (it understands condition syntax and action
-   options, not just the JSON shape). Write each snippet to a temp file and pass
-   it with `--config-file` (the command auto-detects `.mergify.yml` from the
-   current directory otherwise, so the flag is required for a temp file):
-
-   ```bash
-   printf '%s\n' "$CODE" > /tmp/mergify-example.yml
-   mergify config validate --config-file /tmp/mergify-example.yml
-   ```
-
-   If the **mergify:mergify-config** plugin skill is installed, you can invoke it
-   instead of calling the CLI directly — it wraps the same command. As a
-   structural fallback when the CLI is unavailable, validate against
-   `public/mergify-configuration-schema.json` (JSON Schema).
-
-4. **Report** each failure as `file:line — <error>`. Fix clear errors directly in
-   the MDX (wrong key name, bad indentation, invalid condition). For anything
-   ambiguous, flag rather than guess — a wrong "fix" to a config example is worse
-   than a flagged one.
-
-## The fragment convention
-
-Many doc examples are partial on purpose — they show one rule, not a whole file.
-A bare fragment cannot be validated standalone and will false-positive.
-
-The script treats a snippet as `partial` (and skips it) when:
-
-- it has no unindented top-level key (it is clearly a fragment), or
-- its first lines contain a `# partial` marker comment.
-
-When you intentionally show a fragment that the script can't auto-detect, add a
-leading `# partial` comment to the code block so it is skipped cleanly. Do not add
-the marker to examples that are meant to be complete, valid configs — those should
-validate.
-
-## Scope & guardrails
-
-- Only validates YAML/`yml` fences; ignores all other languages.
-- Skips GitHub Actions and other non-Mergify YAML automatically.
-- Never edits `src/content/changelog/`.
-- Fix only genuine errors; leave valid examples alone even if you'd phrase the
-  surrounding prose differently.
+broken example erodes trust instantly. A `config-examples` CI job validates them
+on every PR; this skill is how you run the same check locally and fix what it
+finds.
+
+## Run it
+
+```bash
+pnpm check:config-examples          # validate everything under src/content/docs
+```
+
+Under the hood this runs `node scripts/validate-config-examples.mjs`. It scans
+MDX for ```yaml / ```yml fences, classifies each, YAML-parses the complete
+Mergify configs, and validates them against `public/mergify-configuration-schema.json`
+with [Ajv](https://ajv.js.org/). Exit code is non-zero if any example is invalid;
+each failure prints as `file:line — <error>`.
+
+To validate only the files you touched, pass paths:
+
+```bash
+node scripts/validate-config-examples.mjs src/content/docs/merge-queue
+```
+
+To inspect the classification (which blocks are validated vs skipped and why),
+use `--json`:
+
+```bash
+node scripts/validate-config-examples.mjs --json src/content/docs | jq '.[] | {file, line, classification}'
+```
+
+## What gets validated vs skipped
+
+The script validates a block only when it is a **complete** Mergify config —
+i.e. it has an unindented top-level key from the config schema (`queue_rules`,
+`pull_request_rules`, `merge_protections`, `merge_protections_settings`,
+`scopes`, `defaults`, `commands_restrictions`, `extends`, `shared`, ...).
+
+It automatically **skips**:
+
+- **Fragments** — a snippet with no top-level key (e.g. a single rule or a bare
+  list of conditions shown inline). Cannot be validated as a whole config.
+- **`...` placeholders** — any block containing a bare `...` line is treated as
+  deliberately incomplete.
+- **GitHub Actions / other CI YAML** — detected by `on:` + `jobs:`/`steps:`/etc.
+- **Anything marked `# partial`** in its first lines.
+
+For a block that *looks* like a full config but should not be validated — most
+often **deprecated syntax shown in a migration guide** — put a reader-invisible
+MDX comment on the line before the fence:
+
+````mdx
+{/* validate-config-examples: skip — deprecated partition_rules syntax, shown for migration */}
+
+```yaml
+partition_rules:
+  ...
+```
+````
+
+Do **not** skip a block just to make CI pass. If an example is meant to be a
+real, copy-pasteable config, fix it instead.
+
+## Fixing failures
+
+- **`invalid YAML: ...`** — the snippet isn't valid YAML. The most common cause
+  is an unquoted Mergify template at the start of a value (`{{ ... }}` or a
+  leading `@`): `message: @{{author}} ...` must be `message: "@{{author}} ..."`,
+  and `bot_account: {{ author }}` must be `bot_account: "{{ author }}"`.
+- **`/ must NOT have additional properties`** — an unknown or misspelled key at
+  the top level (or a deprecated top-level key). Fix the key, or mark the block
+  skipped if the old syntax is shown on purpose.
+- **`/path must be <type>`** — a value has the wrong type or shape. Match the
+  schema.
+
+Fix clear errors directly in the MDX. For anything ambiguous, flag it rather than
+guess — a wrong "fix" to a config example is worse than a flagged one.
+
+## Scope, guardrails & limitations
+
+- Validation is **structural** (keys, types, shapes). It does **not** check the
+  semantics of condition expressions (`base=main`), nor Mergify's custom string
+  formats (`duration`, `template`) — a generic JSON-Schema `duration` validator
+  would wrongly reject valid values like `checks_timeout: 5 min`, so all `format`
+  keywords are intentionally ignored. For deep semantic validation of a single
+  config, `mergify config validate --config-file <file>` (the **mergify:mergify-config**
+  skill) remains authoritative — but it fetches the schema from docs.mergify.com,
+  so it is not used in CI.
+- The schema at `public/mergify-configuration-schema.json` is kept fresh by an
+  automated sync ("chore: sync Mergify JSON Schema files"); the check always runs
+  against the copy in the repo, so a PR is validated against its own schema.
+- Only `yaml`/`yml` fences are considered. Never edits `src/content/changelog/`.
+- Fix only genuine errors; leave valid examples alone.

.github/workflows/ci.yaml

@@ -65,6 +65,27 @@ jobs:
           pnpm install --frozen-lockfile
           pnpm test
 
+  config-examples:
+    timeout-minutes: 10
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout 🛎️
+        uses: actions/checkout@v7.0.0
+
+      - name: Setup pnpm 📦
+        uses: pnpm/action-setup@v6.0.9
+
+      - name: Setup Node 🔧
+        uses: actions/setup-node@v6.4.0
+        with:
+          cache: pnpm
+          node-version-file: .node-version
+
+      - name: Validate embedded Mergify config examples
+        run: |
+          pnpm install --frozen-lockfile
+          pnpm check:config-examples
+
   build:
     timeout-minutes: 20
     runs-on: ubuntu-24.04

package.json

@@ -14,6 +14,7 @@
     "format": "biome format --write .",
     "format:check": "biome format src integrations plugins scripts",
     "check": "astro check && eslint . && biome check .",
+    "check:config-examples": "node scripts/validate-config-examples.mjs",
     "check:links": "linkinator dist/ --recurse --concurrency 25 --verbosity error --skip 'https?://'"
   },
   "devDependencies": {
@@ -35,6 +36,7 @@
     "@typescript-eslint/eslint-plugin": "^8.63.0",
     "@typescript-eslint/parser": "^8.63.0",
     "agentation": "^3.0.2",
+    "ajv": "^8.20.0",
     "astro": "^7.0.7",
     "astro-auto-import": "^0.5.1",
     "astro-eslint-parser": "^2.1.0",