feat(ci): validate embedded Mergify config examples against the schema#12142
Merged
Conversation
Contributor
Merge Protections🟢 All 6 merge protections satisfied — ready to merge. Show 6 satisfied protections🟢 🤖 Continuous Integration
🟢 👀 Review Requirements
🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 🔎 Reviews
🟢 📕 PR description
🟢 🚦 Auto-queueWhen all merge protections are satisfied, this pull request will be queued automatically. |
There was a problem hiding this comment.
Pull request overview
Adds a repo-native CI check to prevent .mergify.yml examples embedded in docs from drifting out of sync with the vendored JSON Schema, so invalid snippets fail PR validation instead of silently shipping.
Changes:
- Introduces
scripts/validate-config-examples.mjs(Ajv + js-yaml) plus Vitest coverage to extract/classify YAML fences in MDX and structurally validate complete Mergify configs againstpublic/mergify-configuration-schema.json. - Adds
pnpm check:config-examplesand a dedicated GitHub Actions job to run it in CI. - Fixes/marks up a few docs snippets so they either validate (quote needed templates) or are explicitly skipped (deprecated
partition_rulesmigration examples).
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/content/docs/workflow/actions/comment.mdx | Quotes a template-containing string so the YAML example parses and validates. |
| src/content/docs/workflow/actions/backport.mdx | Quotes a template-containing value so the YAML example parses and validates. |
| src/content/docs/merge-queue/migrate-partitions-to-scopes.mdx | Adds skip directives for intentionally-deprecated partition_rules examples. |
| scripts/validate-config-examples.mjs | New validator/extractor CLI for MDX YAML fences, using Ajv 2020 + js-yaml. |
| scripts/validate-config-examples.test.mjs | Unit + integration tests covering classification, schema validation, skip directive behavior, and a docs-wide validation pass. |
| package.json | Adds check:config-examples script entry. |
| pnpm-lock.yaml | Locks the new Ajv dev dependency. |
| .github/workflows/ci.yaml | Adds a config-examples CI job to run the new check. |
| .claude/skills/validate-config-examples/SKILL.md | Updates the skill docs to describe the new Node-based workflow (needs a small fence fix per review comment). |
| .claude/skills/validate-config-examples/scripts/extract_yaml.py | Removes the superseded Python extractor. |
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Docs config snippets are the most-copied content on the site, but nothing
stopped them from drifting out of sync with the real Mergify config schema.
A wrong example erodes trust instantly. This adds CI that extracts every
Mergify config example from the MDX and validates it, so a broken snippet
fails CI instead of silently misleading users.
This started as a spike ("not sure it's possible"). It is possible, cleanly.
The three feasibility risks resolved as follows:
1. Schema source. The repo already vendors the exact schema at
`public/mergify-configuration-schema.json` (draft 2020-12), and it is kept
fresh by the automated "chore: sync Mergify JSON Schema files" commits. We
validate against that in-repo copy — offline, no auth, no network. The
`mergify config validate` CLI was evaluated and rejected for CI: it fetches
the schema over the network from docs.mergify.com (the served copy even lags
the just-synced repo file), which is both flaky and circular — validating the
docs against a schema served by the docs. It does the same structural
JSON-Schema check we do, so we lose nothing. It remains the right tool for
deep, single-config semantic validation locally.
2. Extraction / partial-vs-complete. A block is validated only when it has an
unindented top-level Mergify key (`queue_rules`, `pull_request_rules`,
`scopes`, `merge_protections_settings`, ...). Fragments, `...` placeholders,
GitHub Actions YAML, and `# partial`-marked blocks are skipped automatically.
For a complete-looking block that must not be validated (deprecated syntax in
a migration guide), a reader-invisible MDX comment
`{/* validate-config-examples: skip — why */}` before the fence excludes it.
3. CI shape. A new `config-examples` job in the existing GitHub Actions
workflow runs `pnpm check:config-examples`, mirroring the other jobs.
Running the validator over the current docs surfaced 111 complete config
examples; 2 were genuinely broken and are fixed here:
- `comment.mdx`: `message: @{{author}} ...` is invalid YAML (leading `@`), now
quoted.
- `backport.mdx`: `bot_account: {{ author }}` is invalid YAML (flow mapping),
now quoted.
The 2 deprecated `partition_rules` examples in the partitions→scopes migration
guide are marked skipped.
- Validation is structural only. Mergify's custom string formats
(`duration`, `template`, ...) are intentionally not enforced — a generic
JSON-Schema `duration` validator reads it as ISO-8601 and would wrongly reject
valid values like `checks_timeout: 5 min`.
- Consolidated on a single Node implementation (`scripts/validate-config-examples.mjs`,
repo-native, with unit + integration tests) and retired the earlier standalone
Python extractor; the `validate-config-examples` skill now drives the same script.
- `pnpm check:config-examples` — green (111 checked, 2 skipped, 0 invalid)
- `pnpm test` — 73 passing, incl. a negative test proving the validator rejects
an unknown key (not a no-op)
- `pnpm check` — clean (astro check + eslint + biome)
- actionlint — clean on the updated workflow
Change-Id: Iec4826a97e941255bcfcf0133a654edaf1f07585
jd
force-pushed
the
devs/jd/jd/validate-config-examples/validate-embedded-mergify-config-examples-against--ec4826a9
branch
from
July 17, 2026 15:59
d41044e to
50d4f0c
Compare
Member
Author
Revision history
|
JulianMaurin
approved these changes
Jul 18, 2026
kozlek
approved these changes
Jul 20, 2026
Contributor
Merge Queue Status
This pull request spent 2 minutes 30 seconds in the queue, including 2 minutes 14 seconds running CI. Required conditions to merge
|
53 tasks
mergify
Bot
deleted the
devs/jd/jd/validate-config-examples/validate-embedded-mergify-config-examples-against--ec4826a9
branch
July 20, 2026 07:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs config snippets are the most-copied content on the site, but nothing
stopped them from drifting out of sync with the real Mergify config schema.
A wrong example erodes trust instantly. This adds CI that extracts every
Mergify config example from the MDX and validates it, so a broken snippet
fails CI instead of silently misleading users.
This started as a spike ("not sure it's possible"). It is possible, cleanly.
The three feasibility risks resolved as follows:
Schema source. The repo already vendors the exact schema at
public/mergify-configuration-schema.json(draft 2020-12), and it is keptfresh by the automated "chore: sync Mergify JSON Schema files" commits. We
validate against that in-repo copy — offline, no auth, no network. The
mergify config validateCLI was evaluated and rejected for CI: it fetchesthe schema over the network from docs.mergify.com (the served copy even lags
the just-synced repo file), which is both flaky and circular — validating the
docs against a schema served by the docs. It does the same structural
JSON-Schema check we do, so we lose nothing. It remains the right tool for
deep, single-config semantic validation locally.
Extraction / partial-vs-complete. A block is validated only when it has an
unindented top-level Mergify key (
queue_rules,pull_request_rules,scopes,merge_protections_settings, ...). Fragments,...placeholders,GitHub Actions YAML, and
# partial-marked blocks are skipped automatically.For a complete-looking block that must not be validated (deprecated syntax in
a migration guide), a reader-invisible MDX comment
{/* validate-config-examples: skip — why */}before the fence excludes it.CI shape. A new
config-examplesjob in the existing GitHub Actionsworkflow runs
pnpm check:config-examples, mirroring the other jobs.Running the validator over the current docs surfaced 111 complete config
examples; 2 were genuinely broken and are fixed here:
comment.mdx:message: @{{author}} ...is invalid YAML (leading@), nowquoted.
backport.mdx:bot_account: {{ author }}is invalid YAML (flow mapping),now quoted.
The 2 deprecated
partition_rulesexamples in the partitions→scopes migrationguide are marked skipped.
Validation is structural only. Mergify's custom string formats
(
duration,template, ...) are intentionally not enforced — a genericJSON-Schema
durationvalidator reads it as ISO-8601 and would wrongly rejectvalid values like
checks_timeout: 5 min.Consolidated on a single Node implementation (
scripts/validate-config-examples.mjs,repo-native, with unit + integration tests) and retired the earlier standalone
Python extractor; the
validate-config-examplesskill now drives the same script.pnpm check:config-examples— green (111 checked, 2 skipped, 0 invalid)pnpm test— 73 passing, incl. a negative test proving the validator rejectsan unknown key (not a no-op)
pnpm check— clean (astro check + eslint + biome)actionlint — clean on the updated workflow