-
Notifications
You must be signed in to change notification settings - Fork 3
Add visual regression testing for shared layout and theme #1164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kantord
wants to merge
10
commits into
main
Choose a base branch
from
issue-1163
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
d05a972
Add visual regression testing for shared layout and theme
kantord 320fa06
Fix dubious-ownership git error in container jobs
kantord 46c65a9
Add automatic baseline auto-fix on check failure
kantord 14077d6
Fix auto-fix job being silently skipped, gh missing --repo
kantord d83167a
chore: update visual baselines for 14077d61412c
github-actions[bot] 42740e9
Document the auto-fix approval-gate limitation
kantord 4120ae7
Capture visual baselines at 2x for crisper PR thumbnails
kantord 1a4168f
Refine visual regression coverage
danbarr 0d1c549
Stabilize visual regression snapshots
danbarr 2369c2f
Separate trusted workflow writes
danbarr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| name: Visual regression | ||
|
|
||
| on: | ||
| workflow_call: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| visual-regression: | ||
| name: Playwright visual snapshots | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
| # Pinned to the exact @playwright/test version in package.json β the | ||
| # image and npm package must match exactly, or browser launch breaks. | ||
| # Screenshot comparison is sensitive to font rendering and software | ||
| # rasterization, which differ by host OS/GPU β running inside this | ||
| # exact image keeps this job and the /update-snapshots regeneration | ||
| # job (update-visual-baselines.yaml, same digest) pixel-consistent | ||
| # with each other regardless of which runner picks up the job. | ||
| container: | ||
| image: mcr.microsoft.com/playwright@sha256:eff16c30e6f3f4af0a03fa4b706120d5e9b0891c344a27d64559aff5900a4a27 # v1.63.0-noble | ||
| options: --ipc=host | ||
| env: | ||
| # Keep in sync with the image tag in the comment above β the | ||
| # "Verify Playwright version" step below checks the installed | ||
| # @playwright/test against this, so a dependency bump without a | ||
| # matching image bump fails loud and named instead of as a | ||
| # confusing browser-launch error on an unrelated PR. | ||
| PLAYWRIGHT_IMAGE_VERSION: '1.63.0' | ||
| # Keep the build-time MCP metadata format stable. The regular site | ||
| # build still validates the current ToolHive release independently. | ||
| TOOLHIVE_VERSION: '0.49.0' | ||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| # actions/checkout marks the repo safe in the RUNNER's git config, | ||
| # but a run: step inside `container:` executes via `docker exec` | ||
| # into a separate environment that never got that exception β | ||
| # without this, Docusaurus's last-update-date git log call (and | ||
| # anything else here shelling out to git) fails with "detected | ||
| # dubious ownership in repository". | ||
| - name: Mark workspace as a safe git directory | ||
| run: git config --global --add safe.directory "$GITHUB_WORKSPACE" | ||
|
|
||
| # The Playwright image is minimal β it ships neither jq, which | ||
| # scripts/install-thv.sh requires, nor a Node version manager, since | ||
| # it already bundles the Node build this image's Chromium was | ||
| # tested against. A pinned static binary (checksum-verified) avoids | ||
| # `apt-get update`, which syncs the full Ubuntu archive index just | ||
| # to install one small tool. | ||
| - name: Install jq | ||
| run: | | ||
| curl -sSL -o /usr/local/bin/jq https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-linux-amd64 | ||
| echo "b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f /usr/local/bin/jq" | sha256sum -c - | ||
| chmod +x /usr/local/bin/jq | ||
|
|
||
| - name: Cache dependencies | ||
| id: cache | ||
| uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 | ||
| with: | ||
| path: ./node_modules | ||
| key: modules-${{ hashFiles('package-lock.json') }} | ||
|
|
||
| - name: Install dependencies | ||
| if: steps.cache.outputs.cache-hit != 'true' | ||
| run: npm ci | ||
|
|
||
| - name: Verify Playwright version matches pinned image | ||
| run: | | ||
| installed="$(npx playwright --version | sed -n 's/^Version \([0-9.]*\)$/\1/p')" | ||
| if [ "$installed" != "$PLAYWRIGHT_IMAGE_VERSION" ]; then | ||
| echo "::error::@playwright/test resolved to $installed but the pinned image is for $PLAYWRIGHT_IMAGE_VERSION β bump the image digest (see container.image comment) to match." | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: Install ToolHive CLI | ||
| run: ./scripts/install-thv.sh | ||
|
|
||
| - name: Build site | ||
| run: npm run build | ||
|
|
||
| - name: Run visual regression tests | ||
| run: npm run test:visual | ||
|
danbarr marked this conversation as resolved.
|
||
|
|
||
| - name: Upload test artifacts | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| if: failure() | ||
| with: | ||
| name: playwright-report | ||
| path: | | ||
| test-results/ | ||
| playwright-report/ | ||
| retention-days: 7 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| name: PR Screenshot Summary | ||
|
|
||
| # workflow_run always loads this workflow from the default branch. That makes | ||
| # it a trusted follow-up to the read-only PR workflow even though this job can | ||
| # update pull request descriptions. | ||
| # The privileged job checks out only the API-resolved base SHA and treats the | ||
| # PR head as inert Git data. Zizmor cannot infer that cross-step trust boundary. | ||
| on: # zizmor: ignore[dangerous-triggers] | ||
| workflow_run: | ||
| workflows: ['On PR'] | ||
| types: [completed] | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.workflow_run.pull_requests[0].number }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: {} | ||
|
|
||
| jobs: | ||
| update-description: | ||
| name: Update PR description with screenshot summary | ||
| if: | | ||
| github.event.workflow_run.event == 'pull_request' && | ||
| github.event.workflow_run.pull_requests[0] != null | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| pull-requests: write | ||
| steps: | ||
| - name: Resolve current PR context | ||
| id: pr | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| PR_NUMBER: ${{ github.event.workflow_run.pull_requests[0].number }} | ||
| REPO: ${{ github.repository }} | ||
| RUN_HEAD_SHA: ${{ github.event.workflow_run.head_sha }} | ||
| run: | | ||
| PR_JSON="$(gh api "repos/$REPO/pulls/$PR_NUMBER")" | ||
| BASE_SHA="$(echo "$PR_JSON" | jq -r '.base.sha')" | ||
| HEAD_SHA="$(echo "$PR_JSON" | jq -r '.head.sha')" | ||
| if [ "$HEAD_SHA" != "$RUN_HEAD_SHA" ]; then | ||
| echo "The PR advanced after this run; its newer run will update the summary." | ||
| echo "current=false" >> "$GITHUB_OUTPUT" | ||
| exit 0 | ||
| fi | ||
| { | ||
| echo "current=true" | ||
| echo "base_sha=$BASE_SHA" | ||
| echo "head_sha=$HEAD_SHA" | ||
| } >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Checkout trusted base revision | ||
| if: steps.pr.outputs.current == 'true' | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| ref: ${{ steps.pr.outputs.base_sha }} | ||
| fetch-depth: 0 | ||
| persist-credentials: false | ||
|
|
||
| - name: Fetch PR head for read-only inspection | ||
| if: steps.pr.outputs.current == 'true' | ||
| env: | ||
| EXPECTED_HEAD_SHA: ${{ steps.pr.outputs.head_sha }} | ||
| PR_NUMBER: ${{ github.event.workflow_run.pull_requests[0].number }} | ||
| run: | | ||
| git fetch --no-tags origin \ | ||
| "+refs/pull/$PR_NUMBER/head:refs/remotes/origin/pr/$PR_NUMBER/head" | ||
| ACTUAL_HEAD_SHA="$(git rev-parse "refs/remotes/origin/pr/$PR_NUMBER/head")" | ||
| if [ "$ACTUAL_HEAD_SHA" != "$EXPECTED_HEAD_SHA" ]; then | ||
| echo "::error::PR head changed while preparing the summary; retry against the current head." | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: Update PR description with screenshot summary | ||
| if: steps.pr.outputs.current == 'true' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| PR_NUMBER: ${{ github.event.workflow_run.pull_requests[0].number }} | ||
| BASE_SHA: ${{ steps.pr.outputs.base_sha }} | ||
| HEAD_SHA: ${{ steps.pr.outputs.head_sha }} | ||
| # This script comes from the checked-out base revision, never the PR. | ||
| run: node scripts/update-pr-screenshot-summary.mjs | ||
|
danbarr marked this conversation as resolved.
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.