chore: Add workflow to override visual regression test results - #4954
chore: Add workflow to override visual regression test results#4954jperals wants to merge 7 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4954 +/- ##
=======================================
Coverage 97.66% 97.66%
=======================================
Files 959 959
Lines 31345 31354 +9
Branches 11579 11583 +4
=======================================
+ Hits 30614 30623 +9
+ Misses 724 685 -39
- Partials 7 46 +39 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
8ea1878 to
a92f7c8
Compare
fba0b42 to
85f92b8
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The override workflow is missing required token permissions for its Pulls API call, and the documentation currently claims PR commenting behavior that the workflow does not implement.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a per-commit override mechanism for visual regression results so maintainers can explicitly approve intentional visual diffs and allow the required “Visual regression result” check to pass without rebuilding baselines.
Changes:
- Adds a new Override visual regression workflow that can be triggered via PR comment or manual dispatch and posts a
visual-regression-overridecommit status. - Updates the reusable visual regression workflow to check for the override status and skip expensive screenshot comparison when overridden.
- Updates the deploy workflow to surface a dedicated required “Visual regression result” gate and documents the new override flow.
File summaries
| File | Description |
|---|---|
| docs/RUNNING_TESTS.md | Documents the override-based approval flow for intentional visual diffs. |
| .github/workflows/visual-regression.yml | Adds a commit-status override check and skips visual shards when overridden. |
| .github/workflows/visual-regression-override.yml | Introduces the workflow that applies the per-commit override status and reruns failed deploy jobs. |
| .github/workflows/deploy.yml | Adds statuses read permission, passes commit SHA into the visual workflow, and adds a required “Visual regression result” gate. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Skipping the visual shard job on override can still leave the reusable workflow failing due to downstream report jobs running without artifacts, which undermines the intended “override makes it green” behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 5
- Review effort level: Lite
| needs: check-override | ||
| # Skip the (expensive) screenshot comparison when the changes have been | ||
| # explicitly overridden by a maintainer. | ||
| if: ${{ needs.check-override.outputs.overridden != 'true' }} |
| case "$PERMISSION" in | ||
| admin|write|maintain) | ||
| echo "Authorized." ;; | ||
| *) | ||
| echo "::error::@${ACTOR} is not authorized to override the visual regression check (requires write access)." | ||
| exit 1 ;; |
| To override, first confirm the diffs in the Allure report are intentional, then use either method: | ||
|
|
||
| - **From the PR page (recommended):** comment `/override-visual-regression <Reason to override>` on the pull request. | ||
| - **From the Actions tab:** run the **Override visual regression** workflow (`.github/workflows/visual-regression-override.yml`) manually, passing the commit SHA to approve. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Description
This is a prerequisite to making the visual regression test workflow mandatory.
Ticket:
D417180663Doc:
J5RmAnz6YwPaHow has this been tested?
On this PR, two new checks are run:
On a PR without visual changes like this one, there are no functional differences.
Once this change is merged, it will be possible to test the actual override workflow (GitHub workflows can only be triggered from comments or actions if they are in main).
Review checklist
The following items are to be evaluated by the author(s) and the reviewer(s).
Correctness
CONTRIBUTING.md.CONTRIBUTING.md.Security
checkSafeUrlfunction.Testing
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.