Baseline every Percy build against main - #6077
Open
backspace wants to merge 1 commit into
Open
Conversation
Percy picks a build's baseline from git history when nothing tells it otherwise, so a branch stacked on another feature branch is compared against its parent rather than against main — and inherits whatever that parent's most recent build happened to contain. A parent build that lost snapshots is the case that hurts. The child compares its full set against a partial one, every snapshot with no counterpart in the baseline reads as new, and the build lands with most of the suite needing review. One such build compared 59 snapshots against a four-snapshot baseline on a sibling branch and reported 55 changes, none of them real. The reject step already keeps a partial build from becoming a baseline on `main`. It does not cover a partial build on a feature branch, which is not rejected and can still be inherited by anything stacked on it. Pinning the target branch closes that path without needing to. It also means a stacked PR's diff shows its cumulative effect against main, which is what an unstacked branch already gets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
Host Test Results 1 files 1 suites 1h 51m 47s ⏱️ Results for commit c5dbe85. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Percy picks a build's baseline from git history when nothing tells it otherwise. A branch stacked on another feature branch is therefore compared against its parent rather than against main, and inherits whatever that parent's most recent build happened to contain.
The case that hurts is a parent build that lost snapshots. The child compares its full set against a partial one, every snapshot with no counterpart reads as new, and the build lands with most of the suite needing review — with nothing on the build explaining why.
The build that prompted this
A 59-snapshot build was baselined against a four-snapshot build on a sibling branch, whose own review state was
missing_snapshots. 55 snapshots reported as changed, atdiff-ratio: 1— the ratio for "no counterpart to compare against", not for a visual change. None of the 55 were real.That has now happened on two branches, and it is indistinguishable at a glance from a change that genuinely moved the whole suite.
Why the existing gate does not cover it
The reject step in this workflow already stops a partial build becoming a baseline on
main, and its comment describes exactly this failure: a partial set becomes the baseline every later branch is compared against. What it cannot do is stop a partial build on a feature branch from being inherited by something stacked on it — nothing rejects those, because nothing needs to until they are used as a baseline.Pinning the target branch closes that path without having to extend the gate.
What changes
Every build now compares against main, which is gated by that reject step and auto-approved. For a stacked PR the diff becomes its cumulative effect against main rather than only its increment over its parent — the same comparison an unstacked branch already gets, and the one a reviewer of the eventual merge cares about.
Set on the step that takes snapshots rather than the finalize step: the baseline is chosen when the build is created by the first shard to upload, and finalize only seals it.