ci(playwright): auto-refresh timing-baseline.json after every successful full merge_group

[chirag-madlani]

Summary

Follow-up to #30871's one-shot manual refresh. Without an automated refresh, .github/playwright/timing-baseline.json drifts fast — the 12-day gap that #30871 closed had produced 23% dead titles and 30 uncovered spec files on main, enough to make the planner materially wrong on both LPT balance and atomic-unit weight decisions.

Adds a refresh-timing-baseline job that runs after playwright-summary on every successful full-mode merge_group run and opens (or updates) a single tracked PR with the refreshed baseline. A human still clicks merge — the auto-PR just eliminates the manual regeneration step.

What the job does

1. Downloads the same playwright-timing-history-full-* artifact the summary job already uploaded.
2. Runs .github/scripts/refresh_timing_baseline.py, which:
   • normalizes the reporter's schema (title = leaf, drop retryDurationMs) to match the checked-in baseline;
   • preserves retainedUnstableTestIds + retainedSourceRunId + retainedSourceSha from the current baseline (curated fields);
   • writes compact JSON so the diff stays a one-line reserialization;
   • refuses (exit 2) if drift > 40% of test-id entries — a safety valve against an accidentally-broken source run.
3. Skips if the file is byte-identical.
4. Force-pushes a single tracked branch ci/auto-refresh-timing-baseline and opens or updates one PR against main.

Guards

• event == merge_group only (safest ci-status signal — PRs are targeted, schedule/dispatch have less trust).
• execution_mode == full only (targeted plans skip specs).
• playwright-summary.result == success only (no partial data on failures).
• Single tracked PR — successive runs update it in place rather than piling up open PRs.

Files

• .github/scripts/refresh_timing_baseline.py (new, 160 lines) — normalization + drift-cap script.
• .github/scripts/tests/test_refresh_timing_baseline.py (new, 6 pytest cases) — covers normalization, retention preservation, diff, drift cap, non-full rejection, end-to-end main().
• .github/workflows/playwright-postgresql-e2e.yml — new refresh-timing-baseline job appended after playwright-summary.

Verified

• 94 total tests in .github/scripts/tests/ pass (88 existing + 6 new).
• YAML syntax validated locally.
• The script's behavior matches what chore(playwright): refresh timing-baseline.json from run 30802611748 #30871 did by hand.

Test plan

• First successful full merge_group after merge: the job runs, finds a byte-identical baseline (since chore(playwright): refresh timing-baseline.json from run 30802611748 #30871's refresh is current), skips PR creation.
• After enough merge_group runs to shift the baseline (new tests, changed durations): the job opens PR against main titled chore(playwright): auto-refresh timing-baseline.json.
• Subsequent merge_group runs update the same PR in place, not create new ones.
• If a merge_group run produces a broken baseline (>40% drift), the job fails cleanly with a message pointing at manual regeneration.

Related: #30812, #30827, #30835, #30871.

🤖 Generated with Claude Code

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[gitar-bot]

Code Review ✅ Approved

Adds an automated GitHub Actions job to refresh the Playwright timing baseline after successful full merge group runs, preventing configuration drift. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source}

[Copilot]

Pull request overview

This PR adds automation to keep Playwright’s checked-in timing baseline (.github/playwright/timing-baseline.json) up to date by running a post-summary refresh on successful full-mode merge_group runs and opening/updating a single tracked PR with the refreshed file.

Changes:

• Adds a new refresh-timing-baseline job to the Playwright PostgreSQL E2E workflow that downloads the full timing-history artifact, regenerates the baseline, and opens/updates a single PR.
• Introduces .github/scripts/refresh_timing_baseline.py to normalize merged timing history into the baseline schema, preserve curated retention fields, and enforce a drift cap.
• Adds pytest coverage for the new refresh script.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
.github/workflows/playwright-postgresql-e2e.yml	Adds a guarded post-summary job that regenerates and PRs baseline updates.
.github/scripts/refresh_timing_baseline.py	New baseline refresh/normalization script with drift summary + drift cap.
.github/scripts/tests/test_refresh_timing_baseline.py	New pytest suite validating normalization, retention, drift cap, and main() behavior.

Suppressed comments (1)

.github/scripts/refresh_timing_baseline.py:93

• compute_diff() keys tests by (file, title), but timing-baseline.json contains many distinct tests that share the same file+title (e.g., repeated leaf titles in Pages/EntityHeaderBreadcrumb.spec.ts). This collapses entries in the diff calculation and makes the drift safety valve inaccurate. Key by stable test identity (project,id) instead so added/removed/drift are computed per test-id entry as intended.

    def key(t: dict[str, Any]) -> tuple[str, str]:
        return (t.get("file", ""), t.get("title", ""))

    cur_by_key = {key(t): t for t in current.get("tests", [])}
    new_by_key = {key(t): t for t in new.get("tests", [])}

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit ed4dd397a64f85c7b72fe86bf7b5d30f5a40abd7 in Playwright run 30809595913, attempt 1.

✅ 550 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 3 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 53m 11s

⏱️ Max setup 2m 58s · max shard execution 17m 8s · max shard-job elapsed before upload 20m 4s · reporting 5s

🌐 201.26 requests/attempt · 2.83 app boots/UI scenario · 16.15% common-shard skew

Optimization targets still in progress:

• Common shard skew was 16.15% (convergence target: at most 15%).
• Browser traffic was 201.26 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 2.83 per UI scenario (1617 boots / 571 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	149	0	0	3	0	0
✅ Shard chromium-02	131	0	0	0	0	0
✅ Shard chromium-03	119	0	0	0	0	0
✅ Shard data-asset-rules-01	61	0	0	0	0	0
✅ Shard domain-isolation-01	14	0	0	0	0	0
✅ Shard global-state-01	34	0	0	0	0	0
✅ Shard ingestion-01	1	0	0	0	0	0
✅ Shard reindex-01	2	0	0	0	0	0
✅ Shard search-01	10	0	0	0	0	0
✅ Shard search-rbac-01	29	0	0	0	0	0

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace