Skip to content

Run acceptance tests after merges - #107

Merged
IlyaasK merged 3 commits into
mainfrom
hypeship/run-acceptance-on-prs
Jul 31, 2026
Merged

Run acceptance tests after merges#107
IlyaasK merged 3 commits into
mainfrom
hypeship/run-acceptance-on-prs

Conversation

@IlyaasK

@IlyaasK IlyaasK commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • run the existing live acceptance matrix after changes reach main
  • retain manual workflow dispatch
  • keep live acceptance tests out of pull-request CI
  • align the README and release checklist with that policy

Why this is release work

The post-merge workflow catches live API, SDK, and Terraform integration
regressions without running side-effecting tests concurrently across every
pull-request branch. Manual dispatch remains available for targeted validation,
and the complete matrix remains a pre-tag release gate.

Tests run

  • go test -short -timeout=2m ./...
  • git diff --check

Acceptance status

The existing browser-pool and project acceptance tests previously passed on
this branch. They were not rerun for this workflow-trigger and documentation
change.

Deferred work

Remaining release gate

Run the complete selected-surface matrix against the release commit before the
first public tag.


Note

Low Risk
CI trigger and documentation only; no provider or test logic changes.

Overview
Post-merge live acceptance — The Acceptance GitHub Actions workflow now runs on every push to main, in addition to manual workflow_dispatch. PR CI is unchanged; live tests still do not run on pull requests.

DocsREADME.md, docs/acceptance.md, and docs/release.md now describe automatic runs after merges to main, manual dispatch, and local opt-in as the three ways to exercise the selected-surface matrix.

Reviewed by Cursor Bugbot for commit 06dd269. Bugbot is set up for automated code reviews on this repo. Configure here.

@IlyaasK
IlyaasK requested a review from Sayan- July 28, 2026 19:59

@Sayan- Sayan- left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security model is fine — this is pull_request, not pull_request_target, so forks get no secrets, and the secret checks run before checkout.

The trigger timing is what I'd change. These tests create real projects, pools, profiles, proxies, and extensions in one shared org through one API key, and cleanup is registered as a t.Cleanup, so it only runs if the test process finishes — a cancelled or timed-out job leaks. docs/release.md already carries a manual sweep runbook for leaked kernel-tf-* resources, so that isn't hypothetical. With ~52 open PRs and concurrency.group: acceptance-${{ github.ref }} (per-branch, so runs are parallel rather than serialized), every push fires the full live suite against that one org.

Worth reconciling too: #85 adds docs/acceptance.md whose gate rules say "Keep live tests out of pull-request CI" and "Live acceptance remains a manual pre-tag gate," while this PR removes the equivalent line from docs/release.md. Two open PRs currently point opposite directions on the same policy.

The ask: rework this to trigger on merge to main (a push: trigger on main) instead of on pull requests. That still catches API-contract breakage within minutes of landing, but it's one run per merge instead of N parallel runs per push across the open PR set, and there's no cancellation storm to leak from. Keep workflow_dispatch and the pre-tag full-matrix run exactly as they are, and docs/acceptance.md then stays accurate as written.

On ordering: this touches acceptance.yml, README.md, and docs/release.md, all of which the chain also edits — #87/#91/#93/#88 each add a matrix entry, and #85/#95 rewrite the same doc sections. Rebasing this after the chain lands will be a lot cheaper than the reverse.

@IlyaasK IlyaasK changed the title Run acceptance tests on trusted pull requests Run acceptance tests after merges Jul 29, 2026
@IlyaasK
IlyaasK requested a review from Sayan- July 29, 2026 17:52

@Sayan- Sayan- left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly the shape I was asking for.

One follow-through: #85's docs/acceptance.md still says acceptance tests "run only through explicit local opt-in or the manual GitHub Actions workflow" and that "live acceptance remains a manual pre-tag gate." Once this lands, the document #85 designates as the source of truth for the gate no longer describes the post-merge trigger. Worth fixing those two sentences when #85 rebases.

Preserve the six-surface release gate and align acceptance documentation with the main-branch trigger.
@IlyaasK
IlyaasK merged commit 5dba1de into main Jul 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants