Run acceptance tests after merges - #107
Conversation
Sayan-
left a comment
There was a problem hiding this comment.
Security model is fine — this is pull_request, not pull_request_target, so forks get no secrets, and the secret checks run before checkout.
The trigger timing is what I'd change. These tests create real projects, pools, profiles, proxies, and extensions in one shared org through one API key, and cleanup is registered as a t.Cleanup, so it only runs if the test process finishes — a cancelled or timed-out job leaks. docs/release.md already carries a manual sweep runbook for leaked kernel-tf-* resources, so that isn't hypothetical. With ~52 open PRs and concurrency.group: acceptance-${{ github.ref }} (per-branch, so runs are parallel rather than serialized), every push fires the full live suite against that one org.
Worth reconciling too: #85 adds docs/acceptance.md whose gate rules say "Keep live tests out of pull-request CI" and "Live acceptance remains a manual pre-tag gate," while this PR removes the equivalent line from docs/release.md. Two open PRs currently point opposite directions on the same policy.
The ask: rework this to trigger on merge to main (a push: trigger on main) instead of on pull requests. That still catches API-contract breakage within minutes of landing, but it's one run per merge instead of N parallel runs per push across the open PR set, and there's no cancellation storm to leak from. Keep workflow_dispatch and the pre-tag full-matrix run exactly as they are, and docs/acceptance.md then stays accurate as written.
On ordering: this touches acceptance.yml, README.md, and docs/release.md, all of which the chain also edits — #87/#91/#93/#88 each add a matrix entry, and #85/#95 rewrite the same doc sections. Rebasing this after the chain lands will be a lot cheaper than the reverse.
Sayan-
left a comment
There was a problem hiding this comment.
Exactly the shape I was asking for.
One follow-through: #85's docs/acceptance.md still says acceptance tests "run only through explicit local opt-in or the manual GitHub Actions workflow" and that "live acceptance remains a manual pre-tag gate." Once this lands, the document #85 designates as the source of truth for the gate no longer describes the post-merge trigger. Worth fixing those two sentences when #85 rebases.
Preserve the six-surface release gate and align acceptance documentation with the main-branch trigger.
Summary
mainWhy this is release work
The post-merge workflow catches live API, SDK, and Terraform integration
regressions without running side-effecting tests concurrently across every
pull-request branch. Manual dispatch remains available for targeted validation,
and the complete matrix remains a pre-tag release gate.
Tests run
go test -short -timeout=2m ./...git diff --checkAcceptance status
The existing browser-pool and project acceptance tests previously passed on
this branch. They were not rerun for this workflow-trigger and documentation
change.
Deferred work
matrix.
Remaining release gate
Run the complete selected-surface matrix against the release commit before the
first public tag.
Note
Low Risk
CI trigger and documentation only; no provider or test logic changes.
Overview
Post-merge live acceptance — The
AcceptanceGitHub Actions workflow now runs on every push tomain, in addition to manualworkflow_dispatch. PR CI is unchanged; live tests still do not run on pull requests.Docs —
README.md,docs/acceptance.md, anddocs/release.mdnow describe automatic runs after merges tomain, manual dispatch, and local opt-in as the three ways to exercise the selected-surface matrix.Reviewed by Cursor Bugbot for commit 06dd269. Bugbot is set up for automated code reviews on this repo. Configure here.