Skip to content

Allow the e2e suites to be run manually against master - #833

Merged
wallrj-cyberark merged 1 commit into
masterfrom
e2e-workflow-dispatch
Sep 9, 2026
Merged

Allow the e2e suites to be run manually against master#833
wallrj-cyberark merged 1 commit into
masterfrom
e2e-workflow-dispatch

Conversation

@wallrj-cyberark

Copy link
Copy Markdown
Contributor

Why now?

We want to run the e2e suites against master before tagging v1.12.0-alpha.0. Right now there is no way to do that.

All three e2e jobs are gated on github.event.pull_request.labels, which does not exist for a push event. So they skip on every push to master as well as on every unlabelled pull request. Nothing has run them on master for some time.

The workaround is to open a pull request and label it, which is what we did for #831. That has two problems: it tests the pull request's merge commit rather than master itself, and it has to be repeated by hand every release.

What this changes

Adds workflow_dispatch and lets the three e2e gates fire on it:

if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'test-ark')

That gives a Run workflow button which runs all three suites against any ref, including master. The label route is unchanged for pull requests, so existing behaviour is untouched.

Two things I checked rather than assumed

The GKE cluster still gets deleted on a manual run. The cleanup step is gated on always() && !contains(github.event.pull_request.labels.*.name, 'keep-e2e-cluster'). On a workflow_dispatch run github.event.pull_request is null, so the filter yields nothing, contains is false, and the negation is true. The cluster is cleaned up. The failure mode here would have been leaking a GKE cluster on every manual run, so it is worth being explicit that it does not.

The cluster name does not depend on the pull request. It is test-secretless-$(date +'%y%m%d-%H%M%S'), with a comment explaining that extracting from the PR name would need sanitising for GKE naming rules. So it is already safe outside a pull request context.

One quirk to expect

The Run workflow button will not appear until this is merged. GitHub only reads workflow_dispatch from the workflow file on the default branch, so you cannot test the button from this pull request. The if expressions are the only new logic and they are plain string comparisons.

Deliberately not doing yet

I have not added a schedule, and I have not made e2e run on every push to master. Both are tempting, but ark-test-e2e still carries its TEMPORARY comment about a recurring 400 "conflicting tagging values" error, described as flaky rather than broken. Making a known-flaky suite gate every merge would be worse than the current situation. A nightly schedule is the better long-term answer and is tracked internally along with the question of whether that flake is still real.

Testing

make verify passes. The YAML parses and the three gates resolve as intended:

triggers: ['push', 'pull_request', 'workflow_dispatch']
ark-test-e2e  => github.event_name == 'workflow_dispatch' || contains(...'test-ark')
ngts-test-e2e => github.event_name == 'workflow_dispatch' || contains(...'test-ngts')
test-e2e      => github.event_name == 'workflow_dispatch' || contains(...'test-e2e')

The real proof is dispatching it against master once merged, which is the next step before tagging.

The three e2e jobs are gated on `github.event.pull_request.labels`, which
is empty for a push. They therefore skip on every push to master as well
as on every unlabelled pull request, so nothing has run them on master
for some time.

That matters most just before a release, when we want to know that the
commit we are about to tag actually passes end to end. Until now the only
way to get an e2e run was to open a pull request and label it, which
tests a merge commit rather than master itself.

Adding `workflow_dispatch` gives a Run workflow button that runs all
three suites against any ref, including master. The label route is
unchanged for pull requests.

Two things stay as they are:

- The GKE cluster cleanup guard reads the same missing labels field, so
  on a manual run it evaluates to true and the cluster is still deleted.
- The cluster name is derived from a timestamp, not the pull request, so
  it is already safe outside a pull request context.

Note that the Run workflow button only appears once this is on the
default branch, since that is where GitHub reads workflow_dispatch from.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard.wall@cyberark.com>
@wallrj-cyberark

Copy link
Copy Markdown
Contributor Author

The label route still works

Added test-e2e, test-ark and test-ngts to this PR to prove the change does not break the existing path. All five checks pass:

Job Result
verify pass, 1m12s
test pass, 42s
ark-test-e2e pass, 3m04s
ngts-test-e2e pass, 2m46s
test-e2e pass, 14m19s

This is the case that could plausibly have regressed. The gate went from contains(...) to github.event_name == 'workflow_dispatch' || contains(...). On a pull_request event the first operand is false, so the whole thing depends on the fall-through to contains(...) still evaluating as before. It does — the three jobs ran rather than skipped.

Checked the e2e jobs did real work rather than passing vacuously, since two finishing in ~3m looks like a no-op

Log counts from run 34340761638:

  • ark-test-e2e — 21 "Successfully gathered" lines and a "Data sent successfully".
  • ngts-test-e2e — 36 "Successfully gathered" lines and a "Data sent successfully".
  • test-e2e — created GKE cluster test-secretless-260909-103321 and logged one Deleted [...], so the cleanup guard fired and the cluster did not leak.

That last point also exercises the cleanup path on a pull_request event, which is the behaviour I reasoned about in the description for the dispatch case.

Still not exercised here

The workflow_dispatch path itself. GitHub only offers the Run workflow button for workflows on the default branch, so it cannot be triggered from this branch. The new logic is a plain string comparison against github.event_name, and this run confirms the other half of the || is intact, but the dispatch route is untested until this merges.

First thing to do after merging is dispatch it against master. That is the step the release is waiting on.

@wallrj-cyberark
wallrj-cyberark merged commit 7e728f6 into master Sep 9, 2026
10 checks passed
@wallrj-cyberark
wallrj-cyberark deleted the e2e-workflow-dispatch branch September 9, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test-ark test-e2e To signal e2e test job to be run test-ngts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants