Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 5 additions & 8 deletions .github/workflows/cluster-regression.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# Cluster-backed regression (S33). Not on pull requests — Kind/Tekton is too
# slow and too easy to confuse with the PR-gated --local-only job.
#
# Runs: Playwright (no cluster) + Kind (isolation-eval --cluster, Phase 2
# stack-dag-verify, Newman vs orchestrator).
# Cluster-backed regression (S33). Expensive Kind + Tekton.
# Everyday PRs use local-regression.yml. This job is nightly, version tags,
# manual dispatch, and PRs that change cluster install / Helm / this workflow.
name: cluster regression

on:
Expand All @@ -27,9 +25,8 @@ on:
type: boolean
default: true
schedule:
# Daily Kind + Phase 2 + Newman on the default branch (cron does not run on PRs).
# Changing this workflow file on a PR runs the same Kind job; isolation-eval is
# skipped on pull_request and remains part of nightly / dispatch / tags.
# Daily Kind + Phase 2 + Newman on the default branch.
# Isolation-eval is skipped on pull_request and remains part of nightly / dispatch / tags.
- cron: "17 6 * * *"
push:
tags:
Expand Down
25 changes: 3 additions & 22 deletions .github/workflows/intercept-e2e.yml
Original file line number Diff line number Diff line change
@@ -1,39 +1,20 @@
name: intercept product E2E

# Expensive Kind + Telepresence/mirrord. Not an every-PR job.
# Everyday coverage is local-regression.yml. This workflow is weekly, manual,
# and PRs that change the E2E script itself.
on:
pull_request:
paths:
- ".github/workflows/intercept-e2e.yml"
- "helm/tekton-dag/**"
- "libs/tekton-dag-common/**"
- "operator/**"
- "orchestrator/**"
- "pipeline/**"
- "stacks/**"
- "tasks/**"
- "scripts/bootstrap-namespace.sh"
- "scripts/install-tekton.sh"
- "scripts/run-cluster-ci.sh"
- "scripts/run-product-intercept-e2e.sh"
push:
branches: [main]
paths:
- ".github/workflows/intercept-e2e.yml"
- "helm/tekton-dag/**"
- "libs/tekton-dag-common/**"
- "operator/**"
- "orchestrator/**"
- "pipeline/**"
- "stacks/**"
- "tasks/**"
- "scripts/bootstrap-namespace.sh"
- "scripts/install-tekton.sh"
- "scripts/run-cluster-ci.sh"
- "scripts/run-product-intercept-e2e.sh"
workflow_dispatch:
schedule:
# Weekly product-path evidence on the default branch (Monday).
# Changing this workflow file on a PR runs the same telepresence + mirrord jobs as the cron.
- cron: "43 7 * * 1"

permissions:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/local-regression.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
name: local regression

# Default gate: every PR and every push to main. Fast, no Kind cluster.
on:
pull_request:
push:
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/results-regression.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
name: Tekton Results regression

# Expensive Kind + Postgres + Results. Not an every-PR job.
# Everyday coverage is local-regression.yml. This workflow is weekly, manual,
# and PRs that change Results installers / the strict runner.
on:
pull_request:
paths:
Expand All @@ -11,15 +14,13 @@ on:
push:
branches: [main]
paths:
- ".github/workflows/results-regression.yml"
- "scripts/install-postgres-kind.sh"
- "scripts/install-tekton-results.sh"
- "scripts/run-regression-agent-full.sh"
- "scripts/run-lang-unit-tests.sh"
workflow_dispatch:
schedule:
# Weekly strict Results/Postgres evidence on the default branch.
# Changing this workflow file on a PR runs the same Kind + Results job as the cron.
- cron: "19 9 * * 3"

permissions:
Expand Down
19 changes: 17 additions & 2 deletions docs/REGRESSION.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,24 @@ Do **not** confuse these:
| Scope | What runs | Typical trigger |
|-------|-----------|-----------------|
| **Application PR** (`stack-pr-test` on an **app** repo) | Stack-defined tests only — e.g. that app’s Newman/Playwright/Artillery as declared in `stacks/*.yaml`, against the intercept build. | Every PR on the **application** repository (when webhooks/Tekton are wired). |
| **Platform regression** (`scripts/run-regression*.sh` on **this** repo) | **System / integration** tiers: Phase 1 + orchestrator + shared libs + GUI pytest, Playwright for **management-gui**, real **`stack-dag-verify`** PipelineRun, Newman against **orchestrator** API, optional Tekton Results, optional Kind E2E. | **PRs / `main`:** [`.github/workflows/local-regression.yml`](../.github/workflows/local-regression.yml) runs **`--local-only --require-lang-tests`**. **Nightly / dispatch / `v*` tags:** [`.github/workflows/cluster-regression.yml`](../.github/workflows/cluster-regression.yml) runs Playwright + Kind. **Weekly / dispatch:** [`intercept-e2e.yml`](../.github/workflows/intercept-e2e.yml) runs both intercept backends and [`results-regression.yml`](../.github/workflows/results-regression.yml) runs strict Results/Postgres verification. |
| **Platform regression** (`scripts/run-regression*.sh` on **this** repo) | **System / integration** tiers: Phase 1 + orchestrator + shared libs + GUI pytest, Playwright for **management-gui**, real **`stack-dag-verify`** PipelineRun, Newman against **orchestrator** API, optional Tekton Results, optional Kind E2E. | See the trigger table below. |

So: **not all tests run on every PR.** `--local-only` (including Java/PHP/operator) is PR-gated. Playwright, Newman, Phase 2, and Kind isolation measurements run on **cluster-regression** (nightly / `workflow_dispatch` / version tags), not on pull requests. The slower Telepresence and mirrord product paths run weekly and on dispatch. App PRs run a narrower, stack-scoped test stage.
**GitHub Actions — what runs when (Kind is not every PR):**

| Workflow | Every PR / every `main` push | Also runs |
|----------|------------------------------|-----------|
| **local regression** | Yes (`--local-only --require-lang-tests`) | dispatch |
| **docgen demo-function** | Yes (cheap smoke) | |
| **dependency review** | PRs only | |
| **operator** | Only if `operator/**` (or install-tekton) changed | dispatch |
| **cluster regression** | No | Nightly cron, `v*` tags, dispatch, or PRs that touch Helm / cluster-ci scripts |
| **intercept product E2E** | No | Weekly cron, dispatch, or PRs that touch `run-product-intercept-e2e.sh` |
| **Tekton Results** | No | Weekly cron, dispatch, or PRs that touch Results installers |
| **Pages** | No | `main` pushes that touch `docs/**` |

The Actions sidebar can still list **deleted** workflow names (compatibility matrix, demo validation, Graph/GUI Newman, static quality, supply-chain scan). Those files are gone; disable them in the repo **Actions → workflow → … → Disable** so they stop cluttering the list. **Dependabot Updates** is GitHub-managed, not a repo workflow.

So: **not all tests run on every PR.** `--local-only` (including Java/PHP/Go) is the default gate. Playwright + Kind Phase 2 + Newman are **nightly**. Telepresence/mirrord and Results/Postgres are **weekly**. App PRs run a narrower, stack-scoped test stage.

The existence of the intercept workflow is not proof that either backend is
currently healthy. Treat only a recent successful matrix job and its retained
Expand Down
21 changes: 14 additions & 7 deletions libs/tekton-dag-common/tests/test_m17_intercept_automation.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,20 @@ def test_intercept_workflow_has_explicit_backend_cadence_and_evidence():
assert "push:" in workflow
assert "branches: [main]" in workflow
assert '".github/workflows/intercept-e2e.yml"' in workflow
assert '"helm/tekton-dag/**"' in workflow
assert '"operator/**"' in workflow
assert '"orchestrator/**"' in workflow
assert '"pipeline/**"' in workflow
assert '"tasks/**"' in workflow
assert '"scripts/install-tekton.sh"' in workflow
assert '"scripts/bootstrap-namespace.sh"' in workflow
assert '"scripts/run-product-intercept-e2e.sh"' in workflow
# Kind intercept is weekly/manual, not every product PR.
for everyday in (
'"helm/tekton-dag/**"',
'"operator/**"',
'"orchestrator/**"',
'"pipeline/**"',
'"stacks/**"',
'"tasks/**"',
'"scripts/install-tekton.sh"',
'"scripts/bootstrap-namespace.sh"',
'"scripts/run-cluster-ci.sh"',
):
assert everyday not in workflow.split("jobs:", 1)[0]
assert "workflow_dispatch:" in workflow
assert "schedule:" in workflow
assert "backend: [telepresence, mirrord]" in workflow
Expand Down
40 changes: 40 additions & 0 deletions scripts/check-ci-policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,49 @@ def check_operator_go_pin() -> None:
_fail("operator workflow must check Dockerfile Go against go.mod")


def _workflow_on(path: Path) -> dict:
data = yaml.safe_load(path.read_text())
# PyYAML 1.1 treats the key `on` as boolean True.
return data.get("on") or data.get(True) or {}


def check_kind_workflows_are_not_every_pr() -> None:
"""Kind jobs must stay path-filtered / scheduled, not fire on every tasks change."""
on = _workflow_on(ROOT / ".github/workflows/intercept-e2e.yml")
forbidden = {
"helm/tekton-dag/**",
"libs/tekton-dag-common/**",
"operator/**",
"orchestrator/**",
"pipeline/**",
"stacks/**",
"tasks/**",
"scripts/bootstrap-namespace.sh",
"scripts/install-tekton.sh",
"scripts/run-cluster-ci.sh",
}
for event in ("pull_request", "push"):
paths = set((on.get(event) or {}).get("paths") or [])
overlap = paths & forbidden
if overlap:
_fail(f"intercept {event} paths are too broad (Kind on every product PR): {sorted(overlap)}")
if "schedule" not in on or "workflow_dispatch" not in on:
_fail("intercept-e2e.yml must keep schedule + workflow_dispatch")

results_on = _workflow_on(ROOT / ".github/workflows/results-regression.yml")
results_push = set((results_on.get("push") or {}).get("paths") or [])
if ".github/workflows/results-regression.yml" in results_push:
_fail("results push.paths must not include the workflow file (avoids re-running Kind on merge)")

intercept_push = set((on.get("push") or {}).get("paths") or [])
if ".github/workflows/intercept-e2e.yml" in intercept_push:
_fail("intercept push.paths must not include the workflow file (avoids re-running Kind on merge)")


def main() -> None:
check_dependabot()
check_operator_go_pin()
check_kind_workflows_are_not_every_pr()
print("CI policy checks passed")


Expand Down
Loading