Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/build-and-publish.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/cd.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/check-release-tag.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 23 additions & 1 deletion .github/workflows/checks.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/ci.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/dependency-update.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/fast-tests.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/gh-pages.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/matrix-all.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/matrix-exasol.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/matrix-python.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/merge-gate.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/periodic-validation.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/pr-merge.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/report.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,7 @@ nosetests.xml

# Emacs
TAGS

# AI
.codex
.serena
7 changes: 6 additions & 1 deletion doc/changes/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,18 @@

## Summary

In this minor release, the nox session `workflow:check` was added and is now used in the `checks.yml`.
Comment thread
ArBridgeman marked this conversation as resolved.
If this job is active in your CI, please double-check if additional files should be added into your project's `.gitattributes`.

## Bugfix

* #840: Added `export` plugin installation within `dependency-update.yml`

## Feature

* #722: Added check in `workflow:generate` to compare the generated and existing content before writing out
* #722: Added check in `workflow:generate` to compare the generated and existing content before writing out and nox session `workflow:check`
* #642: Added nox session `workflow:check` into the `checks.yml`
* #698: Added a comment in the top of all workflows maintained by the PTB

## Refactoring

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,12 @@ Add all Workflows to Your Project

poetry run -- nox -s workflow:generate -- all

After regenerating the workflows, you can verify that they are up-to-date with:

.. code-block:: shell

poetry run -- nox -s workflow:check -- all

.. warning::
Some workflows depend on other workflows. Please ensure you have all
the required workflows if you do not install all of them.
11 changes: 7 additions & 4 deletions doc/user_guide/features/github_workflows/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,15 @@ The PTB ships with configurable GitHub workflow templates covering the most comm
CI/CD setup variants for Python projects. The templates are defined in:
`exasol/toolbox/templates/github/workflows <https://github.com/exasol/python-toolbox/tree/main/exasol/toolbox/templates/github/workflows>`__.

The PTB provides a command line interface (CLI) for generating and updating actual
workflows from the templates.
The PTB provides a command line interface (CLI) for managing workflows from the templates.

.. code-block:: bash

poetry run -- nox -s workflow:generate --help
poetry run -- nox -s workflow:check --help

Use ``workflow:generate`` to create or update workflows and ``workflow:check`` to
compare the rendered workflow templates against the files in ``.github/workflows``.

.. attention::

Expand Down Expand Up @@ -62,8 +65,8 @@ Maintained by the PTB
* - ``checks.yml``
- Workflow call
- Executes many small & fast checks: builds documentation, validates
cross-references and links in the documentation to be valid, and runs various
linters (security, type checks, etc.).
cross-references and links in the documentation to be valid, runs various
linters (security, type checks, etc.), and validates PTB generated workflows.
* - ``ci.yml``
- Pull request
- Executes the continuous integration suite by calling ``merge-gate.yml`` and
Expand Down
10 changes: 10 additions & 0 deletions exasol/toolbox/config.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import inspect
import re
import warnings
from collections.abc import Callable
from pathlib import Path
Expand All @@ -19,12 +20,20 @@
)
from pydantic_core.core_schema import ValidationInfo

from exasol.toolbox import __version__
from exasol.toolbox.nox.plugin import (
METHODS_SPECIFIED_FOR_HOOKS,
PLUGIN_ATTR_NAME,
)
from exasol.toolbox.util.version import Version

WORKFLOW_HEADER_PREFIX = (
"# Generated and maintained by the exasol-toolbox.\n"
"# Last generated with exasol-toolbox version "
)

WORKFLOW_HEADER_PATTERN = rf"\A{re.escape(WORKFLOW_HEADER_PREFIX)}[^\n]+\.\n"


def get_methods_with_hook_implementation(
plugin_class: type[Any],
Expand Down Expand Up @@ -291,6 +300,7 @@ def github_template_dict(self) -> dict[str, Any]:
"minimum_python_version": self.minimum_python_version,
"os_version": self.os_version,
"python_versions": self.python_versions,
"workflow_header": f"{WORKFLOW_HEADER_PREFIX}{__version__}.",
"workflow_extension": {
"fast_tests": fast_tests_extension.is_file(),
"merge_gate": merge_gate_extension.is_file(),
Expand Down
31 changes: 27 additions & 4 deletions exasol/toolbox/nox/_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@
from noxconfig import PROJECT_CONFIG


def _create_parser() -> argparse.ArgumentParser:
def _create_parser(session_name: str) -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="nox -s workflow:generate",
usage="nox -s workflow:generate -- [-h] <workflow_choice>",
prog=f"nox -s {session_name}",
usage=f"nox -s {session_name} -- [-h] <workflow_choice>",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)

Expand All @@ -27,12 +27,35 @@ def _create_parser() -> argparse.ArgumentParser:
return parser


@nox.session(name="workflow:check", python=False)
def check_workflow(session: Session) -> None:
"""
Check the specified GitHub workflow or all of them to see if any differ from
the generated values. If any differ, an error is raised.
"""
parser = _create_parser("workflow:check")
args = parser.parse_args(session.posargs)

PROJECT_CONFIG.github_workflow_directory.mkdir(parents=True, exist_ok=True)

outdated_workflows = WorkflowOrchestrator(
workflow_choice=args.workflow_choice,
config=PROJECT_CONFIG,
).find_differing_workflows()

if outdated_workflows:
count = len(outdated_workflows)
count_label = "workflow is" if count == 1 else "workflows are"
workflow_list = "\n".join(f"- {workflow}" for workflow in outdated_workflows)
session.error(f"\n{count} {count_label} out of date:\n{workflow_list}")


@nox.session(name="workflow:generate", python=False)
def generate_workflow(session: Session) -> None:
"""
Generate or update the specified GitHub workflow or all of them.
"""
parser = _create_parser()
parser = _create_parser("workflow:generate")
args = parser.parse_args(session.posargs)

PROJECT_CONFIG.github_workflow_directory.mkdir(parents=True, exist_ok=True)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build & Publish

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/cd.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: CD

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Check Release Tag

on:
Expand Down
23 changes: 22 additions & 1 deletion exasol/toolbox/templates/github/workflows/checks.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Checks

on:
Expand Down Expand Up @@ -168,7 +169,6 @@ jobs:
id: check-format
run: poetry run -- nox -s format:check


build-package:
name: Build Package
runs-on: "(( os_version ))"
Expand All @@ -189,3 +189,24 @@ jobs:
- name: Build Package
id: build-package
run: poetry run -- nox -s package:check

check-workflows:
name: Check Workflows
runs-on: "(( os_version ))"
permissions:
contents: read
steps:
- name: Check out Repository
id: check-out-repository
uses: actions/checkout@v6

- name: Set up Python & Poetry Environment
id: set-up-python-and-poetry-environment
uses: exasol/python-toolbox/.github/actions/python-environment@v8
with:
python-version: "(( minimum_python_version ))"
poetry-version: "(( dependency_manager_version ))"

- name: Check Workflows
id: check-workflows
run: poetry run -- nox -s workflow:check -- all
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: CI

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Dependency Update

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/fast-tests.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Fast-Tests

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/gh-pages.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Publish Documentation

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/matrix-all.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build Matrix (All Versions)

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build Matrix (Exasol)

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build Matrix (Python)

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/merge-gate.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Merge-Gate

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Periodic-Validation

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/pr-merge.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: PR-Merge

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/report.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Status Report

on:
Expand Down
Loading