Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/changes/changes_10.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ This release fixes vulnerabilities by updating dependencies:
* #699: Added `all-extras` support to the Python environment GitHub action
* #875: Added `name` attribute to generated workflow jobs using `-extension.yml` workflows

## Bug
## Bug Fix

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


* #744: Updated nox DB-version handling to use `BaseConfig.minimum_exasol_version` instead hardcoded `7.1.9`

Expand Down Expand Up @@ -80,4 +80,4 @@ This release fixes vulnerabilities by updating dependencies:
* Updated dependency `pip-audit:2.10.0` to `2.10.1`
* Updated dependency `pylint:4.0.5` to `4.0.6`
* Updated dependency `pytest:9.0.3` to `9.1.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
2 changes: 1 addition & 1 deletion doc/changes/changes_10.2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This minor release adds automated vulnerability updates through Nox session
`vulnerabilities:update` and improves the `dependency-update.yml`. It also includes a
few workflow-related bug fixes and documentation updates.

## Bug
## Bug Fix

* #909: Updated `cd.yml` workflow so that `cd-extension.yml` workflow depends on `build-and-publish`. This ensures that the custom release workflow only runs when the PyPi release was successful.
* #910: Added `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
Expand Down
2 changes: 0 additions & 2 deletions doc/changes/changes_10.2.1.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# 10.2.1 - 2026-07-08

## Summary

## Bug Fix

* #920: Ensured extracted secrets are unique and alphabetically sorted from the custom workflows
4 changes: 4 additions & 0 deletions doc/changes/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Summary

## Feature

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


* #922: Extended `custom_workflows` of `github_template_dict` for automatic custom workflow permissions extraction

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* #922: Extended `custom_workflows` of `github_template_dict` for automatic custom workflow permissions extraction
* #922: Extended `custom_workflows` of `github_template_dict` for automatic custom workflow permissions extraction


## Refactoring

* #924: Removed `lint:dependencies` usage from `report.yml` and added deprecation notice
45 changes: 41 additions & 4 deletions doc/user_guide/features/github_workflows/workflow_variables.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,22 @@ standardized baseline that can be overridden in individual projects.
:start-at: github_template_dict
:end-before: @computed_field

.. _custom_workflow_metadata:

Custom Workflow Metadata
^^^^^^^^^^^^^^^^^^^^^^^^

The PTB extracts metadata from reusable custom workflow files and exposes it

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the general and technical description.

Could we, for more straight-forward and impatient users, add a small section with concrete instructions?

  • What do I need to do?
  • When? (In which Scenario / Use Case / Problem)
  • Example, maybe a pointer to one of the PTB's workflow templates illustrating the usage?

through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
``custom_workflows`` entry. PTB-controlled workflow templates use that metadata
when they call reusable workflows.

.. _custom_workflow_secrets:

Custom Workflow Secrets
^^^^^^^^^^^^^^^^^^^^^^^
Secrets
-------

The PTB extracts secret names from reusable custom workflow files and exposes them
The PTB extracts secret names from custom workflow files and exposes them

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The PTB extracts secret names from custom workflow files and exposes them
The PTB extracts the names of secrets from custom workflow files and exposes them

through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
``custom_workflows`` entry. PTB-controlled workflow templates use those extracted
names when they call reusable workflows and forward secrets via ``secrets:``.
Expand All @@ -50,6 +60,33 @@ For example, ``slow-checks.yml`` can define its reusable workflow header like th
Those extracted secret names are then made available to the PTB templates that
reference the custom workflow.

.. _custom_workflow_permissions:

Permissions
-----------

The PTB extracts the permissions required by custom workflow files. It reads every job's
``permissions`` block and combines the results into a single ordered mapping, where
the most permissive level wins.

Please only configure the minimum required permissions by granting the least required
access. In practice, ``contents: read`` is the most common baseline for workflows, and
other permissions should only be added when it is truly required.

For example, a custom workflow can declare permissions like this:

.. code-block:: yaml

name: Slow-Checks

on:
workflow_call:

jobs:
run-integration-tests:
permissions:
contents: read

.. _workflow_matrix:

Matrix Combinations
Expand All @@ -64,7 +101,7 @@ Extending the Matrix
^^^^^^^^^^^^^^^^^^^^

If you need to expose additional values via the ``matrix.yml``, you can extend
:class:`exasol.toolbox.config.BaseConfig`.
:class:`exasol.toolbox.config.BaseConfig`.

The example adds two additional matrix dimensions: A declared one
`extra_matrix_value` and a computed one `computed_matrix_value`. Each of them
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["cd-extension"].permissions %)
permissions:
contents: write
(% for permission_name, permission_level in custom_workflows["cd-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)

(% if has_documentation %)
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["merge-gate"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["merge-gate"].permissions.items() %)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this pattern repeats 3-4 times - would it make sense to move the functionality into an API function
and in the jinja template only have a call like this (not sure if this is possible)?:

(% workflow_permissions("merge-gate") %)

(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

report:
name: Report
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/fast-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["fast-tests-extension"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["fast-tests-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)
12 changes: 10 additions & 2 deletions exasol/toolbox/templates/github/workflows/merge-gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["slow-checks"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["slow-checks"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

(% if custom_workflows["merge-gate-extension"].exists %)
merge-gate-extension:
Expand All @@ -72,8 +76,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["merge-gate-extension"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["merge-gate-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)

# This job ensures inputs have been executed successfully.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["slow-checks"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["slow-checks"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

report:
name: Report
Expand Down
59 changes: 58 additions & 1 deletion exasol/toolbox/util/workflows/custom_workflow.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

from enum import IntEnum
from pathlib import Path

from pydantic import (
Expand All @@ -13,12 +14,43 @@
from exasol.toolbox.util.workflows.render_yaml import parse_yaml_text


class PermissionRank(IntEnum):
"""GitHub permission levels ranked from least to most permissive."""

NONE = 0
READ = 1
WRITE = 2

@classmethod
def _missing_(cls, value: object) -> PermissionRank:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get an error, that PermissionRank is unknown!?

Suggested change
def _missing_(cls, value: object) -> PermissionRank:
def _missing_(cls, value: object) -> Self:

"""Convert a GitHub permission string to its rank."""
if isinstance(value, str):
for rank in cls:
if rank.name.lower() == value:
return rank
raise ValueError(f"Unknown GitHub permission level: {value!r}")


def merge_permissions(permission_maps: list[dict[str, str]]) -> dict[str, str]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can get the goal of the implementation, but my stomach says there must be a simpler, shorter way.

How about this approach?

Permissions: TypeAlias = dict[str, str]
"""
Each permission is defined by a name as it key and a value, which is
either None, or "read" or "write".
"""

def merge_permissions(permission_maps: list[Permissions]) -> Permissions:
    """Merge permission maps to keep the greater permission."""
    rank = {None: 0, "read": 1, "write": 2}
    result = {}
    def max_permission(perm: str, value: str) -> str:
        current = result.get(perm)
        return value if rank[value] > rank[current] else current

    for other in permission_maps:
        result |= {p: max_permission(p, v) for p, v in other.items()}
    return result

No enum required, < 70% characters, tests still green.

"""Merge permission maps to keep the greater permission."""
merged_permissions: dict[str, str] = {}
for permission_map in permission_maps:
for permission_name, requested_level in permission_map.items():
Comment thread
ArBridgeman marked this conversation as resolved.
current_level = merged_permissions.get(permission_name, None)
if current_level is None:
merged_permissions[permission_name] = requested_level
continue
if PermissionRank(requested_level) > PermissionRank(current_level): # type: ignore[arg-type]
merged_permissions[permission_name] = requested_level
return merged_permissions


class CustomWorkflow(BaseModel):
"""A project-owned workflow used for seeded workflows and extensions.

These workflows are seeded by the PTB or extend PTB-provided workflows, but
they are maintained by the project itself rather than the PTB. See
`Custom Workflows <file:///home/chku/git/ptb/.html-documentation/user_guide/features/github_workflows/index.html#custom-workflows>`__.
`Custom Workflows <https://exasol.github.io/python-toolbox/main/user_guide/features/github_workflows/index.html#custom-workflows>`__.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uh - Oh - sorry for this accidental artifact!

"""

model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)
Expand Down Expand Up @@ -53,3 +85,28 @@ def extract_secrets(self) -> tuple[str, ...]:
if secrets := workflow_call.get("secrets", {}):
return tuple(secrets.keys())
return ()

def extract_permissions(self) -> dict[str, str]:
"""Return the effective job permissions required by the workflow.

The extractor scans all jobs and merges their ``permissions`` blocks into a
single mapping. When the same permission appears multiple times, the more
permissive level wins while preserving the first-seen order of the keys.

For example, a custom workflow can declare permissions like this:

.. code-block:: yaml

name: Slow-Checks

on:
workflow_call:

jobs:
run-integration-tests:
permissions:
contents: read
"""
jobs = self.yaml_content.get("jobs", {})
permission_maps = [job.get("permissions", {}) for job in jobs.values()]
return merge_permissions(permission_maps)
Comment thread
ArBridgeman marked this conversation as resolved.
28 changes: 26 additions & 2 deletions exasol/toolbox/util/workflows/custom_workflow_extractor.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,35 @@
field_validator,
)

from exasol.toolbox.util.workflows.custom_workflow import CustomWorkflow
from exasol.toolbox.util.workflows.custom_workflow import (
CustomWorkflow,
merge_permissions,
)

MINIMUM_GITHUB_PERMISSIONS: dict[str, str] = {"contents": "read"}


class CustomWorkflowEntry(BaseModel):
model_config = ConfigDict(frozen=True)

exists: bool
secrets: tuple[str, ...]
permissions: dict[str, str]

@field_validator("secrets", mode="before")
@classmethod
def _normalize_secrets(cls, secrets: tuple[str, ...]) -> list[str]:
"""Return unique secret names in alphabetical order."""
return sorted(set(secrets))

@field_validator("permissions", mode="before")
@classmethod
def _normalize_permissions(
cls, permissions: list[dict[str, str]]
) -> dict[str, str]:
"""Merge permission maps while preserving first-seen order."""
return merge_permissions(permissions)


class CustomWorkflowExtractor(BaseModel):
model_config = ConfigDict(frozen=True)
Expand All @@ -43,13 +57,18 @@ def _build_custom_workflow_entry(
file_path = self.github_workflow_directory / f"{workflow}.yml"

secrets: tuple[str, ...] = ()
permissions: list[dict[str, str]] = []
if file_path.is_file():
custom_workflow = CustomWorkflow.load_from_file(file_path=file_path)
secrets = custom_workflow.extract_secrets()

permissions = [
MINIMUM_GITHUB_PERMISSIONS,
custom_workflow.extract_permissions(),
]
return CustomWorkflowEntry(
exists=file_path.exists(),
secrets=secrets,
permissions=permissions,
)

def _build_merge_gate_entry(
Expand All @@ -61,6 +80,11 @@ def _build_merge_gate_entry(
+ custom_workflows_dict["slow-checks"].secrets
# from the `report.yml`
+ (self.sonar_token_name,),
permissions=[
MINIMUM_GITHUB_PERMISSIONS,
custom_workflows_dict["merge-gate-extension"].permissions,
custom_workflows_dict["slow-checks"].permissions,
],
)

def build_custom_workflow_dict(
Expand Down
12 changes: 0 additions & 12 deletions exasol/toolbox/util/workflows/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,18 +99,6 @@ def __init__(self, workflow_name: str, valid_workflows):
)


class NotMaintainedWorkflowError(ValueError):
"""
Raised when a PTB-seeded workflow is requested in an existing project.
"""

def __init__(self, workflow_name: str):
super().__init__(
f"Workflow '{workflow_name}' is a PTB-seeded workflow that is "
"originally provided by the PTB and can only be seeded for a new project."
)


class YamlKeyError(Exception):
"""
Base exception for when a specified value cannot be found in a YAML.
Expand Down
Loading