Skip to content
Open
22 changes: 22 additions & 0 deletions .github/actions/plugin-check-report/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,8 @@ the check to require in branch protection is `plugin-check / Plugin Check`.
| `cache-paths` | empty | Build caches kept between runs, one path per line, relative to the repository |
| `mode` | `baseline` | `baseline` or `strict`. See [Modes](#modes) |
| `submodules` | `false` | Passed to `actions/checkout` |
| `head-ref` | empty | Manual runs only: the branch, tag or commit to check. Empty checks the ref the run started on. See [Manual runs](#manual-runs) |
| `base-ref` | empty | Manual runs only: the branch, tag or commit to compare with. See [Manual runs](#manual-runs) |

| Secret | Becomes, in the build step only |
|---|---|
Expand All @@ -257,6 +259,25 @@ the check to require in branch protection is `plugin-check / Plugin Check`.
The secrets never reach the job's environment, so the third-party Plugin Check step that runs later
in the same job does not see them there.

### Manual runs

A `workflow_dispatch` run has no pull request, so it has no head, no target and no comment. Two
inputs say what a pull request would say:

- `head-ref` is what is built and checked. Empty means the ref the run was started on.
- `base-ref` is what it is compared with. The baseline is `.plugin-check-baseline.json` on that ref,
and the excluded checks are its `.plugin-check.json`, so a branch older than the Plugin Check
setup still runs with the right ones. With no baseline file there, `base-ref` is built and checked
too, as a pull request's target is. The `exclude-checks` input still replaces the file.

The caller passes them from its own `workflow_dispatch` inputs; see
`templates/workflows/plugin-check.yml`. For a pull request or a push both are empty, and nothing
changes.

There is no comment. The result is in the run's summary page, in the `plugin-check-report` artifact,
and in the status of the run, which fails by the same rule as a pull request: a rule with more errors
than the baseline in `baseline` mode, any error in `strict` mode.

### Permissions the caller grants

```yaml
Expand All @@ -281,6 +302,7 @@ not start the run. The job that runs the Plugin Check action lowers itself to `c
| `base-results` | no | none | Results of the target branch, used as the baseline when there is no baseline file. Not used in `strict` mode |
| `base-required` | no | `false` | `true` when the caller checked the target branch: a missing `base-results` file then fails the compare |
| `baseline-changed` | no | `false` | `true` when the pull request changes the baseline file; the comment says so |
| `base-ref` | no | none | A manual run's `base-ref`. When the baseline file is missing, the notes name this ref instead of saying "there is no baseline file" |

| Output | What it is |
|---|---|
Expand Down
7 changes: 7 additions & 0 deletions .github/actions/plugin-check-report/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@ inputs:
passes as "no baseline".
required: false
default: 'false'
base-ref:
description: >-
The ref a manual run compares with. Empty for a pull request or a push. When the baseline file
is missing, the notes name this ref, because the checked ref may have a baseline file of its own.
required: false
default: ''
baseline-changed:
description: >-
`true` when the pull request changes the baseline file. The comment then says so, because the
Expand Down Expand Up @@ -125,6 +131,7 @@ runs:
PLUGIN_CHECK_BASE_RESULTS: ${{ inputs.base-results }}
PLUGIN_CHECK_BASE_REQUIRED: ${{ inputs.base-required }}
PLUGIN_CHECK_BASELINE_CHANGED: ${{ inputs.baseline-changed }}
PLUGIN_CHECK_BASE_REF: ${{ inputs.base-ref }}
COMMENT_MARKER: ${{ steps.paths.outputs.marker }}
ARTIFACT_URL: ${{ steps.artifact.outputs.artifact-url }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
Expand Down
27 changes: 20 additions & 7 deletions .github/actions/plugin-check-report/plugin-check-report.js
Original file line number Diff line number Diff line change
Expand Up @@ -623,10 +623,16 @@ function buildComment({ marker, status, badge, column, noBaseline, totals, wentU
* @param {string} input.source `file`, `target` or `none`.
* @param {boolean} input.changed The pull request changes the baseline file, so the comparison is
* with its own change. A reviewer should look at the diff of the file.
* @param {string} input.baseRef The ref a manual run compares with, or ''. The note names it,
* because the checked ref may have a baseline file of its own.
*
* @returns {string} The line, or ''.
*/
function baselineNote({ source, changed }) {
function baselineNote({ source, changed, baseRef = '' }) {
if (source === 'target' && baseRef) {
return `ℹ️ ${baseRef} has no baseline file, so its own results are the baseline. See the job summary to add one.`
}

if (source === 'target') {
return "ℹ️ There is no baseline file, so the baseline is the target branch's results. See the job summary to add one."
}
Expand Down Expand Up @@ -713,16 +719,19 @@ function readTargetResults(env, name) {
* @param {object} input.verdict From evaluate.
* @param {string} input.baselineName The baseline file's name.
* @param {object} input.env GITHUB_RUN_ID and GITHUB_REPOSITORY, for the real command.
* @param {string} input.baseRef The ref a manual run compares with, or ''. Names where the file is missing.
*
* @returns {string} Markdown with a final line break, or ''.
*/
function updateSection({ mode, noFile, verdict, baselineName, env }) {
function updateSection({ mode, noFile, verdict, baselineName, env, baseRef = '' }) {
if (mode === 'strict') {
return ''
}

const lines = []
if (noFile) {
if (noFile && baseRef) {
lines.push(`${baseRef} has no ${baselineName} yet. To start one there, add the file from this run:`)
} else if (noFile) {
lines.push(`There is no ${baselineName} yet. To start one, add the file from this run:`)
} else if (verdict.errorsDown > 0) {
lines.push(`Error counts went down. To lower the baseline, replace ${baselineName} with the file from this run:`)
Expand Down Expand Up @@ -761,7 +770,8 @@ function updateSection({ mode, noFile, verdict, baselineName, env }) {
* used as the baseline when there is no file) and
* PLUGIN_CHECK_BASE_REQUIRED (`true`: they must exist),
* PLUGIN_CHECK_BASELINE_CHANGED (`true`: this pull request changes the
* baseline file), COMMENT_BADGE,
* baseline file), PLUGIN_CHECK_BASE_REF (a manual run's base-ref, named
* in the notes about a missing baseline file), COMMENT_BADGE,
* RUN_URL, ARTIFACT_URL, GITHUB_STEP_SUMMARY.
* @param {object} input.log Where messages go: `out`, `err`, `error`, `warning`, `notice`.
*
Expand Down Expand Up @@ -875,8 +885,11 @@ function compareOrThrow({ args, env, log }) {
} else if (env.PLUGIN_CHECK_BASE_RESULTS || env.PLUGIN_CHECK_BASE_REQUIRED === 'true') {
base = readTargetResults(env, baselineName)
source = 'target'
const baseRef = env.PLUGIN_CHECK_BASE_REF || ''
log.warning(
`No baseline file at ${baselineName}, so the target branch was checked too, and its results are the baseline for this run. That makes the run slower. To stop it, commit the baseline.json from this run's plugin-check-report artifact as ${baselineName}.`
baseRef
? `${baseRef} has no baseline file at ${baselineName}, so ${baseRef} was checked too, and its results are the baseline for this run. That makes the run slower. To stop it, commit the baseline.json from this run's plugin-check-report artifact as ${baselineName} on ${baseRef}.`
: `No baseline file at ${baselineName}, so the target branch was checked too, and its results are the baseline for this run. That makes the run slower. To stop it, commit the baseline.json from this run's plugin-check-report artifact as ${baselineName}.`
)
} else {
log.warning(
Expand Down Expand Up @@ -913,7 +926,7 @@ function compareOrThrow({ args, env, log }) {
wentUp: verdict.wentUp,
wentDown: verdict.wentDown,
updateHint: mode !== 'strict' && hasFile && verdict.errorsDown > 0,
note: baselineNote({ source, changed: env.PLUGIN_CHECK_BASELINE_CHANGED === 'true' }),
note: baselineNote({ source, changed: env.PLUGIN_CHECK_BASELINE_CHANGED === 'true', baseRef: env.PLUGIN_CHECK_BASE_REF || '' }),
upList: verdict.upList,
links: linksLine({ artifactUrl: env.ARTIFACT_URL || '', runUrl, resultsPath }),
})
Expand All @@ -922,7 +935,7 @@ function compareOrThrow({ args, env, log }) {
writeText(commentFile, `${body}\n`, 'write')
}
if (env.GITHUB_STEP_SUMMARY) {
const how = updateSection({ mode, noFile: !hasFile, verdict, baselineName, env })
const how = updateSection({ mode, noFile: !hasFile, verdict, baselineName, env, baseRef: env.PLUGIN_CHECK_BASE_REF || '' })
writeText(env.GITHUB_STEP_SUMMARY, `${body}\n${how}`, 'append')
}

Expand Down
99 changes: 91 additions & 8 deletions .github/workflows/plugin-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ name: Plugin Check
# more than the caller gave. `concurrency` is the caller's too: here it would be evaluated with the
# caller's context and could clash with the caller's own group.
#
# A manual run (workflow_dispatch) can pass `head-ref` and `base-ref`: the branch to check, and the
# branch to compare it with. There is no pull request then, so no comment is written: the result is
# in the run's summary and in the plugin-check-report artifact, and the run fails the same way.
# `base-ref` names where the settings come from: its `.plugin-check-baseline.json` and its
# `.plugin-check.json` (the excluded checks), so a branch older than the Plugin Check setup still
# runs with the right ones. With no baseline file there, `base-ref` is built and checked too. Both inputs are empty for a pull request or a push.
#
# Run it locally: see "Running it locally" in .github/actions/plugin-check-report/README.md. In
# short, `wp plugin check <slug> --format=json > results.txt` in a local WordPress, then
# `node .github/actions/plugin-check-report/plugin-check-report.js compare results.txt`.
Expand Down Expand Up @@ -109,6 +116,21 @@ on:
required: false
type: string
default: 'false'
head-ref:
description: >-
For a manual run: the branch, tag or commit to check. Empty checks the ref the run was
started on. Leave it empty for a pull request or a push.
required: false
type: string
default: ''
base-ref:
description: >-
For a manual run: the branch, tag or commit to compare with. The baseline is its
`.plugin-check-baseline.json`; with no such file, it is built and checked too. Empty keeps
the baseline file of the checked ref. Leave it empty for a pull request or a push.
required: false
type: string
default: ''
secrets:
composer-token:
description: >-
Expand Down Expand Up @@ -173,36 +195,91 @@ jobs:
matrix: ${{ steps.target.outputs.matrix }}
base: ${{ steps.target.outputs.base }}
checks: ${{ steps.exclude.outputs.checks }}
head-sha: ${{ steps.refs.outputs.head }}
base-sha: ${{ steps.refs.outputs.base }}
steps:
- uses: actions/checkout@v7
with:
path: source
ref: ${{ inputs.head-ref }}
sparse-checkout: |
.plugin-check.json
.plugin-check-baseline.json
sparse-checkout-cone-mode: false
persist-credentials: false

# The two files of base-ref, when a manual run names one. A branch older than the Plugin
# Check setup has neither, and without .plugin-check.json it would run the checks that cannot
# work, so base-ref decides both the baseline and the excluded checks.
- name: Check out the settings of base-ref
if: ${{ inputs.base-ref != '' }}
uses: actions/checkout@v7
with:
path: base
ref: ${{ inputs.base-ref }}
sparse-checkout: |
.plugin-check.json
.plugin-check-baseline.json
sparse-checkout-cone-mode: false
persist-credentials: false

# A manual run's refs, as commits, once: the jobs after this one use these, so a push during
# the run cannot make the settings, the build and the report come from different commits.
# Empty for a pull request or a push, which use the event's own refs.
- name: Resolve head-ref and base-ref to commits
id: refs
env:
HEAD_REF: ${{ inputs.head-ref }}
BASE_REF: ${{ inputs.base-ref }}
run: |
if [ -n "$HEAD_REF" ]; then
echo "head=$(git -C source rev-parse HEAD)" >> "$GITHUB_OUTPUT"
fi
if [ -n "$BASE_REF" ]; then
echo "base=$(git -C base rev-parse HEAD)" >> "$GITHUB_OUTPUT"
fi

# The file the settings come from: the one on base-ref when a manual run names it and has the
# file, else the checked ref's own.
- name: Pick the settings file
id: settings
env:
BASE_REF: ${{ inputs.base-ref }}
run: |
config=source/.plugin-check.json
if [ -n "$BASE_REF" ] && [ -f base/.plugin-check.json ]; then
config=base/.plugin-check.json
fi
echo "config=$GITHUB_WORKSPACE/$config" >> "$GITHUB_OUTPUT"

# The checks to skip: the `exclude-checks` input when it names any, else the list in the
# repository's .plugin-check.json. Like the report job, this needs a runner of version 2.336.0
# settings file (.plugin-check.json). Like the report job, this needs a runner of version 2.336.0
# or newer for `$/`; the two small jobs run on ubuntu-latest, which has it.
- name: List the excluded checks
id: exclude
uses: $/.github/actions/plugin-check-report/exclude-checks
with:
exclude-checks: ${{ inputs.exclude-checks }}
config-file: ${{ github.workspace }}/source/.plugin-check.json
config-file: ${{ steps.settings.outputs.config }}

# A pull request in baseline mode, in a repository with no baseline file, is compared with the
# target branch, so that commit is built and checked too. Strict mode needs no baseline, and a
# push has no target to compare with.
# A pull request, or a manual run with a base-ref, in baseline mode, in a repository with no
# baseline file, is compared with the target branch, so that commit is built and checked too.
# Strict mode needs no baseline, and a push has no target to compare with. The baseline file
# is the one on base-ref when there is one, else the one of the checked ref.
- name: Decide whether to check the target branch
id: target
env:
EVENT: ${{ github.event_name }}
MODE: ${{ inputs.mode }}
HEAD_REF: ${{ inputs.head-ref }}
BASE_REF: ${{ inputs.base-ref }}
run: |
if [ "$EVENT" = pull_request ] && [ "$MODE" != strict ] && [ ! -f source/.plugin-check-baseline.json ]; then
baseline=source/.plugin-check-baseline.json
[ -z "$BASE_REF" ] || baseline=base/.plugin-check-baseline.json
if [ "$EVENT" = workflow_dispatch ]; then
echo "::notice title=Plugin Check::Checking ${HEAD_REF:-$GITHUB_REF_NAME} against ${BASE_REF:-the baseline file of ${HEAD_REF:-$GITHUB_REF_NAME}}."
fi
if { [ "$EVENT" = pull_request ] || [ -n "$BASE_REF" ]; } && [ "$MODE" != strict ] && [ ! -f "$baseline" ]; then
echo 'matrix={"target":["head","base"]}' >> "$GITHUB_OUTPUT"
echo 'base=true' >> "$GITHUB_OUTPUT"
echo "::notice title=Plugin Check::There is no .plugin-check-baseline.json, so the target branch is built and checked too. Commit the baseline.json from the plugin-check-report artifact of this run to stop that."
Expand Down Expand Up @@ -241,8 +318,10 @@ jobs:
- uses: actions/checkout@v7
with:
path: source
# "base" is the commit the pull request targets, as it is without the pull request.
ref: ${{ matrix.target == 'base' && github.event.pull_request.base.sha || '' }}
# "base" is the commit the pull request targets, as it is without the pull request, or the
# base-ref of a manual run. "head" is the head-ref of a manual run, else the run's own ref.
# A manual run's refs are the commits the plan job resolved.
ref: ${{ matrix.target == 'base' && (github.event.pull_request.base.sha || needs.plan.outputs.base-sha) || needs.plan.outputs.head-sha }}
submodules: ${{ inputs.submodules }}
# Nothing here pushes over git, and the job goes on to run third-party code.
persist-credentials: false
Expand Down Expand Up @@ -484,6 +563,9 @@ jobs:
uses: actions/checkout@v7
with:
path: source
# A manual run takes the baseline file from base-ref, or from head-ref when there is none:
# the same commits the plan job resolved.
ref: ${{ needs.plan.outputs.base-sha || needs.plan.outputs.head-sha }}
sparse-checkout: .plugin-check-baseline.json
sparse-checkout-cone-mode: false
# The merge commit and its two parents, to see whether the pull request changes the file.
Expand Down Expand Up @@ -549,6 +631,7 @@ jobs:
base-results: ${{ needs.plan.outputs.base == 'true' && env.BASE_RESULTS || '' }}
base-required: ${{ needs.plan.outputs.base == 'true' }}
baseline-changed: ${{ steps.changed.outputs.changed == 'true' }}
base-ref: ${{ inputs.base-ref }}

- name: Post the result
if: ${{ !cancelled() && env.COMMENT == 'true' }}
Expand Down
21 changes: 19 additions & 2 deletions templates/workflows/plugin-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,25 @@ on:
# The branches pull requests target. A push there keeps them honest after a merge.
push:
branches: [main]
# A manual run: the branch to check, and the branch whose baseline it is compared with. The
# result is in the run's summary, as there is no pull request to comment on.
workflow_dispatch:
inputs:
head-ref:
description: Branch to check
required: true
base-ref:
description: Branch to compare with
required: true
# Change this to the branch your pull requests target. It also has to hold the plugin's
# .plugin-check.json and .plugin-check-baseline.json, which the run reads from it.
default: main

# A new push to a pull request cancels the run still going for the older commit.
# A new push to a pull request cancels the run still going for the older commit. A manual run has a
# group of its own (the run id), so it never cancels another: group names ignore case, and two
# branches can differ only by case.
concurrency:
group: plugin-check-${{ github.event.pull_request.number || github.ref }}
group: plugin-check-${{ github.event.pull_request.number || (github.event_name == 'workflow_dispatch' && github.run_id) || github.ref }}
cancel-in-progress: true

permissions:
Expand All @@ -41,3 +55,6 @@ jobs:
plugin-slug: my-plugin
# The pup release your release build uses. It builds the plugin for the check.
pup-version: '2.0.0'
# Empty for a pull request or a push, so those behave as before.
head-ref: ${{ inputs.head-ref }}
base-ref: ${{ inputs.base-ref }}
Loading