Do not pick up before the trigger condition fires.
Trigger: #1480 reports. That spike classifies every core-tier section as
invariant policy or process layer, and the section this issue adds to is
classified as moving. Writing the rule first means writing it into a file whose
home is undecided, and the split would then have to carry it twice.
If #1480 is rejected and the core tier stays as it is, this becomes ready
immediately.
quality-gates-scope-agreement carries "A configured scope is verified by
coverage, not exit status", which is the right rule and covers one half of the
problem. Every row of its table is a configuration parameter that
collapses:
| Gate |
Scope parameter |
Collapsed scope still exits zero |
| History secret scan |
checkout fetch depth |
scans the tip only |
| Linter |
path filter or glob |
lints zero files |
| ... |
... |
... |
The other half is a scope that is configured correctly and replaced at the
call site. The configuration is right, the invocation is right in isolation,
and the effective scope is neither.
The instance
A project declared its test scope in the manifest:
testpaths = ["tests", "src"]
The src entry existed for one reason: it is what collects the package's
docstring examples, added after an example was found asserting something its
class did not provide, green on the main branch.
The pipeline ran:
python -m pytest tests --cov=... --cov-fail-under=80
pytest honours testpaths only when given no path of its own. The tests
argument replaced the list rather than extending it, so src never applied.
Locally a bare run collected 627 items including 37 doctests; the pipeline
collected 590 and none of them.
So every docstring example ran on contributors' machines and none ran where a
merge was decided, for as long as the configuration existed.
Why the existing rules do not reach it
quality-gates-scope-agreement asks whether the scope parameter
collapsed. It did not -- the manifest named both directories throughout
quality-gates-check-runs asks the check to state what it inspected.
The run did report a collected count. Nobody compared it against the count a
local run produces, because there was no reason to think the two differed
- The project had a purpose-built test asserting the doctest gate could not go
blind. Its own docstring enumerated three failure modes, all of them
properties of the configuration, and the configuration was correct. It passed
throughout
That last point is the sharp part. A project can write the guard this rule
family asks for, keep it green, and still have the gate not run -- because the
guard and the rule are both aimed at the configuration, and the invocation is a
third thing neither reads.
Suggested addition
A short subsection under quality-gates-scope-agreement:
- Where a tool reads its scope from a manifest, the scope MUST be verified at
the invocation, not only in the configuration. Most such tools honour the
declared scope only when the caller supplies none, so a caller supplying one
replaces it silently and the two artifacts are each individually correct
- The check compares what the gating invocation actually collects, selects or
scans against what the declared scope produces. A count from the pipeline and
a count from a bare local run disagreeing is the finding
- Prefer removing the argument over duplicating the scope at the call site. A
scope named in two places drifts, and the second copy is the one nothing reads
The tell is a gate whose command names something the configuration also names.
Wherever a workflow repeats a path, a marker, a package list or a glob that a
manifest already declares, one of the two is not in force and nothing says
which.
Generic form
A declared scope and an invocation that overrides it are two correct artifacts
producing a wrong effective scope, and no rule aimed at either one alone can
see it.
Do not pick up before the trigger condition fires.
Trigger: #1480 reports. That spike classifies every core-tier section as
invariant policy or process layer, and the section this issue adds to is
classified as moving. Writing the rule first means writing it into a file whose
home is undecided, and the split would then have to carry it twice.
If #1480 is rejected and the core tier stays as it is, this becomes ready
immediately.
quality-gates-scope-agreementcarries "A configured scope is verified bycoverage, not exit status", which is the right rule and covers one half of the
problem. Every row of its table is a configuration parameter that
collapses:
The other half is a scope that is configured correctly and replaced at the
call site. The configuration is right, the invocation is right in isolation,
and the effective scope is neither.
The instance
A project declared its test scope in the manifest:
The
srcentry existed for one reason: it is what collects the package'sdocstring examples, added after an example was found asserting something its
class did not provide, green on the main branch.
The pipeline ran:
pytest honours
testpathsonly when given no path of its own. Thetestsargument replaced the list rather than extending it, so
srcnever applied.Locally a bare run collected 627 items including 37 doctests; the pipeline
collected 590 and none of them.
So every docstring example ran on contributors' machines and none ran where a
merge was decided, for as long as the configuration existed.
Why the existing rules do not reach it
quality-gates-scope-agreementasks whether the scope parametercollapsed. It did not -- the manifest named both directories throughout
quality-gates-check-runsasks the check to state what it inspected.The run did report a collected count. Nobody compared it against the count a
local run produces, because there was no reason to think the two differed
blind. Its own docstring enumerated three failure modes, all of them
properties of the configuration, and the configuration was correct. It passed
throughout
That last point is the sharp part. A project can write the guard this rule
family asks for, keep it green, and still have the gate not run -- because the
guard and the rule are both aimed at the configuration, and the invocation is a
third thing neither reads.
Suggested addition
A short subsection under
quality-gates-scope-agreement:the invocation, not only in the configuration. Most such tools honour the
declared scope only when the caller supplies none, so a caller supplying one
replaces it silently and the two artifacts are each individually correct
scans against what the declared scope produces. A count from the pipeline and
a count from a bare local run disagreeing is the finding
scope named in two places drifts, and the second copy is the one nothing reads
The tell is a gate whose command names something the configuration also names.
Wherever a workflow repeats a path, a marker, a package list or a glob that a
manifest already declares, one of the two is not in force and nothing says
which.
Generic form
A declared scope and an invocation that overrides it are two correct artifacts
producing a wrong effective scope, and no rule aimed at either one alone can
see it.