Skip to content

A correct configured scope can be overridden by the invocation, and the scope rule does not reach it #1497

Description

@braboj

Do not pick up before the trigger condition fires.

Trigger: #1480 reports. That spike classifies every core-tier section as
invariant policy or process layer, and the section this issue adds to is
classified as moving. Writing the rule first means writing it into a file whose
home is undecided, and the split would then have to carry it twice.

If #1480 is rejected and the core tier stays as it is, this becomes ready
immediately.


quality-gates-scope-agreement carries "A configured scope is verified by
coverage, not exit status", which is the right rule and covers one half of the
problem. Every row of its table is a configuration parameter that
collapses:

Gate Scope parameter Collapsed scope still exits zero
History secret scan checkout fetch depth scans the tip only
Linter path filter or glob lints zero files
... ... ...

The other half is a scope that is configured correctly and replaced at the
call site. The configuration is right, the invocation is right in isolation,
and the effective scope is neither.

The instance

A project declared its test scope in the manifest:

testpaths = ["tests", "src"]

The src entry existed for one reason: it is what collects the package's
docstring examples, added after an example was found asserting something its
class did not provide, green on the main branch.

The pipeline ran:

python -m pytest tests --cov=... --cov-fail-under=80

pytest honours testpaths only when given no path of its own. The tests
argument replaced the list rather than extending it, so src never applied.
Locally a bare run collected 627 items including 37 doctests; the pipeline
collected 590 and none of them.

So every docstring example ran on contributors' machines and none ran where a
merge was decided, for as long as the configuration existed.

Why the existing rules do not reach it

  • quality-gates-scope-agreement asks whether the scope parameter
    collapsed. It did not -- the manifest named both directories throughout
  • quality-gates-check-runs asks the check to state what it inspected.
    The run did report a collected count. Nobody compared it against the count a
    local run produces, because there was no reason to think the two differed
  • The project had a purpose-built test asserting the doctest gate could not go
    blind. Its own docstring enumerated three failure modes, all of them
    properties of the configuration, and the configuration was correct. It passed
    throughout

That last point is the sharp part. A project can write the guard this rule
family asks for, keep it green, and still have the gate not run -- because the
guard and the rule are both aimed at the configuration, and the invocation is a
third thing neither reads.

Suggested addition

A short subsection under quality-gates-scope-agreement:

  • Where a tool reads its scope from a manifest, the scope MUST be verified at
    the invocation, not only in the configuration. Most such tools honour the
    declared scope only when the caller supplies none, so a caller supplying one
    replaces it silently and the two artifacts are each individually correct
  • The check compares what the gating invocation actually collects, selects or
    scans against what the declared scope produces. A count from the pipeline and
    a count from a bare local run disagreeing is the finding
  • Prefer removing the argument over duplicating the scope at the call site. A
    scope named in two places drifts, and the second copy is the one nothing reads

The tell is a gate whose command names something the configuration also names.
Wherever a workflow repeats a path, a marker, a package list or a glob that a
manifest already declares, one of the two is not in force and nothing says
which.

Generic form

A declared scope and an invocation that overrides it are two correct artifacts
producing a wrong effective scope, and no rule aimed at either one alone can
see it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — important but not blockingtaskAtomic implementable work

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions