Skip to content

16 file-level @e2e tags in sbom-import.spec.ts claim coverage the file explicitly says it does not provide — real gate-19 debt is 153, not 137 #484

Description

@rubenvdlinde

Summary

tests/e2e/sbom-import.spec.ts opens with a docblock that says, verbatim:

The parse/replace/batch/matching CONTRACTS are pure server-side or
pure-function logic, verified by PHPUnit ... and vitest ... against real
CycloneDX fixtures — excluded from Playwright coverage below:

and then lists sixteen annotations in the POSITIVE form:

 * @e2e sbom-import::a-valid-cyclonedx-16-document-parses-into-components
 * @e2e sbom-import::an-unsupported-bomformat-or-specversion-is-rejected
 ... 14 more ...

The file contains exactly two real Playwright tests. gate-19 credits all
eighteen, so it reports sbom-import as 1 uncovered scenario instead of
17.

This is ConductionNL/.github#343 (a file-level @e2e tag credited without
checking the test body) landing in this repo. The author's intent was clearly
an exclusion — they wrote the word "excluded" — but reached for @e2e <ref>,
the token the gate rewards.

Why this matters beyond the number

softwarecatalog's gate-19 baseline is 137. The honest figure is 153.
Closing 137 does not mean the app has e2e coverage of those sixteen scenarios;
it means sixteen of them were never counted.

What NOT to do

Deleting the sixteen tags raises the gate number by 16 and buys nothing. The
right resolution is per-scenario judgement:

  • Several are genuinely browser-observable and deserve a real test — e.g.
    an unsupported bomFormat is rejected (HTTP 422 + the sbom-upload-error
    testid), import requires admin or manage-ACL (403), a component with
    VEX-declared CVE data gets a confirmed match
    (sbom-match-confirmed),
    a successful import records provenance (sbom-provenance).
  • Several genuinely cannot be observed from a browser and should become
    reason-bearing @e2e exclude markers in the spec file, which is the
    mechanism the gate provides for exactly this — e.g. a prior replace's
    trashed rows are not reprocessed
    , no outbound HTTP call is made during
    matching
    , existing versions are unaffected by the schema addition.

Also found in the same file

sbom-import::a-second-import-replaces-the-first has neither an @e2e tag
nor an @e2e exclude — yet the behaviour is already exercised by the real
test at tests/e2e/sbom-import.spec.ts:144 (second setInputFiles, expects
only 2 rows). That one is a genuine one-line fix: add the annotation above the
existing test. It is the single uncovered sbom-import scenario gate-19
currently reports.

Deliberately not fixed in the gate-19 PR

Converting sixteen positive coverage claims into sixteen exclusions inside a PR
whose stated purpose is lowering the gate-19 count is indistinguishable from
gaming the count, however well-reasoned each one is. It needs its own PR and
its own review.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions