test(e2e): gate-19 186 → 161 — 25 verified anchors, 1 refusal, 2 false anchors removed - #179
Merged
Merged
Conversation
… 2 that did not gate-19 (@e2e traceability), FULL scope, measured with hydra-gates @ main (b8c7ead) against openbuild@development: before {scenarios: 767, covered: 72, excluded: 509, uncovered: 186} after {scenarios: 767, covered: 97, excluded: 509, uncovered: 161} 186 -> 161. No test was skipped, no timeout widened, no exclusion added. WHAT THE 25 ARE. Not new tests: `builder-undo-redo.spec.ts` and `form-editor-logic.spec.ts` already drive these scenarios and already run green; they carried no `@e2e` reference, so the gate could not see them. Every mapping was re-derived by reading the test BODY against the scenario text, not by matching titles. builder-undo-redo 11 anchors across 6 tests form-editor-logic 14 anchors across 5 tests One of the 14 needed the test EXTENDED before it could be anchored honestly: `deleting-a-step-returns-its-fields-to-the-unassigned-pool` asserted the removal and the pool re-appearance but not the spec's final AND (save auto-assigns an unassigned field to the last step). The surface exists (`assignUnassignedFieldsToFinalStep()`, called from PageDesignerHost's save()), so the test now re-adds a step, asserts the pool note, saves, and polls the PERSISTED manifest for the assignment. ONE SCENARIO WAS REFUSED, deliberately, with no exclusion written: `form-editor-logic::raw-json-authored-logic-survives-unrelated-editor-edits` asserts a Design <-> Raw JSON tab round-trip. That tab does not exist on /builder/:slug/pages — the raw editor is a sidebar tab on a different page, writing a different object. Anchoring it would have been false and excluding it would have been a false statement about observability, so it stays uncovered and an issue is the right next step. AND TWO ANCHORS WERE REMOVED, which RAISES the covered count's honesty at the cost of the number. `save-as-template.spec.ts` credited `viewer-cannot-save-a-template` and `seeded-cards-remain-read-only` to a test whose own comment, three lines below the tags, states that it proves neither: the assertion is toHaveCount(0) on a button, under an ADMIN session, on a page where the button is not scoped to appear. Filed as #178. Removing them did NOT raise `uncovered` (161 either way) — because both scenarios are ALSO swallowed by a requirement-level `@e2e exclude` whose stated reason is about something else entirely. Measuring that led to the wider finding: 471 of openbuild's 509 exclusions (92.5%) share a reason with a sibling, 42 reasons covering them, one covering 32 scenarios. Filed upstream as ConductionNL/.github#356. NEGATIVE CONTROL (scope-independent): 161 with tests/e2e/builder-undo-redo.spec.ts present 172 with it removed (+11 = exactly its 11 anchors) 161 restored Positive control that the gate validates slugs rather than the tag's presence: corrupting one slug to #adding-a-step-groups-fields-by-BOGUS dropped covered 99 -> 98; restoring returned it to 99. Refs #178
…dy proves 161 -> 160. `component-blocks.spec.ts` anchored `component-blocks::blocks-filter-shows-blocks-without-the-clone-action`. That scenario lives in `openbuild-template-catalogue`, not `component-blocks`, so the ref resolved to NOTHING: gate-19 credited no scenario and reported no error, while the real scenario sat in the uncovered list and a green test that proves it was sitting right there. A dangling anchor is silent in both directions — it looks like coverage in the file and like an absence in the gate. Found by walking every anchor in the suite against the gate's own parser rather than by eye. openbuild has 15 of them. This is the only one that was a simple mis-spelled target. The other 14 are in `spec-coverage/page-designer-ui.spec.ts`, name slugs that exist in no spec, sit on seven tests CI never runs (`test.skip(!LIVE)`, `OPENBUILD_E2E_LIVE` is never set), and belong to ten scenarios that are ALL `@e2e exclude`d on the grounds that another spec's Playwright tests cover them. Repointing those while the tests stay skipped would move them from silently uncounted to counted against tests that never execute, so they are filed rather than patched — #181. What this anchor does NOT claim is recorded next to it: the scenario's THEN lists "name, description, category and a preview"; the test asserts the name and the AND (no clone action). The distinguishing behaviour — browse-only, no clone affordance — is covered. Refs #181
Contributor
Quality Report — ConductionNL/openbuild @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| test-l10n | ✅ | ||||
| check-gitignore | ✅ | ||||
| check-nc-floor | ✅ | ||||
| composer | ✅ | ✅ 101/101 | |||
| npm | ✅ | ✅ 654/654 | |||
| PHPUnit | ✅ | ||||
| Newman | ⏭️ | ||||
| Playwright | ❌ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-11 13:48 UTC
Download the full PDF report from the workflow artifacts.
The new REQ-OBFEL-001 poll carried `timeout: 30_000`, which is exactly the per-test budget in tests/e2e/playwright.config.ts. The test would always die first, so the poll's own message — "saving must append the still-unassigned field keys to the final step" — could never be printed, and the failure would surface as a bare test timeout instead. That is the precise failure mode the config's shorter `expect` timeout exists to prevent; its own comment says so. Use the repo default, 15_000. Not a widening: this LOWERS a number, and it makes the assertion able to report itself.
Contributor
Quality Report — ConductionNL/openbuild @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ❌ | ||||
| check-manifest | ✅ | ||||
| test-l10n | ✅ | ||||
| check-gitignore | ✅ | ||||
| check-nc-floor | ✅ | ||||
| composer | ✅ | ✅ 101/101 | |||
| npm | ✅ | ✅ 654/654 | |||
| PHPUnit | ❌ | ||||
| Newman | ⏭️ | ||||
| Playwright | ❌ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-11 13:51 UTC
Download the full PDF report from the workflow artifacts.
Contributor
Quality Report — ConductionNL/openbuild @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| test-l10n | ✅ | ||||
| check-gitignore | ✅ | ||||
| check-nc-floor | ✅ | ||||
| composer | ✅ | ✅ 101/101 | |||
| npm | ✅ | ✅ 654/654 | |||
| PHPUnit | ✅ | ||||
| Newman | ⏭️ | ||||
| Playwright | ❌ | ||||
| Hydra gates | ✅ |
Quality workflow — 2026-08-11 14:26 UTC
Download the full PDF report from the workflow artifacts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gate-19: 186 → 161
Measured full scope (
hydra-gates --full) with hydra-gates @main(b8c7ead, 1735-line checker — the ref CI pins), againstopenbuild@development:The 186 baseline reproduces run 31459739568's figure exactly.
Negative control
tests/e2e/builder-undo-redo.spec.tsremovedPositive control
Corrupting one slug to
#adding-a-step-groups-fields-by-BOGUSdroppedcovered99 → 98; restoring returned it to 99. The gate validates slugs, so the other 24 are real matches — not tags it accepted without looking.What the 25 anchors are
Not new tests.
builder-undo-redo.spec.tsandform-editor-logic.spec.tsalready drive these scenarios and already run green; they carried no@e2ereference, so the gate could not see them. Every mapping was re-derived by reading the test body against the scenario text — a title that resembles a scenario is not coverage of it.builder-undo-redo— 11 anchors across 6 testsform-editor-logic— 14 anchors across 5 testsOne of the 14 needed the test extended before it could be anchored honestly:
deleting-a-step-returns-its-fields-to-the-unassigned-poolasserted the removal and the pool re-appearance but not the spec's final AND (save auto-assigns an unassigned field to the last step, with a warning mark). The surface exists —assignUnassignedFieldsToFinalStep()insrc/services/manifestValidation/formLogic.js, called fromPageDesignerHost.vue'ssave()— so the test now re-adds a step, asserts the pool note, saves, and polls the persisted manifest for the assignment.One scenario refused, with no exclusion written
form-editor-logic::raw-json-authored-logic-survives-unrelated-editor-editsasserts a Design ↔ Raw JSON tab round-trip. That tab does not exist on/builder/:slug/pages; the raw editor is a sidebar tab on the VirtualAppDetail page, writing the Application object while the designer saves onto the ApplicationVersion. Different surface, different storage target.Anchoring it would have been false. Excluding it would have claimed "a browser cannot observe this", which is not the reason — the reason is that the spec describes a surface the product does not have. So it stays uncovered, with an in-file comment explaining why, so nobody later "fixes" it with a false anchor.
Two anchors REMOVED — this costs the number and is the right call
save-as-template.spec.tscreditedviewer-cannot-save-a-templateandseeded-cards-remain-read-onlyto a test whose own comment, three lines below the tags, says it proves neither: the assertion istoHaveCount(0)on a button, under an admin session, on a page where the button is not scoped to appear. Filed as #178.Removing them did not raise
uncovered— 161 either way. Both scenarios are also swallowed by a requirement-level@e2e excludewhose stated reason is about something else (REQ-SAT-001's reason is about "the captured record's internals"; the scenario it exempts is about whether a viewer sees a button). The false anchor was a second green layer over a scenario that was already invisibly exempt.Chasing that produced a fleet-wide finding, filed as ConductionNL/.github#356: 471 of openbuild's 509 exclusions (92.5%) share a reason with a sibling — 42 reasons, one covering 32 scenarios, several beginning "mixed spec — …". Also worth knowing when reading any gate-19 percentage:
coverage_pctiscovered / (scenarios − excluded), so openbuild's headline 27.9% is 9.4% (72/767) unqualified.Not done here
No test was skipped, no timeout widened, no
mode: 'serial', nowaitForLoadState('networkidle'), and no exclusion added.ESLint on all three changed files reports only the pre-existing
n/no-unpublished-importon the@playwright/testimport, which every test file in the repo carries.The red
E2E Tests (Playwright)job is a separate, pre-existing failure — diagnosed and fixed in #177, which is open for review.