ci: turn Newman on — it has never run, and its seed could not fail

[rubenvdlinde]

What this measures

enable-newman: false makes the Integration Tests (Newman) job report skipped, and a skipped job is indistinguishable from a passing one in the run summary. Eighteen Postman collections sit in tests/integration/ and CI has never executed one of them. This PR turns the job on so it produces a verdict.

The seed could not fail, so "it would just be red anyway" was never measured

The shared workflow interpolated newman-seed-command into an unquoted eval with no status check. This repository is the one caller in the fleet that passes an && chain:

php occ app:disable openbuild && php occ app:enable openbuild

The outer shell split that before eval ever saw it, eval ran only the disable, and bash's set -e exemption for && lists returned 0 regardless. A failed disable meant the enable never ran and Newman would have tested whatever state was left — green.

Fixed upstream in ConductionNL/.github#212: the seed now reaches eval as one quoted unit and its combined status is the step's, with an ::error:: and a hard exit when it fails. So a red Newman leg here is now a genuine finding, and so is a green one.

The two blockers the flag carried, re-measured 2026-08-09

#	Blocker	Status
1	SeedHelloWorld failed: Call to undefined function React\Async\await() — OR's runtime-schema-API pulls react/async but it wasn't surfacing in the CI install	react/async ^4.3 is a require in openregister@development's composer.json and is present in its committed composer.lock. The CI install resolves it.
2	[PermissionHandler] no such function: REGEXP from OR's MagicMapper under SQLite	Dead. database: is now pgsql, and that switch was made for exactly this family of failure. Newman reads the same database input.

Neither was re-measured after database: pgsql landed. This change makes CI answer the question instead of the comment answering it.

Honest expectation

I am not claiming this will be green. I am claiming the number it produces will be real for the first time. If it is red, the failures are a finding to act on, not a regression this PR introduced — the leg reported skipped before, which was never evidence of anything.

Refs openbuild#151

[rubenvdlinde]

The honest Newman number, measured

Run 31316691991, job Integration Tests (Newman). This is the first verdict this leg has ever produced in this repository.

Both documented blockers are dead

success  Install and enable app
success  Start PHP built-in server
success  Seed test data          <-- the step that was supposed to be impossible
success  Validate Newman collections
failure  Run Newman tests

Seed test data passed. No React\Async\await(), no no such function: REGEXP. The two blockers written into enable-newman: false were both retired by changes made elsewhere (react/async landing in openregister's lockfile, and database: pgsql) and nobody re-measured. The comment outlived the condition it described.

And it passed through the fixed seed path — .github#212 — so this is a seed that could have failed and did not, not a seed that cannot fail.

The collections: 80 failed assertions of 242, 14 of 18 collections red

failed	executed	collection
15	17	openbuild-versioning
13	18	openbuild-application-versions
13	32	openbuild-rbac
10	37	openbuild
5	12	openbuild-component-blocks
5	8	openbuild-page-editor
4	38	openbuild-api-contract
3	3	openbuild-automation-approval-steps
3	7	openbuild-export-to-real-app
3	3	openbuild-version-promotion
2	8	openbuild-docudesk-documents
2	9	openbuild-templates-marketplace
1	14	openbuild-app-creation-wizard
1	7	openbuild-icon-endpoints
0	9	openbuild-app-detail-overview
0	3	openbuild-nldesign-theme
0	9	openbuild-schema-editor
0	8	openbuild-version-routing
80	242	18 collections — 4 fully green

Zero iterations failed anywhere — every collection ran to completion. These are assertion failures against a live server, not infrastructure failures.

Causes visible in the server log

Three distinct classes, none of them "Newman is broken":

1. A real app defect. openbuild-export-to-real-app posts applicationVersion: "production"; the schema requires semver: "Property 'applicationVersion' should match pattern ^\d+\.\d+\.\d+(?:[-+][\w.-]+)?$ but 'production' does not." Either the collection is wrong about the contract or the contract is wrong about channels. That is exactly the kind of question this leg exists to ask.
2. A missing fixture. openbuild-docudesk-documents → RegisterNotConfiguredException: Template register/schema not configured. Docudesk is installed by additional-apps but its template register is only configured by the Playwright global-setup.ts, which the Newman job does not run. Environmental, and fixable in the seed.
3. A missing test user. openbuild-rbac and openbuild-nldesign-theme log Login failed: 'viewer'. The RBAC collections assume a viewer account the seed never creates — which alone accounts for a large share of the 13 RBAC failures.

What I am asking for

This PR turns the measurement on; it does not claim the measurement is green. The 80 are now visible, attributable and countable instead of hidden behind a skipped badge. Before this PR the number was unknown and unknowable, and the leg reported skipped — which reads identically to a pass in the run summary.

Refs openbuild#151

[github-actions]

Quality Report — ConductionNL/openbuild @ 8951fa3

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					✅

---

Quality workflow — 2026-08-09 14:27 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ dc42a3f

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:02 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ b1ffb65

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:10 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ 96fb104

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:16 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ 9aca0c8

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:22 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ 44dedb1

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:28 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ 90ececd

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:35 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ cd824e6

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:41 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ e5941f1

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 04:48 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ e4d97bc

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					✅

---

Quality workflow — 2026-08-10 05:29 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ 7a4130f

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					❌
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 14:39 UTC

Download the full PDF report from the workflow artifacts.

[rubenvdlinde]

Newman burn-down: 45 → 30 measured, three more batches pushed

Measured, not projected

Job 93487483023 (run 31398481286) reached completed/failure with a full per-collection tally before that run was cancelled by the next push. A cancelled run is no verdict, but a job that completed with its own conclusion and printed its summary tables is a real measurement:

collection	before	after
openbuild-api-contract	4	0
openbuild-app-creation-wizard	1	0
openbuild-application-versions	3	0
openbuild-icon-endpoints	1	0
openbuild-version-promotion	1	0
openbuild-versioning	11	6
(untouched in that batch)	24	24
total	45	30

1. Five "unauthenticated" probes were logged-in requests — 6 assertions, all now green

auth: { type: noauth } removes the Authorization header. It does not remove the session cookie Newman already holds for that origin. Measured with newman --verbose, one origin apart, same request:

auth,   {{base_url}}    -> 324B up, 9 headers  (Authorization, no Cookie)
noauth, {{base_url}}    -> 576B up, 9 headers  (no Authorization, COOKIE)
noauth, {{noAuthBase}}  -> 285B up, 8 headers  (neither — truly anonymous)

The header count is the tell: 9 vs 8.

In CI the consequence was visible in run order: the first anonymous probe returned 401, and every one after the first authenticated write returned 412 CSRF. Nextcloud's 401 fires before the CSRF check, so a 412 is proof the probe was authenticated. openbuild-icon-endpoints was the same defect wearing the opposite face — its probe returned 200 and was on its way to being read as an authorization hole. It was the admin, correctly served an icon.

Fixed the way this repo already solved it once: openbuild-api-contract reaches the same server by its other name (noAuthBase = http://127.0.0.1:8080), so the origin-keyed cookie jar has nothing for it — and all five of its anonymous assertions passed in the very run these failed.

Rejected: pm.cookies.jar().clear(). It reports success and does nothing — getAll() returns an empty list for these origins, so the clear "succeeds" over a view that never had the cookies, while --verbose shows the upload size unchanged at 576B with the Cookie still attached. A control that passes because it cannot see its subject.

2. PRODUCT — export submit 500s on a malformed applicationVersion (4 → 0)

exportJob.applicationVersion is declared a semver and ExportDialog.vue sends one; the controller checked only non-empty, so anything else died three layers down where OR refused the record and queue() rethrew → HTTP 500, with nothing telling the caller which field was wrong. target one block up was already validated exactly this way. Now 422. All four failures were one request, one cause. The collection was also wrong (it sent the slug production) and now sends a semver, with a new request keeping a bad one on purpose so the 4xx stays proven.

3. PRODUCT — diffVersions turned a 404 into a 500 (measured 500 → 404)

resolveVersionBlob() says "returns null on miss so the caller can surface 404"; ObjectService::find() throws. Both 404 branches were dead code and the throw became internal_error. Eighth instance of the family #159 fixed — missed there because gate-49 only flags an untranslated lookup outside a try/catch, and being inside one is exactly what converted a precise 404 into an opaque 500 on a #[NoAdminRequired] endpoint.

4. openbuild-versioning — a filter that cannot match, and a PATCH that isn't (11 → 6)

applicationVersion?_limit=5000        -> 119 rows
applicationVersion?semver=1.0.0       ->  20 rows  (scalar prop: filters)
applicationVersion?application=<uuid> ->   0 rows  (relation: does NOT)

The third used a uuid taken straight out of row 1 of the first — the value was demonstrably present. OR does not apply a query-string filter to an x-openregister-relation. An empty result from a filter that cannot match looks exactly like "the seed never ran" — the misdiagnosis this suite's own comments say it already made once.

The three PUTs sent {version, status, manifest} only; OR's saveObject() nulls omitted properties and Application.required = ['slug','name'], so OR refused with 400 and said nothing about slug or name. Now merged from the full record — and 1.1 additionally asserts slug/name survived, so trimming the merge back can't pass quietly.

5. The last 6 in that collection asserted a model ADR-002 retired

openspec/specs/openbuild-version-snapshots/spec.md (status: done) — REQ-OBV-002: "The system SHALL NOT spawn sibling ApplicationVersion rows on draft → published transitions", "SHALL NOT subscribe any PHP listener (ApplicationVersionSnapshotListener or any successor)", scenario "Publishing does NOT create a sibling ApplicationVersion". And the listener does not exist — lib/Listener/ holds six, none of them it.

The collection demanded "the listener must have appended a v1.1.0 row", "at least three rows after rollback" and "a rollback row like 1.0.0-rb1". The only way any of those goes green is for the product to regress to an architecture the spec forbids. Inverted to assert REQ-OBV-002/003 as written, non-vacuously: the row count is captured before the publish and asserted unchanged, and the before-count is separately required ≥ 1 — otherwise a broken list read leaves both sides at zero and "unchanged" passes while measuring nothing.

6. openbuild-component-blocks (5) and openbuild (4)

body.uuid is undefined — OR puts the identity in @self. Postman interpolates an undefined variable as the literal string null, so three requests went to /component-block/null and 404'd. ⚠️ And one assertion was green over the same two broken values: "imported record gets a NEW uuid" compared undefined with 'null', found them unequal, and passed.

openbuild: strategy=hard is not a strategy (delete-now | orphan-grace | keep-register); and the round-trip never registered a BuiltAppRoute, which is how resolveApplicationBySlug() resolves a slug at all — so the manifest 404'd after every preceding step had succeeded, and the cause was in none of the three failing assertions.

Left red on purpose

The three REQ-OBV-005 diff assertions → #170. The 500 is fixed; what remains is a spec-vs-controller divergence (spec: version slug / current: / history:<slug>:<rev>; controller: draft or a UUID). Rewriting them to match today's behaviour would lock in the divergence and delete the only evidence of it.

Still unfixed and not yet diagnosed to a conclusion: openbuild-rbac (6), openbuild-automation-approval-steps (3), openbuild-docudesk-documents (2), openbuild-page-editor (2), openbuild-templates-marketplace (2). Working notes in the review thread — notably, rbac's three permission PUTs are the same PUT-semantics 400 fixed above, and its three "outsider" assertions returned byte-identical responses for viewer/editor/outsider, which is the cookie-carryover shape again and would mean that collection has never actually tested RBAC.

[rubenvdlinde]

Working notes on the collections I did not finish

Diagnoses from the run-31356594486 log, so the next person does not re-derive them. None of these is asserted anywhere yet — they are read off the request/response lines, and each needs its own measurement before being acted on.

openbuild-automation-approval-steps (3)

The collection still contains a literal unsubstituted placeholder:

GET /apps/openbuild/api/automations/REPLACE-WITH-SEEDED-APPROVAL-AUTOMATION-UUID/status  [200 OK, 10.23kB]

⚠️ That 200 is not a pass — it is 10 kB of Nextcloud's HTML page. The route requires [a-f0-9-]{8,}, the placeholder is uppercase, so the route never matched and NC fell through to the SPA. A 200 that is HTML is the most dangerous shape a collection can produce, because a status-only assertion accepts it.

The other two: GET /apps/openregister/api/approval-steps?status=pending → 412 (the same session-cookie-then-CSRF mechanism fixed for the anonymous probes, here on an authenticated request), and POST /apps/openbuild/api/approval-steps/1/approve → 405 where the test asserts 404 ("OpenBuild has no approve pass-through route"). 405 vs 404 is a different claim — the path matched a route with another verb — so the assertion needs rewording against what appinfo/routes.php actually registers.

openbuild-page-editor (2)

PUT the Application's manifest → 200, then GET /api/applications/hello-world/manifest → the newman-added page is not there. That looks correct, not broken: under the versioned model the manifest endpoint resolves slug → built-app-route → productionVersion → version.manifest. Editing Application.manifest no longer feeds it. The collection is writing to the wrong object and should target the ApplicationVersion.

"Invalid manifest is rejected with 4xx" → got 200 for the same reason: a raw OR object write is validated against the JSON schema, not against openbuild's manifest validator. The validator lives on openbuild's own version-update path, so this assertion is aimed at the wrong surface — it is currently testing OpenRegister.

openbuild-templates-marketplace (2)

Second POST /api/applications/from-template/permit-tracker → 409 slug_collision where the test expects 201 "for different owner". Genuinely ambiguous and needs a product decision, not a test edit: either clone-from-template should mint a per-clone slug, or slug uniqueness is global and the assertion's premise is wrong. Do not "fix" this by relaxing the assertion to accept 409 — that would encode an answer nobody has given.

openbuild-docudesk-documents (2)

generate with an unknown templateId is a 4xx, not a 500 → 500, and the Nextcloud log names it:

Correspondence generation failed: Template register/schema not configured
OCA\DocuDesk\Exception\RegisterNotConfiguredException

Two separable things: the CI instance never configures Docudesk's template_register/template_schema for the Newman job (tests/e2e/global-setup.ts does it for Playwright only), and Docudesk answers an unconfigured register with a 500 rather than a 4xx — which is a docudesk defect, not an openbuild one. The honest fix is to configure it in ci-seed.sh and, if the 500 survives, file it upstream.

openbuild-rbac — the half I did not fix

The three permission PUTs are fixed in 1faecd91f (bare body + unprefixed principals). The three remaining "outsider" assertions are suspected, not diagnosed: viewer, editor and outsider received byte-identical responses (1.94 kB ×3 on the list, 1.1 kB ×3 on the manifest), which is the cookie-carryover shape — a session established by an earlier authenticated request outliving the per-request basic auth, so all three requests were the same user.

If that holds, this collection has never actually tested RBAC, and the fix is harder than for the anonymous probes: there are four identities and only two names for one host, so the noAuthBase trick does not generalise. pm.cookies.jar().clear() is already proven inert here. Needs its own measurement first — e.g. asserting the response body's identity rather than trusting the request's auth block.

[github-actions]

Quality Report — ConductionNL/openbuild @ a469aa9

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					❌
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 14:47 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ e9c03a4

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					❌
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 14:56 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ 259e7af

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					❌
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-10 15:01 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/openbuild @ f7d1c10

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					✅

---

Quality workflow — 2026-08-10 15:31 UTC

Download the full PDF report from the workflow artifacts.

[rubenvdlinde]

Final measurement on this branch: 45 → 15

Run 31401193088 (SHA 6d9fcee02), Newman job 93496964640. Executed 258 assertions, 15 failing.

collection	baseline	now
openbuild	4	0
openbuild-api-contract	4	0
openbuild-app-creation-wizard	1	0
openbuild-application-versions	3	0
openbuild-component-blocks	5	0
openbuild-icon-endpoints	1	0
openbuild-version-promotion	1	0
openbuild-versioning	11	3
openbuild-rbac	6	3
openbuild-automation-approval-steps	3	3
openbuild-docudesk-documents	2	2
openbuild-page-editor	2	2
openbuild-templates-marketplace	2	2
total	45	15

Seven collections fully green. Note the assertion count went 251 → 258: the fixes added regression guards rather than removing checks (a deliberate non-semver 422 probe, a permissions read-back, a production-delete guard, a slug/name-survived check).

PHPUnit (PHP 8.3, NC stable32) is green again

It was never a failing test — OK (797 tests, 2486 assertions) followed by FAIL: coverage dropped by 0.05% against the merge base. That drop was mine: both product fixes added branches with nothing behind them. Four tests (797 → 801) cleared it. The ratchet was right and did its job.

The 15 that remain, and why each is still red

• versioning (3) → diff endpoint implements a ref grammar the spec does not define (REQ-OBV-005: slug / current: / history:, controller: draft | UUID) #170. The 500 is fixed; what is left is a spec-vs-controller divergence (REQ-OBV-005 defines the ref grammar as a version slug / current: / history:<slug>:<rev>; resolveVersionBlob() accepts draft or a UUID). Rewriting the assertions to today's behaviour would go green and delete the only evidence of it.
• rbac (3) — suspected cookie carryover: viewer, editor and outsider received byte-identical responses (1.94 kB ×3 on the list, 1.1 kB ×3 on the manifest). If that holds, this collection has never actually tested RBAC. Not fixed because noAuthBase does not generalise to four identities on one host and jar.clear() is proven inert — it needs a different instrument (assert the identity in the response body, not the request's auth block).
• automation-approval-steps (3) — a literal unsubstituted REPLACE-WITH-SEEDED-APPROVAL-AUTOMATION-UUID, whose 200 is 10 kB of Nextcloud HTML, not JSON.
• page-editor (2) — writes Application.manifest, but the manifest endpoint resolves through built-app-route → productionVersion → version.manifest. Wrong object, and the invalid-manifest assertion is aimed at OpenRegister rather than openbuild's validator.
• templates-marketplace (2) — a second clone-from-template 409s. Whether that should mint a per-clone slug is a product decision; relaxing the assertion to accept 409 would encode an answer nobody has given.
• docudesk-documents (2) — Docudesk's template register/schema is never configured for the Newman job (global-setup.ts does it for Playwright only), and Docudesk answers an unconfigured register with a 500 rather than a 4xx — a docudesk defect, not an openbuild one.

Nothing here is waived, skipped, baselined or threshold-shifted.

⚠️ E2E Tests (Playwright) on this branch is red for a reason that is not this PR: the four failing specs are fixed in #169, which targets development separately. This branch predates them.

[rubenvdlinde]

The rbac trio is a harness defect, not three IDOR holes — and two of that collection's PASSES are false greens

Measured on this PR's run 31401193088 (Newman job 93496964640): 258 assertions executed, 15 failed across 18 collections. openbuild-rbac accounts for 3 of the 15:

1. AssertionError  Outsider sees the seeded app in NO list — IDOR closed
                   outsider must NOT see the hello-world app:
                   expected { slug: 'hello-world', …(17) } to be undefined
2. AssertionError  Outsider on existing slug gets 403 (not 404)
                   expected response to have status code 403 but got 200
3. AssertionError  Error envelope carries the fixed code openbuild.rbac.no_role
                   expected { version: '1.0.0', …(5) } to have property 'error'

Read on its own, that says the list endpoint leaks and the manifest endpoint is open to a non-member. It does not.

Cause

Newman keeps ONE cookie jar per host, and Nextcloud resolves the session cookie BEFORE it reads Authorization: Basic. A request carrying both runs as whoever authenticated last.

In tests/integration/openbuild-rbac.postman_collection.json:

• collection-level auth is Basic as {{admin_user}};
• the Setup folder (1.1–1.10) runs ten requests as admin, which seeds an admin session cookie into that single jar;
• requests 2.1 / 2.2 / 2.3 / 3.1 then override auth to viewer / editor / outsider — but still carry the admin cookie, so every one of them executes as admin.

The collection contains no cookie-jar clear at all — pm.cookies appears 0 times in the whole file.

That is the entire explanation for all three failures: the "outsider" is admin, so it sees the app and gets 200.

The part that matters more than the three reds

2.1 "Viewer GET /api/applications sees the test app" and 2.2 "Editor … sees the test app" are passing for the same reason — admin sees everything. Those two assertions have never demonstrated that the viewer/editor grants work. This collection has not tested RBAC in either direction; it has run 7 requests as admin and scored them as four different principals.

So the honest count for this collection is not "3 failures"; it is "3 failures and 2 unverified passes".

Fix, with the control that makes it real

1. Clear the jar (collection-level pre-request, pm.cookies.jar() → jar.clear(...)) so a role request cannot inherit the previous request's session.
2. Add an identity guard in the same chain: GET /ocs/v2.php/cloud/user (with OCS-APIRequest: true) as each role, asserting ocs.data.id is that role's uid. Without this, a future regression of the jar is silent again.
3. Prove both directions: remove the clear, show the identity guard goes RED (reports admin); restore it, show GREEN. A guard that cannot fail is worth nothing.

⚠️ curl cannot reproduce this — it uses a fresh jar per invocation, so a curl-based "these endpoints are fine" check proves nothing about the Newman result either way. The measurement has to happen inside the same Newman chain.

Expect 2.1/2.2 to be capable of going red once the jar is honest. If they do, that is a real grant bug to chase (cf. #76, which fixed exactly this class for reads via read: ["authenticated"]), not something to paper over.

On the framing of this PR

Agreed, and worth restating for whoever picks this up: enable-newman: true has never existed in .github/workflows/code-quality.yml in this repo's history, so on development the job still reports skipped and the 15/258 figure exists only on this branch. Until this merges, "openbuild's Newman has 15 failures" is a statement about chore/enable-newman-0809, not about development, where Newman has produced no verdict at all.

[rubenvdlinde]

Retraction: the "single cookie jar makes the outsider run as admin" premise does not reproduce

I was about to fix openbuild-rbac.postman_collection.json on the theory that its Setup folder (ten admin requests) seeds one per-host cookie jar, so requests 2.1 / 2.2 / 2.3 / 3.1 — which override auth to viewer/editor/outsider — still carry the admin session and therefore all run as admin, making both the passing and the failing assertions meaningless. pm.cookies does appear 0 times in the file, so the shape is right.

I measured it before writing the fix, and the mechanism is not there. Nextcloud 34.0.0.12, Newman 6.2.2.

Arm 1 — Newman, no jar clear, request-level Basic override (exactly the collection's shape)

→ 1 admin identity (seeds the jar)
  GET /ocs/v2.php/cloud/user [200 OK]
  ✓  runs as admin
→ 2 probe identity (Basic overridden to the probe user)
  GET /ocs/v2.php/cloud/user [200 OK]
  │ 'OBSERVED IDENTITY: nmjarprobe'
  ✓  the request that says probe_user actually RUNS as probe_user

Arm 2 — curl control, decisive

Seeded a real jar from an admin Basic request to /index.php/apps/openbuild/api/applications. It genuinely contains session state:

oc_sessionPassphrase   BvDoyENTNf8Kis5j918kfpTGyrjG...
ocy2k459bs75           446c96972d226cc408de774ec8279a69
nc_sameSiteCookiestrict / nc_sameSiteCookielax

Then, reusing that jar:

request	result
admin jar + probe Basic header	identity = nmjarprobe
admin jar + no Basic header	997 "Current user is not logged in"

The second row is the one that settles it: those admin cookies authenticate nobody. NC does not persist a reusable login session for Basic-auth API requests, so there is no admin identity for a later request to inherit — whether or not the client sends the cookies. The Basic header is the only thing deciding identity.

So the collection's auth-override design is sound as written, and I have not pushed a jar-clear fix. Writing one, with a comment asserting a defect I could not reproduce, would have been inventing the finding.

Scope of the retraction — please read before relying on it

• Measured on NC 34. This job's PHPUnit legs run stable32 / stable33; I did not measure those. Same-mechanism-different-version is exactly the kind of thing that flips.
• Measured against /ocs/v2.php/cloud/user and /index.php/apps/openbuild/api/applications. Not every route in the collection.

What is worth doing instead

An identity guard — a GET /ocs/v2.php/cloud/user asserting ocs.data.id before each role-scoped block — remains worth adding, not as a fix but as an invariant: it converts "these requests run as the user they claim" from an assumption into something the run measures, on whatever NC version CI happens to boot. I ran out of budget before landing it; it is a small, self-contained follow-up and I did not want to push it under a false rationale.

Blocker status for enable-newman: true

The enable-newman: false comment names two blockers.

• Blocker 2 (SQLite: no such function: REGEXP in OR's MagicMapper) is already dead. development sets database: pgsql. The comment above the flag was not updated when that landed and still reads as if both are live.
• Blocker 1 (React\Async\await() not autoloading) is unverified but looks like an install problem, not a missing dependency. OpenRegister's composer.json declares "react/async": "^4.3", vendor/react/async is present in a working install, and HyperFacetHandler.php:500 already carries a comment about the undefined-function case. Whatever fails in the CI install is not "the package was never required".

This PR is MERGEABLE but BLOCKED, and Newman has never once run in this repository — enable-newman: true has never appeared in this workflow's history — so the first run is the first real verdict on both. I have not flipped or merged anything here.

[rubenvdlinde]

Re-ran this against current development — the failures reproduce, but the two RBAC ones are probably measuring admin

Updated the branch onto development (now a34d92cce, carrying #176, #185, #184
and 45f445e07) and re-ran, so these numbers are current rather than three days
old. Newman run 31572961038.

The failure set is byte-identical to the 2026-08-10 run. Nothing that landed in
between moved any of it, so none of these are stale.

The headline: this PR is doing exactly what it claims

Newman genuinely had never run, and turning it on surfaces 15 failing assertions
across 6 collections. That is the honest outcome and the PR should not be judged
by it being red. But the red is not yet mergeable, because turning this on lands a
permanently-failing cell on development.

⚠️ Two of the failures look like a live IDOR and I do not believe they are

REQ-OBRBAC-003 · 2.3 Outsider GET /api/applications returns empty list
  outsider must NOT see the hello-world app: expected { slug: 'hello-world', …(17) } to be undefined

REQ-OBRBAC-002 · 3.1 Outsider on existing slug — 403 BEFORE 404
  expected response to have status code 403 but got 200

Read alone, that says any authenticated user can list every Application and read
any manifest. I read every guard in both paths on this exact tree and all of them
are deny-by-default:

link	verdict
PermissionResolver::matchesCaller() :107	if ($permissions === []) return false; — fails closed on an empty block
ApplicationsController::filterApplicationsByRole() :905-945	per-app matchesCaller(..., allowAdminBypass: false, roles: [owners, editors, viewers]); no role ⇒ skipped
ApplicationsController::requirePermission() :1206-1247	same check, allowAdminBypass: false, then an explicit admin branch, then 403 openbuild.rbac.no_role
getManifest() :216-220	requirePermission() is called before any branch that emits the manifest

The fixture is not the problem either — 1.10's own assertions (which pass) prove the
permissions block reads back with correctly user:/group:-prefixed principals,
and the outsider is in none of them.

So the code cannot produce a 200 for a genuinely role-less caller, and yet a 200
came back. Something other than the guard answered.

The likely mechanism: a shared cookie jar, so every request runs as admin

Newman keeps one cookie jar for the whole run. The collection authenticates at
collection level as admin; Nextcloud issues a session cookie; the per-request
basic auth on 2.3 / 3.1 then competes with a session that is already established.

The corroboration is in this run's own output, in a different collection:

expected { message: 'CSRF check failed' } to be an array

Nextcloud only enforces CSRF when a session cookie is present — a pure
basic-auth request never trips it. That error is therefore direct evidence that a
session cookie is in play across these collections.

🔑 And this inverts the positive control. If every request executes as admin,
then:

• 2.1 "Viewer sees the test app" and 2.2 "Editor sees the test app" pass for
  the wrong reason — admin sees the app through the audited bypass regardless of
  role;
• 2.3 and 3.1 fail for the wrong reason, for the same reason.

That is the whole RBAC collection measuring one principal. It is the shillinq lesson
turned around: a suite that runs every request as admin cannot fail — or pass — a
membership check on its merits.

What I did not do

I did not file an IDOR issue. This repo already has one open claim of that shape
(#171) that was refuted after measurement, and filing a second unverified one is
worse than filing none. The evidence I have is a failing assertion contradicted by
four hand-read guards.

The decisive experiment, for whoever picks this up

1. Pin the identity per request — clear the cookie jar between principals
   (newman --cookie-jar per identity, or an explicit Cookie: reset), then re-run.
   If 2.3 and 3.1 go green, they were fixture artifacts and the collection was never
   testing RBAC.
2. If they stay red with a provably role-less identity, it is a real IDOR and
   should be filed immediately — at that point the guards would have to be
   bypassed by something upstream of the controller.
3. Either way, assert the effective UID inside each RBAC request (e.g. a
   /ocs/v2.php/cloud/user probe) so the collection proves which principal it ran
   as. Right now nothing in it does, which is why this is ambiguous at all.

The other 13, briefly

Independent of the identity question, and each worth its own look:

• generate with an unknown templateId returns 500, twice (ExportsController)
  — an unknown id should be a 4xx.
• PUT an invalid manifest returns 200 where a 4xx is expected.
• GET .../hello-world/manifest does not reflect a newly-added page.
• Version diff endpoint returns 404.
• createFromTemplate for a different user with the same slug returns 409
  slug_collision where the collection expects cross-user reuse — that one may well
  be the test's model being wrong rather than the product's.
• approve pass-through returns 405, not the expected 404.

Suggestion

Land the CI wiring and the collections, but keep the Newman job non-blocking
(continue-on-error) in the same change, with an issue tracking the 15 assertions
down to zero. That gets the collections running and visible without parking a
permanently-red required cell on development — and it avoids the alternative
temptation, which is deleting assertions until it goes green.

Not merged in this sweep for that reason.

[github-actions]

Quality Report — ConductionNL/openbuild @ 292e804

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
check-gitignore		✅
check-nc-floor		✅
composer			✅	✅ 101/101
npm			✅	✅ 654/654
PHPUnit					✅
Newman					❌
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-12 07:47 UTC

Download the full PDF report from the workflow artifacts.