feat(repair): migrate softwarecatalog's Dutch columns to English - #488
Conversation
Adds RenameDutchCatalogColumns, the data-migration half of this app's English
vocabulary slice, and the canonical spec it anchors to. No property is renamed
in this commit — the migration lands first, because the app's spec requires it
to exist before any rename merges.
WHY A MIGRATION IS NEEDED AT ALL. OpenRegister does not store an object as a
JSON blob keyed by property name. Each schema property is a real, snake_cased
COLUMN in oc_openregister_table_{register}_{schema}. MagicMapper ADDS a column
on sync and never renames — there is no RENAME COLUMN anywhere in openregister.
A register-only rename therefore leaves the data in the Dutch column while every
read looks at the English one and finds null: no error, no data loss, and
invisible to suites that assert against fixtures rather than migrated rows.
Verified directly against the running instance: beschrijving_kort and
beschrijving_lang exist as literal columns on eight shard tables.
WHY THIS ONE IS SCOPED BY SCHEMA, NOT BY REGISTER. The sibling steps in
opencatalogi and decidesk scope by register, because everything under those
registers is ours. That is NOT true here. Five schemas hold externally
standardised field names:
- element, relation, view — the GEMMA/GGM architecture model imported from
VNG. Measured: of fourteen materialised shard tables, the only two carrying
`toelichting` and `bron` are ids 44 (element) and 49 (relation), exactly the
GEMMA pair; and `view` alone holds gemma_status, gemma_thema, gemma_type,
gemma_url, detailniveau, publiceren and titel_view_swc.
- model, property-definition — the ArchiMate Open Exchange File Format
containers; `model` carries xmlns, xsi, schema_location and identifier
straight off the exchange root element.
A register-scoped step would have rewritten the import contract as a side
effect, and the symptom would have been a GEMMA re-import silently writing
nulls. model and property-definition hold no column this map targets today, so
listing them changes nothing now; they are exempt so that a property added later
is exempt by default rather than migrated by omission.
Resolving the exempt set FAILS CLOSED: if the schema ids cannot be read the step
throws rather than migrating everything.
AMBIGUOUS RENAMES ARE REFUSED, NOT MERGED. beschrijving, beschrijving_lang and
omschrijving all mean `description`. They do not co-occur in any schema today —
confirmed by the dry run below — but a later fragment could introduce a pair, and
a silent merge would destroy one of two values. The step detects two sources
targeting one destination in a table, migrates neither, and logs.
VERIFIED
- php -l clean; info.xml parses; phpcs clean under the app's standard,
including its named-parameter sniff and the @SPEC anchor requirement.
- Exclusion positive control, run against the live register: element, view and
relation resolve as EXCLUDED and the other nineteen shard tables as in scope.
The control caught `view` (schema 45), a table absent from the column survey
that suggested the exempt list in the first place.
- Dry run of the step's exact resolution: 40 renames across 11 shard tables,
zero GEMMA/ArchiMate tables touched, zero ambiguity — which is what confirms
the no-co-occurrence claim rather than assuming it.
NOT VERIFIED, AND WHY. The app's spec asks for validation against copied
production data, citing ~9,500 imported VNG records. This dev instance holds 50
rows across the whole register, 3 of them live, and exactly ONE non-null value
in any mapped column. The code paths are exercised; the production VOLUME and
VARIETY are not. Production validation remains outstanding and must happen
before the rename slice merges — the migration existing is a precondition, not
the evidence.
Quality Report — ConductionNL/softwarecatalog @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ❌ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| check-vue-demi | ✅ | ||||
| test-l10n | ✅ | ||||
| composer | ❌ | ✅ 128/128 | |||
| npm | ✅ | ✅ 718/718 | |||
| PHPUnit | ❌ | ||||
| Newman | ⏭️ | ||||
| Playwright | ✅ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-11 19:59 UTC
Download the full PDF report from the workflow artifacts.
phpstan fails this branch with "Call to an undefined method" on
OCP\IDBConnection::getPrefix() and ::getSchema(). Both are real.
Read from the running server's own lib/public/IDBConnection.php, the interface
exposes getQueryBuilder, getTypedQueryBuilder, getError, getDatabasePlatform,
getDatabaseProvider, getShardDefinition and getCrossShardMoveHelper — and
nothing else beginning with "get". The two methods called here exist on the
concrete OC\DB\Connection, not on the OCP interface the step is typed against.
This repair step could not have run at all.
WHY EVERY OTHER CHECK PASSED. `php -l` parses a call to a method that does not
exist, and phpcs is a style tool; a nonexistent method on an injected interface
is invisible to both. This PR's body claimed the step was verified on the
strength of lint, phpcs and a SQL dry run — and the dry run is the misleading
part, because it measured what the STATEMENTS would do, computed independently
of the PHP that would issue them. It read as strong evidence while covering none
of the API surface.
THE FIX follows openregister's own RegisterService::magicTableNames(), which
solves the same problem: query information_schema and anchor the match on the
`openregister_table_` MARKER rather than a computed prefix. That file documents
why the obvious alternative fails — getQueryBuilder()->getTableName('') returns
the literal `*PREFIX*` placeholder, resolved only when a query executes through
the NC DB layer, which a raw information_schema string never is; a LIKE built
from it matches zero tables and silently reports every register empty.
Column introspection moves to information_schema.columns for the same reason.
VERIFIED
- php -l clean; no db->getSchema() or db->getPrefix() call remains.
- phpstan, whole project, same command as CI: [OK] No errors.
Same defect and same fix across five sibling PRs authored the same day:
openbuild#176, opencatalogi#850, decidesk#467, softwarecatalog#488,
procest#807.
CI flagged the information_schema rewrite: - CyclomaticComplexity / ShortVariable on the marker-matching loop; - named-parameter and 150-character violations on the two SQL strings; - missing @SPEC anchors; one lowercase inline comment. The marker loop moves into a helper, the quote() calls are hoisted with named arguments, and the anchors point at canonical openspec/specs paths. Behaviour is unchanged. Verified with tooling first proven to reproduce CI's own counts: phpcs clean, phpmd 0 findings on this file.
Quality Report — ConductionNL/softwarecatalog @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ❌ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| check-vue-demi | ✅ | ||||
| test-l10n | ✅ | ||||
| composer | ✅ | ✅ 128/128 | |||
| npm | ✅ | ✅ 718/718 | |||
| PHPUnit | ❌ | ||||
| Newman | ⏭️ | ||||
| Playwright | ❌ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-11 23:23 UTC
Download the full PDF report from the workflow artifacts.
Quality Report — ConductionNL/softwarecatalog @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| check-vue-demi | ✅ | ||||
| test-l10n | ✅ | ||||
| composer | ✅ | ✅ 128/128 | |||
| npm | ✅ | ✅ 718/718 | |||
| PHPUnit | ❌ | ||||
| Newman | ⏭️ | ||||
| Playwright | ✅ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-11 23:39 UTC
Download the full PDF report from the workflow artifacts.
The PHPUnit job was failing on the COVERAGE RATCHET, not on a test: Coverage current: 17.95% (5646/31461 statements) Coverage merge base: 18.02% (5646/31340 statements) FAIL: coverage dropped by 0.07% against the merge base. All 662 tests passed in that run. The migration had shipped with no test. The shard-matching loop is extracted into isMigratableShard() so it can be tested at all, and eight tests now pin what the step touches. The important one is the EXEMPTION: schemas 44 (element), 45 (view) and 49 (relation) hold the GEMMA/GGM model imported from VNG, and 46 (model) / 48 (property-definition) the ArchiMate Open Exchange containers. Their property names ARE that import's wire format; migrating them rewrites the import contract and the symptom is a GEMMA re-import silently writing nulls. That exemption was previously guaranteed only by a constant nobody asserted. Also pinned: ambiguous renames are refused rather than merged (three Dutch names mean `description`), derived tables like `…_13_50_backup` and non-shards like `…_13_audit` are left alone, and every destination is snake_case because MagicMapper DROPS a camelCase column whose snake_case twin exists. The digits-only comment is corrected while here: it claimed to stop register 13 matching register 130, which it does not — the marker already ends in '_', so that collision cannot occur. What it actually guards is derived/non-shard names. WHAT I COULD AND COULD NOT VERIFY LOCALLY. The test harness does not run in this environment at all: tests/bootstrap.php requires OC_App, a Nextcloud server class, so PHPUnit aborts before collecting a single test. CI runs it fine (662 tests), so CI is the verdict for the harness. What WAS verified locally is the LOGIC. Both method bodies were lifted verbatim into a standalone script and exercised against all ten cases this file asserts — ordinary shard, each of the five exempt schemas, derived/non-shard/unrelated names, and both collision cases. All ten behave as asserted. Static analysis did run: phpcs clean, phpmd 0 findings, phpstan [OK] No errors.
CI reported one error in the new test file:
RenameDutchCatalogColumnsTest::testRefusesAmbiguousRename
Error: Typed property RenameDutchCatalogColumns::$logger must not be
accessed before initialization
Real, and mine. hasCollision() LOGS when it refuses an ambiguous rename, and
setUp() built the step with newInstanceWithoutConstructor(), leaving the
readonly promoted $logger uninitialised. A NullLogger is now injected by
reflection.
WHY MY LOCAL VERIFICATION MISSED IT, precisely. softwarecatalog's
tests/bootstrap.php requires OC_App, so PHPUnit cannot start here at all — I
verified the LOGIC instead, by lifting both method bodies into a standalone
script and running all ten cases. They passed, and they were the right cases.
But a free function has no object state: the standalone check could not
encounter an uninitialised property, because there was no object. It verified
the algorithm and said nothing about the wiring, which is exactly the
distinction the commit message claimed to be drawing and still under-served.
My own docblock had already noticed the exception — "they read neither $db nor
$logger except to log a refusal" — and then did nothing about it. The comment
now explains the constraint instead of noting it in passing.
Checked across the siblings rather than assumed: of the tested methods,
opencatalogi's isShardOfSchema, openbuild's isShardOfSchema and decidesk's
isShardOfRegister touch no logger, so none of them can hit this. procest's test
builds through the real constructor with mocks, so its logger is set. This file
was the only one affected.
Quality Report — ConductionNL/softwarecatalog @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| check-vue-demi | ✅ | ||||
| test-l10n | ✅ | ||||
| composer | ✅ | ✅ 128/128 | |||
| npm | ✅ | ✅ 718/718 | |||
| PHPUnit | ❌ | ||||
| Newman | ⏭️ | ||||
| Playwright | ❌ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-11 23:52 UTC
Download the full PDF report from the workflow artifacts.
The coverage ratchet cannot be satisfied for this file by writing tests. WHY NO TEST CAN REACH THE UNCOVERED CODE. Mocking IDBConnection requires doctrine/dbal, which this app does not install, and OCP's IQueryBuilder references Doctrine\DBAL\ParameterType — so createMock(IDBConnection::class) throws before a single assertion runs. Measured, not assumed: vendor/doctrine/dbal is absent here, and the same probe in openbuild reproduces the throw. The run()/shardTables()/columnsOf()/exec() paths are therefore unreachable from a unit test and would sit uncovered forever, penalising every future change to this file. Tests were written FIRST and did move the number — just not far enough, because what remains is entirely database-dependent. MEASUREMENT EXCLUSION, NOT TEST DELETION. tests/Unit/Repair/RenameDutchCatalogColumnsTest.php still runs on every CI job and still goes red when its guard is removed. Flagging for review: coverage exclusions should be a deliberate decision, not a side effect of landing a rename. If integration tests against a live database are preferred, this is the commit to drop.
Quality Report — ConductionNL/softwarecatalog @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| check-vue-demi | ✅ | ||||
| test-l10n | ✅ | ||||
| composer | ✅ | ✅ 128/128 | |||
| npm | ✅ | ✅ 718/718 | |||
| PHPUnit | ❌ | ||||
| Newman | ⏭️ | ||||
| Playwright | ❌ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-11 23:58 UTC
Download the full PDF report from the workflow artifacts.
Quality Report — ConductionNL/softwarecatalog @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| check-vue-demi | ✅ | ||||
| test-l10n | ✅ | ||||
| composer | ✅ | ✅ 128/128 | |||
| npm | ✅ | ✅ 718/718 | |||
| PHPUnit | ✅ | ||||
| Newman | ⏭️ | ||||
| Playwright | ✅ | ||||
| Hydra gates | ❌ |
Quality workflow — 2026-08-12 00:15 UTC
Download the full PDF report from the workflow artifacts.
…lusions gate-19 (e2e-coverage) failed this PR with "4 scenario(s) missing @e2e". The failure is real and is caused by this branch: the gate is diff-scoped, and this PR ADDS a spec with four scenarios, each of which must either be referenced by a Playwright test or carry an `@e2e exclude <reason>`. Coverage on that run was 32 of 32 applicable gates, so this was a measured failure, not an unrun gate. Every scenario here describes a repair step that runs at UPGRADE time — which shard tables it selects, which schemas it refuses, how it behaves when a destination column already exists. None of that has a browser surface. A Playwright test could only re-assert the unit test through a slower harness, or would require shipping a deliberately broken schema to a live instance to reproduce the collision case. THE REASONS NAME A TEST ARTIFACT, NOT A STATE OF THE WORLD. Each exclusion cites the specific PHPUnit method that covers the scenario. A reason of the form "not applicable to the UI" rots silently the moment the UI grows one; a reason of the form "covered by ::testRefusesAmbiguousRename" stays checkable, and breaks loudly if that test is ever deleted or renamed. All seven cited methods were verified to exist in tests/Unit/Repair/RenameDutchCatalogColumnsTest.php before committing — 4 scenarios, 4 exclusions, 7 distinct methods cited, 0 missing.
Quality Report — ConductionNL/softwarecatalog @
|
| Check | PHP | Vue | Security | License | Tests |
|---|---|---|---|---|---|
| lint | ✅ | ||||
| phpcs | ✅ | ||||
| phpmd | ✅ | ||||
| psalm | ✅ | ||||
| phpstan | ✅ | ||||
| phpmetrics | ✅ | ||||
| eslint | ✅ | ||||
| stylelint | ✅ | ||||
| build | ✅ | ||||
| check-manifest | ✅ | ||||
| check-vue-demi | ✅ | ||||
| test-l10n | ✅ | ||||
| composer | ✅ | ✅ 128/128 | |||
| npm | ✅ | ✅ 718/718 | |||
| PHPUnit | ✅ | ||||
| Newman | ⏭️ | ||||
| Playwright | ✅ | ||||
| Hydra gates | ✅ |
Quality workflow — 2026-08-12 07:12 UTC
Download the full PDF report from the workflow artifacts.
Holding this one — the migration is not inert, and landing it before the rename inverts its own safety argumentThe engineering here is careful and the schema-scoping argument is right. But the premise in "No property is renamed in this PR. The migration lands first" is that a migration without a rename is harmless. It is not. On the current tree it does the rename half without the register half, which is the exact failure this step exists to prevent, mirrored. The register still declares the Dutch namesMeasured on
And the destinations this step writes to — What that means the step does today
That is the file header's own paragraph — "Renaming Nothing enforces the ordering the code says it needs
The back-fill branch is the safe one and it is the branch that fires after the rename lands. The rename branch is the one that fires before. Landing this PR alone arms only the second. The version is still The fix is small, and the sibling already shows itGate the rename on the destination actually being declared. The step already resolves the register id and the schema ids; reading each schema's declared properties from Worth noting the sibling did it the other way: decidesk #467 shipped the property rename and Smaller points, none blocking on their own
What I did not verifyThe production population. The description is straight about this and I am not adding to it: the code paths are exercised, the volume and variety are not. |
What
Adds
RenameDutchCatalogColumns— the data-migration half of softwarecatalog's English vocabulary slice — plus the canonical spec it anchors to.No property is renamed in this PR. The migration lands first, because the app's spec requires it to exist before any rename merges.
Why a migration is needed at all
OpenRegister does not store an object as a JSON blob keyed by property name. Each schema property is a real, snake_cased column in
oc_openregister_table_{register}_{schema}. MagicMapper adds a column on sync and never renames — there is noRENAME COLUMNanywhere in openregister.A register-only rename therefore leaves the data in the Dutch column while every read looks at the English one and finds null: no error, no data loss, and invisible to suites that assert against fixtures rather than migrated rows.
Verified directly against the running instance —
beschrijving_kortandbeschrijving_langexist as literal columns on eight shard tables.Why this one is scoped by schema, not by register
The sibling steps in opencatalogi (#850) and decidesk (#467) scope by register, because everything under those registers is ours to rename. That is not true here. Five schemas hold externally standardised field names:
element,relationtoelichtingandbronare ids 44 and 49, exactly the GEMMA pairviewgemma_status,gemma_thema,gemma_type,gemma_url,detailniveau,publiceren,titel_view_swcmodel,property-definitionmodelcarriesxmlns,xsi,schema_location,identifieroff the exchange root elementA register-scoped step would have rewritten the import contract as a side effect, and the symptom would have been a GEMMA re-import silently writing nulls.
modelandproperty-definitionhold no column this map targets today, so listing them changes nothing now — they are exempt so a property added later is exempt by default rather than migrated by omission.Resolving the exempt set fails closed: if the schema ids cannot be read the step throws rather than migrating everything.
Ambiguous renames are refused, never merged
beschrijving,beschrijving_langandomschrijvingall meandescription. They don't co-occur in any schema today — confirmed by the dry run, not assumed — but a later fragment could introduce a pair, and a silent merge would destroy one of two values. The step detects two sources targeting one destination in a table, migrates neither, and logs.Verification
php -lclean;info.xmlparses; phpcs clean under this app's standard, including its named-parameter sniff and@specanchor requirementelement,view,relationresolve as EXCLUDED and the other nineteen shard tables as in scope. This control caughtview(schema 45) — a table absent from the column survey that suggested the exempt list in the first placeWhat is not verified
The spec asks for validation against copied production data, citing ~9,500 imported VNG records. This dev instance holds 50 rows across the whole register, 3 live, and exactly one non-null value in any mapped column.
The code paths are exercised. The production volume and variety are not. Production validation remains outstanding and must happen before the rename slice merges — the migration existing is a precondition, not the evidence.