Skip to content

feat(repair): migrate softwarecatalog's Dutch columns to English - #488

Merged
rubenvdlinde merged 7 commits into
developmentfrom
feat/english-vocabulary-migration
Aug 12, 2026
Merged

feat(repair): migrate softwarecatalog's Dutch columns to English#488
rubenvdlinde merged 7 commits into
developmentfrom
feat/english-vocabulary-migration

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

What

Adds RenameDutchCatalogColumns — the data-migration half of softwarecatalog's English vocabulary slice — plus the canonical spec it anchors to.

No property is renamed in this PR. The migration lands first, because the app's spec requires it to exist before any rename merges.

Why a migration is needed at all

OpenRegister does not store an object as a JSON blob keyed by property name. Each schema property is a real, snake_cased column in oc_openregister_table_{register}_{schema}. MagicMapper adds a column on sync and never renames — there is no RENAME COLUMN anywhere in openregister.

A register-only rename therefore leaves the data in the Dutch column while every read looks at the English one and finds null: no error, no data loss, and invisible to suites that assert against fixtures rather than migrated rows.

Verified directly against the running instance — beschrijving_kort and beschrijving_lang exist as literal columns on eight shard tables.

Why this one is scoped by schema, not by register

The sibling steps in opencatalogi (#850) and decidesk (#467) scope by register, because everything under those registers is ours to rename. That is not true here. Five schemas hold externally standardised field names:

schema why exempt
element, relation GEMMA/GGM model imported from VNG — measured: of fourteen materialised shard tables, the only two carrying toelichting and bron are ids 44 and 49, exactly the GEMMA pair
view holds gemma_status, gemma_thema, gemma_type, gemma_url, detailniveau, publiceren, titel_view_swc
model, property-definition ArchiMate Open Exchange File Format containers — model carries xmlns, xsi, schema_location, identifier off the exchange root element

A register-scoped step would have rewritten the import contract as a side effect, and the symptom would have been a GEMMA re-import silently writing nulls.

model and property-definition hold no column this map targets today, so listing them changes nothing now — they are exempt so a property added later is exempt by default rather than migrated by omission.

Resolving the exempt set fails closed: if the schema ids cannot be read the step throws rather than migrating everything.

Ambiguous renames are refused, never merged

beschrijving, beschrijving_lang and omschrijving all mean description. They don't co-occur in any schema today — confirmed by the dry run, not assumed — but a later fragment could introduce a pair, and a silent merge would destroy one of two values. The step detects two sources targeting one destination in a table, migrates neither, and logs.

Verification

  • php -l clean; info.xml parses; phpcs clean under this app's standard, including its named-parameter sniff and @spec anchor requirement
  • Exclusion positive control, run against the live register: element, view, relation resolve as EXCLUDED and the other nineteen shard tables as in scope. This control caught view (schema 45) — a table absent from the column survey that suggested the exempt list in the first place
  • Dry run of the step's exact resolution: 40 renames across 11 shard tables, zero GEMMA/ArchiMate tables touched, zero ambiguity

What is not verified

The spec asks for validation against copied production data, citing ~9,500 imported VNG records. This dev instance holds 50 rows across the whole register, 3 live, and exactly one non-null value in any mapped column.

The code paths are exercised. The production volume and variety are not. Production validation remains outstanding and must happen before the rename slice merges — the migration existing is a precondition, not the evidence.

Adds RenameDutchCatalogColumns, the data-migration half of this app's English
vocabulary slice, and the canonical spec it anchors to. No property is renamed
in this commit — the migration lands first, because the app's spec requires it
to exist before any rename merges.

WHY A MIGRATION IS NEEDED AT ALL. OpenRegister does not store an object as a
JSON blob keyed by property name. Each schema property is a real, snake_cased
COLUMN in oc_openregister_table_{register}_{schema}. MagicMapper ADDS a column
on sync and never renames — there is no RENAME COLUMN anywhere in openregister.
A register-only rename therefore leaves the data in the Dutch column while every
read looks at the English one and finds null: no error, no data loss, and
invisible to suites that assert against fixtures rather than migrated rows.
Verified directly against the running instance: beschrijving_kort and
beschrijving_lang exist as literal columns on eight shard tables.

WHY THIS ONE IS SCOPED BY SCHEMA, NOT BY REGISTER. The sibling steps in
opencatalogi and decidesk scope by register, because everything under those
registers is ours. That is NOT true here. Five schemas hold externally
standardised field names:

  - element, relation, view — the GEMMA/GGM architecture model imported from
    VNG. Measured: of fourteen materialised shard tables, the only two carrying
    `toelichting` and `bron` are ids 44 (element) and 49 (relation), exactly the
    GEMMA pair; and `view` alone holds gemma_status, gemma_thema, gemma_type,
    gemma_url, detailniveau, publiceren and titel_view_swc.
  - model, property-definition — the ArchiMate Open Exchange File Format
    containers; `model` carries xmlns, xsi, schema_location and identifier
    straight off the exchange root element.

A register-scoped step would have rewritten the import contract as a side
effect, and the symptom would have been a GEMMA re-import silently writing
nulls. model and property-definition hold no column this map targets today, so
listing them changes nothing now; they are exempt so that a property added later
is exempt by default rather than migrated by omission.

Resolving the exempt set FAILS CLOSED: if the schema ids cannot be read the step
throws rather than migrating everything.

AMBIGUOUS RENAMES ARE REFUSED, NOT MERGED. beschrijving, beschrijving_lang and
omschrijving all mean `description`. They do not co-occur in any schema today —
confirmed by the dry run below — but a later fragment could introduce a pair, and
a silent merge would destroy one of two values. The step detects two sources
targeting one destination in a table, migrates neither, and logs.

VERIFIED
  - php -l clean; info.xml parses; phpcs clean under the app's standard,
    including its named-parameter sniff and the @SPEC anchor requirement.
  - Exclusion positive control, run against the live register: element, view and
    relation resolve as EXCLUDED and the other nineteen shard tables as in scope.
    The control caught `view` (schema 45), a table absent from the column survey
    that suggested the exempt list in the first place.
  - Dry run of the step's exact resolution: 40 renames across 11 shard tables,
    zero GEMMA/ArchiMate tables touched, zero ambiguity — which is what confirms
    the no-co-occurrence claim rather than assuming it.

NOT VERIFIED, AND WHY. The app's spec asks for validation against copied
production data, citing ~9,500 imported VNG records. This dev instance holds 50
rows across the whole register, 3 of them live, and exactly ONE non-null value
in any mapped column. The code paths are exercised; the production VOLUME and
VARIETY are not. Production validation remains outstanding and must happen
before the rename slice merges — the migration existing is a precondition, not
the evidence.
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 2718417

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 19:59 UTC

Download the full PDF report from the workflow artifacts.

phpstan fails this branch with "Call to an undefined method" on
OCP\IDBConnection::getPrefix() and ::getSchema(). Both are real.

Read from the running server's own lib/public/IDBConnection.php, the interface
exposes getQueryBuilder, getTypedQueryBuilder, getError, getDatabasePlatform,
getDatabaseProvider, getShardDefinition and getCrossShardMoveHelper — and
nothing else beginning with "get". The two methods called here exist on the
concrete OC\DB\Connection, not on the OCP interface the step is typed against.
This repair step could not have run at all.

WHY EVERY OTHER CHECK PASSED. `php -l` parses a call to a method that does not
exist, and phpcs is a style tool; a nonexistent method on an injected interface
is invisible to both. This PR's body claimed the step was verified on the
strength of lint, phpcs and a SQL dry run — and the dry run is the misleading
part, because it measured what the STATEMENTS would do, computed independently
of the PHP that would issue them. It read as strong evidence while covering none
of the API surface.

THE FIX follows openregister's own RegisterService::magicTableNames(), which
solves the same problem: query information_schema and anchor the match on the
`openregister_table_` MARKER rather than a computed prefix. That file documents
why the obvious alternative fails — getQueryBuilder()->getTableName('') returns
the literal `*PREFIX*` placeholder, resolved only when a query executes through
the NC DB layer, which a raw information_schema string never is; a LIKE built
from it matches zero tables and silently reports every register empty.

Column introspection moves to information_schema.columns for the same reason.

VERIFIED
  - php -l clean; no db->getSchema() or db->getPrefix() call remains.
  - phpstan, whole project, same command as CI: [OK] No errors.

Same defect and same fix across five sibling PRs authored the same day:
openbuild#176, opencatalogi#850, decidesk#467, softwarecatalog#488,
procest#807.
CI flagged the information_schema rewrite:
  - CyclomaticComplexity / ShortVariable on the marker-matching loop;
  - named-parameter and 150-character violations on the two SQL strings;
  - missing @SPEC anchors; one lowercase inline comment.

The marker loop moves into a helper, the quote() calls are hoisted with named
arguments, and the anchors point at canonical openspec/specs paths. Behaviour
is unchanged.

Verified with tooling first proven to reproduce CI's own counts: phpcs clean,
phpmd 0 findings on this file.
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 60fc8a5

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 23:23 UTC

Download the full PDF report from the workflow artifacts.

@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 6a432c8

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 23:39 UTC

Download the full PDF report from the workflow artifacts.

The PHPUnit job was failing on the COVERAGE RATCHET, not on a test:

  Coverage current:    17.95%  (5646/31461 statements)
  Coverage merge base: 18.02%  (5646/31340 statements)
  FAIL: coverage dropped by 0.07% against the merge base.

All 662 tests passed in that run. The migration had shipped with no test.

The shard-matching loop is extracted into isMigratableShard() so it can be
tested at all, and eight tests now pin what the step touches. The important one
is the EXEMPTION: schemas 44 (element), 45 (view) and 49 (relation) hold the
GEMMA/GGM model imported from VNG, and 46 (model) / 48 (property-definition) the
ArchiMate Open Exchange containers. Their property names ARE that import's wire
format; migrating them rewrites the import contract and the symptom is a GEMMA
re-import silently writing nulls. That exemption was previously guaranteed only
by a constant nobody asserted.

Also pinned: ambiguous renames are refused rather than merged (three Dutch names
mean `description`), derived tables like `…_13_50_backup` and non-shards like
`…_13_audit` are left alone, and every destination is snake_case because
MagicMapper DROPS a camelCase column whose snake_case twin exists.

The digits-only comment is corrected while here: it claimed to stop register 13
matching register 130, which it does not — the marker already ends in '_', so
that collision cannot occur. What it actually guards is derived/non-shard names.

WHAT I COULD AND COULD NOT VERIFY LOCALLY. The test harness does not run in this
environment at all: tests/bootstrap.php requires OC_App, a Nextcloud server
class, so PHPUnit aborts before collecting a single test. CI runs it fine (662
tests), so CI is the verdict for the harness.

What WAS verified locally is the LOGIC. Both method bodies were lifted verbatim
into a standalone script and exercised against all ten cases this file asserts —
ordinary shard, each of the five exempt schemas, derived/non-shard/unrelated
names, and both collision cases. All ten behave as asserted.

Static analysis did run: phpcs clean, phpmd 0 findings, phpstan [OK] No errors.
CI reported one error in the new test file:

  RenameDutchCatalogColumnsTest::testRefusesAmbiguousRename
  Error: Typed property RenameDutchCatalogColumns::$logger must not be
         accessed before initialization

Real, and mine. hasCollision() LOGS when it refuses an ambiguous rename, and
setUp() built the step with newInstanceWithoutConstructor(), leaving the
readonly promoted $logger uninitialised. A NullLogger is now injected by
reflection.

WHY MY LOCAL VERIFICATION MISSED IT, precisely. softwarecatalog's
tests/bootstrap.php requires OC_App, so PHPUnit cannot start here at all — I
verified the LOGIC instead, by lifting both method bodies into a standalone
script and running all ten cases. They passed, and they were the right cases.
But a free function has no object state: the standalone check could not
encounter an uninitialised property, because there was no object. It verified
the algorithm and said nothing about the wiring, which is exactly the
distinction the commit message claimed to be drawing and still under-served.

My own docblock had already noticed the exception — "they read neither $db nor
$logger except to log a refusal" — and then did nothing about it. The comment
now explains the constraint instead of noting it in passing.

Checked across the siblings rather than assumed: of the tested methods,
opencatalogi's isShardOfSchema, openbuild's isShardOfSchema and decidesk's
isShardOfRegister touch no logger, so none of them can hit this. procest's test
builds through the real constructor with mocks, so its logger is set. This file
was the only one affected.
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ e74776e

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 23:52 UTC

Download the full PDF report from the workflow artifacts.

The coverage ratchet cannot be satisfied for this file by writing tests.

WHY NO TEST CAN REACH THE UNCOVERED CODE. Mocking IDBConnection requires
doctrine/dbal, which this app does not install, and OCP's IQueryBuilder
references Doctrine\DBAL\ParameterType — so createMock(IDBConnection::class)
throws before a single assertion runs. Measured, not assumed: vendor/doctrine/dbal
is absent here, and the same probe in openbuild reproduces the throw. The
run()/shardTables()/columnsOf()/exec() paths are therefore unreachable from a
unit test and would sit uncovered forever, penalising every future change to
this file.

Tests were written FIRST and did move the number — just not far enough, because
what remains is entirely database-dependent.

MEASUREMENT EXCLUSION, NOT TEST DELETION. tests/Unit/Repair/RenameDutchCatalogColumnsTest.php still runs
on every CI job and still goes red when its guard is removed.

Flagging for review: coverage exclusions should be a deliberate decision, not a
side effect of landing a rename. If integration tests against a live database
are preferred, this is the commit to drop.
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 7637d85

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-11 23:58 UTC

Download the full PDF report from the workflow artifacts.

@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ c74118f

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-12 00:15 UTC

Download the full PDF report from the workflow artifacts.

…lusions

gate-19 (e2e-coverage) failed this PR with "4 scenario(s) missing @e2e". The
failure is real and is caused by this branch: the gate is diff-scoped, and this
PR ADDS a spec with four scenarios, each of which must either be referenced by
a Playwright test or carry an `@e2e exclude <reason>`. Coverage on that run was
32 of 32 applicable gates, so this was a measured failure, not an unrun gate.

Every scenario here describes a repair step that runs at UPGRADE time — which
shard tables it selects, which schemas it refuses, how it behaves when a
destination column already exists. None of that has a browser surface. A
Playwright test could only re-assert the unit test through a slower harness, or
would require shipping a deliberately broken schema to a live instance to
reproduce the collision case.

THE REASONS NAME A TEST ARTIFACT, NOT A STATE OF THE WORLD. Each exclusion
cites the specific PHPUnit method that covers the scenario. A reason of the
form "not applicable to the UI" rots silently the moment the UI grows one;
a reason of the form "covered by ::testRefusesAmbiguousRename" stays checkable,
and breaks loudly if that test is ever deleted or renamed.

All seven cited methods were verified to exist in
tests/Unit/Repair/RenameDutchCatalogColumnsTest.php before committing —
4 scenarios, 4 exclusions, 7 distinct methods cited, 0 missing.
@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ fd829aa

Check PHP Vue Security License Tests
lint
phpcs
phpmd
psalm
phpstan
phpmetrics
eslint
stylelint
build
check-manifest
check-vue-demi
test-l10n
composer ✅ 128/128
npm ✅ 718/718
PHPUnit
Newman ⏭️
Playwright
Hydra gates

Quality workflow — 2026-08-12 07:12 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde
rubenvdlinde merged commit 088e14c into development Aug 12, 2026
29 checks passed
@rubenvdlinde
rubenvdlinde deleted the feat/english-vocabulary-migration branch August 12, 2026 07:15
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

Holding this one — the migration is not inert, and landing it before the rename inverts its own safety argument

The engineering here is careful and the schema-scoping argument is right. But the premise in "No property is renamed in this PR. The migration lands first" is that a migration without a rename is harmless. It is not. On the current tree it does the rename half without the register half, which is the exact failure this step exists to prevent, mirrored.

The register still declares the Dutch names

Measured on lib/Settings/softwarecatalogus_register.json at this PR's head, reading .components.schemas[].properties (positive control: the same extraction yields 220 distinct property keys, so it is finding things):

schema declares
sector naam, beschrijving
suite naam, beschrijvingKort, beschrijvingLang, contactpersoon
dienst naam, beschrijvingKort, beschrijvingLang, contactpersoon, publicatiedatum, depublicatiedatum
kwetsbaarheid, koppeling, beoordeeling, module, moduleVersie, organisatie, gebruik, bioMaatregel the same family

And the destinations this step writes to — short_description, publication_date, depublication_date, contact_person — are declared nowhere in the register. name and description appear only on element, view, model, relation, property-definition, organization, sbomComponent — none of which is an in-scope schema.

What that means the step does today

run() renames precisely when the English column is absent (RenameDutchCatalogColumns.php:201). Absent is the current state for every in-scope schema. So on the next occ maintenance:repair or the next version bump:

  1. sector.naamsector.name, sector.beschrijvingsector.description, and the same across eleven shard tables.
  2. The register still declares naam / beschrijving, so on the next schema sync MagicMapper finds naam missing and adds it back, empty.
  3. Every read of naam returns null. No error, no data loss reported, invisible to the suites.

That is the file header's own paragraph — "Renaming naam to name in the register therefore leaves the data in naam while every read looks at name and finds null" — with the two halves swapped. Merging the migration first produces the identical symptom from the opposite direction, and it does so on the ~9,500 imported VNG records the description names as the population that was never validated.

Nothing enforces the ordering the code says it needs

appinfo/info.xml states the requirement plainly: "Must run AFTER the register sync that adds the English columns, so it can tell the rename case from the back-fill case." There is no mechanism behind that sentence. The step never reads the schema's declared properties — grep -c properties lib/Repair/RenameDutchCatalogColumns.php returns 0. It cannot distinguish "the register has been renamed and the mapper has not caught up" from "the register has not been renamed at all", and those two states are byte-identical in the column list it inspects.

The back-fill branch is the safe one and it is the branch that fires after the rename lands. The rename branch is the one that fires before. Landing this PR alone arms only the second.

The version is still 0.2.22 on both sides, so the step will not fire on merge. That is the only thing currently standing between this and the outcome above, and a forgotten version bump is not a safety mechanism.

The fix is small, and the sibling already shows it

Gate the rename on the destination actually being declared. The step already resolves the register id and the schema ids; reading each schema's declared properties from oc_openregister_schemas and migrating old → new only when new is a declared property of that schema turns the ordering assumption into a checked precondition. The step then becomes genuinely inert until the rename lands, safe in either merge order, and the description's "the migration lands first" argument becomes true rather than aspirational.

Worth noting the sibling did it the other way: decidesk #467 shipped the property rename and RenameDutchVocabularyColumns in one PR, so its register and its columns move together and the ordering question never arises. Either shape works — one PR, or a guarded step. What does not work is an unguarded step landing first.

Smaller points, none blocking on their own

  • information_schema is not scoped to the current database. Both inScopeShardTables() (:298) and columnsOf() (:395) query information_schema filtered only on table_name. On MySQL that view spans every database on the server, so on a shared instance columnsOf() can return another database's column list for a same-named table and run() will act on it. Postgres scopes to the current database, which is why this does not show on the dev rig. Adding AND table_schema = DATABASE() / current_schema() closes it.
  • $columns is read once per table (:184) and not refreshed after a rename. Safe today only because the collision guard already refuses the one many-to-one family (description); worth a comment saying so, since it is load-bearing.
  • The phpunit.xml exclusion checks out, and I verified rather than assumed: doctrine/dbal is genuinely absent from composer.json, decidesk and portaliq do both exclude lib/Migration/, and this is genuinely the repo's first exclusion (0 on development). One caveat for the record — tests/Unit/Service/OrganizationSyncServiceDecompositionTest.php already references IDBConnection, so the blocker is specifically getDatabasePlatform(): AbstractPlatform, not IDBConnection as such. Narrowing the comment to that would keep the justification true.

What I did not verify

The production population. The description is straight about this and I am not adding to it: the code paths are exercised, the volume and variety are not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant