diff --git a/appinfo/routes.php b/appinfo/routes.php index 3586e5e40..33451b3fc 100644 --- a/appinfo/routes.php +++ b/appinfo/routes.php @@ -104,6 +104,12 @@ ['name' => 'settings#killArchiMateImport', 'url' => '/api/archimate/import/kill', 'verb' => 'POST'], // deprecated ['name' => 'settings#clearArchiMateExportStatus', 'url' => '/api/archimate/status/export/clear', 'verb' => 'POST'], + // CMDB export import (TOPdesk xlsx) — admin-only, CSRF-protected. + // Progress is read through the existing /api/progress/{operationId}. + // @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + ['name' => 'cmdbImport#import', 'url' => '/api/cmdb-import', 'verb' => 'POST'], + ['name' => 'cmdbImport#cancel', 'url' => '/api/cmdb-import/{operationId}/cancel', 'verb' => 'POST'], + // User Groups management routes ['name' => 'settings#getGenericUserGroups', 'url' => '/api/settings/user-groups/generic', 'verb' => 'GET'], ['name' => 'settings#setGenericUserGroups', 'url' => '/api/settings/user-groups/generic', 'verb' => 'POST'], diff --git a/docs/features/README.md b/docs/features/README.md index 82747caed..7fa964f0a 100644 --- a/docs/features/README.md +++ b/docs/features/README.md @@ -17,6 +17,7 @@ All data is stored as OpenRegister objects (no own database tables). OpenRegiste | [Federated Synchronisation](#federated-synchronisation) | Sync catalogue data across organisations and sources | | [Automatic User Provisioning](#automatic-user-provisioning) | Create Nextcloud users from catalogue contacts | | [ArchiMate Import/Export](#archimate-importexport) | Exchange software landscape data in ArchiMate format | +| [CMDB Import](#cmdb-import) | Import a municipality's TOPdesk CMDB export (xlsx) as applications, suppliers, usages and owners | | [Open Data Publishing](#open-data-publishing) | Expose the catalogue as a public open-data API | | [GEMMA Compliance](#gemma-compliance) | Built around VNG GEMMA Softwarecatalogus reference | @@ -155,6 +156,16 @@ The ArchiMate integration maps GEMMA Softwarecatalogus objects to ArchiMate appl **Key services:** `lib/Service/ArchiMateService.php`, `lib/Service/ArchiMateImportService.php`, `lib/Service/ArchiMateExportService.php` +## CMDB Import + +Import a TOPdesk CMDB export (`.xlsx`) for one municipality from the **CMDB import** section of the admin settings. Every application row of the two CMDB sheets ("Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB") becomes or updates a module, its vendor as a Supplier organisation, a usage that links it to the municipality and records whether maintenance is arranged, and a contact person for its owner (identity in Nextcloud Contacts, never public). A repeat import matches on APPID per municipality, so it updates instead of duplicating, and leaves applications missing from the newer export as they are. The column mapping is declarative JSON executed by OpenRegister's mapping engine. + +See [CMDB import](cmdb-import.md) for the steps, the expected file structure, the error codes and how to adjust the mapping. + +**Key services:** `lib/Service/CmdbExportImportService.php`, `lib/Service/Cmdb/` +**Controller:** `lib/Controller/CmdbImportController.php` +**Endpoint:** `POST /apps/stackiq/api/cmdb-import` + ## Open Data Publishing The catalogue is published as an open-data API. All registered applications, modules, and connections are accessible via public endpoints: diff --git a/docs/features/cmdb-import.md b/docs/features/cmdb-import.md new file mode 100644 index 000000000..5167ccdae --- /dev/null +++ b/docs/features/cmdb-import.md @@ -0,0 +1,272 @@ + + +# CMDB import + +Imports a TOPdesk CMDB export (an Excel workbook, `.xlsx`) for one +municipality. Every application row of the two CMDB sheets becomes, or +updates: + +- a **module** (the application, `schema:SoftwareApplication`); +- its vendor (the maker of the software) as an **organisation** of type + Supplier; +- a **usage** that links the application to the municipality; +- a **contact person** of the municipality for its owner, with the identity + in Nextcloud Contacts. Owners are never readable by the public. + +All of it is stored as OpenRegister objects in the stackiq register. Import a +newer export later and the same applications are updated, not duplicated. + +Specification: [`openspec/changes/cmdb-export-import/`](https://github.com/ConductionNL/stackiq/tree/development/openspec/changes/cmdb-export-import). + +## Who can import + +Only Nextcloud administrators. Members of the `software-catalog-admins` group +who are not Nextcloud administrators cannot import. The section is part of +stackiq's admin settings, under **Administration settings → Stackiq → +CMDB import**. + +## Before you start + +The import itself only needs stackiq and OpenRegister. To see the imported +applications in the other apps, two things must be set up there. The import +does not change either of them. + +**OpenCatalogi (search).** OpenCatalogi lists an application only through a +catalogue that includes the register `stackiq` and the schema `module`. In +OpenCatalogi, open the catalogue that should show the municipality's +applications and add that register and schema. A newly imported module gets +a publication date (the moment the import started), so it is listed from then +on. + +**Portaliq ("Software we use").** Portaliq shows an application to a +municipality through a usage whose consumer is that municipality. The portal +account of the municipality needs the claim `stackiq.organisationId` set to +the uuid of the municipality organisation the import used. The uuid is in +the import result (the municipality line) and on the organisation's detail +page in stackiq. + +## Steps + + + +1. Open **Administration settings → Stackiq** and scroll to **CMDB import**. +2. **Municipality.** Pick an existing organisation of type Municipality from + the list, or type the name of a new one and press Enter. A typed name that + matches an existing municipality (ignoring case and extra spaces) uses that + municipality; otherwise a new organisation of type Municipality with + status Active is created during the import. +3. **File.** Choose the TOPdesk export (`.xlsx`, at most 10 MB). +4. **Update existing records.** On by default. Turn it off to import only + applications that are new for this municipality; rows that match an + existing application are then reported as *skipped* with reason `exists` + and nothing about them changes. +5. Press **Import**. A progress bar shows how many rows have been processed. + **Cancel import** stops the import before the next row; rows that were + already processed stay imported. + + + +When the import finishes, the section shows: + +- the **summary**: rows read, created, updated, unchanged, skipped, failed + and warnings; +- **warnings for the whole file**, for example an optional column that is + missing; +- the **rows** table: sheet, row number, APPID, application, outcome, + and the reasons and warnings for that row. Filter it with **Show rows with + outcome**. The application name links to the module in stackiq. + + + +The municipality stays selected after an import, so a second import goes to +the same organisation. + +## The file + +The import reads the two CMDB sheets of the export and ignores all others, +including the `Invoer` sheets they are derived from: + +| Sheet | What it holds | Recorded on the usage | +|---|---|---| +| `Onbeh Applicaties CMDB` | applications **without** arranged maintenance (from the AIA export) | `Beheer geregeld: nee` | +| `Beheerde Applicaties CMDB` | applications **with** arranged maintenance (from the APP export) | `Beheer geregeld: ja` | + +At least one of the two must be present. Row 1 of each sheet holds the +column names. Columns are found by name per sheet, not by position: case, +surrounding spaces and a trailing `:` or `⚡` do not matter, and the order of +the columns does not matter. A column that one sheet has and the other has +not (such as `Nickname`, only on `Beheerde Applicaties CMDB`) is optional on +the sheet that lacks it. Empty rows, including formatted rows below the data, +are ignored and not counted. A sheet may hold at most 10,000 rows with data. + +Two columns are **required** on every CMDB sheet that is present: `APPID` +and `Applicatie Naam`. Every other column is optional; when one is missing, +the import names it once in the warnings for the whole file. + +**Formulas.** The CMDB sheets are formulas that read the `Invoer` sheets. +The import reads the value Excel stored with each formula cell; formulas are +never calculated. Save the workbook in Excel before importing it, so every +formula has a stored value. A formula without a stored value is read as an +empty cell and the row carries the warning `Column "…": formula without a +cached value, read as empty`; the row is still imported. A stored `0` is what +Excel shows for a reference to an empty cell, and is read as empty too. +External data connections, Power Query queries and links in the workbook are +never opened. Macro-enabled workbooks (`.xlsm`), old Excel files (`.xls`) and +CSV files are not accepted. + +**Placeholder values.** The CMDB sheets fill some empty cells with a +placeholder. These are read as empty: `NB` in `BNN Classificatie`, and the +date 2036-01-01 (Excel serial 49675) in `End-of-Life Functioneel`. + +### Columns and where they go + +| Column | Goes to | Rule | +|---|---|---| +| APPID | module external number, and the match key | required, see [Repeat imports](#repeat-imports) | +| Applicatie Naam | module name | required | +| Applicatie Code | module external id | reference only; it can change in TOPdesk, so it is not the match key | +| Roepnaam, Nickname | module short description | Roepnaam when filled, otherwise Nickname (only on `Beheerde Applicaties CMDB`) | +| Functionele Omschrijving | module long description | | +| Applicatiesoort | module hosting model (`cloudDienstverleningsmodel`) | `Saas` → SaaS, `PaaS` → PaaS, `IaaS` → IaaS, `On-premise(s)` → On-premises (self-managed); another value is dropped with a warning | +| BNN Classificatie | module BBN level | `BBN1`/`BBN 1`/`BNN1` etc. become `BBN1`, `BBN2`, `BBN3`; `NB` is empty; another value is dropped with a warning | +| Datum | module external creation date | Excel date | +| Referentie datum wijziging | module external modification date | Excel date | +| Vendor | Supplier organisation, set as provider on the module and the usage | one organisation per name, see below | +| Applicatie Status | usage status | In productie → In production, In voorraad → Planned, In ontwikkeling → Acquisition, Uit te faseren → To be phased out, Uitgefaseerd → Phased out; another value is dropped with a warning | +| Classificatie | usage TIME classification | Tolereren/Tolerate, Investeren/Invest, Migreren/Migrate, Elimineren/Eliminate | +| End-of-Life Functioneel | usage phase-out date | Excel date; 2036-01-01 is empty | +| (the sheet), Cluster, Applicatie Eigenaar (Afdeling) | usage internal annotation | `Beheer geregeld: ja` or `nee`, the cluster and the department, joined with ` / `; written only when the usage is new or the note is empty | +| Applicatie Eigenaar (Persoon), Applicatie Eigenaar (Functie) | usage business owner (contact person) | see [Owners](#owners) | + +Columns not in this table are not read at all. That includes Hostingpartij +and Leverancier (not mapped yet), the BIV and value columns (Beschikbaarheid, +Integriteit, Vertrouwelijkheid, Applicatienut and the like), Behandelgroep, +Cloud, Rappeldatum, Rappelreden, Locatie BIOToets, Software Suite, Standaard, +Top5, COTS and Applicatie Nummer. + +**Vendors.** Names are compared after trimming, collapsing spaces and +ignoring case, so `Fabfrikant`, `Fabfrikant ` and `FABFRIKANT` are one +Supplier. An existing organisation of type Supplier with the same name is +reused. A row without a vendor is imported without a provider. + +## Repeat imports + +An application is recognised by its **APPID within the municipality**: the +match key is `topdesk::`. The APPID (TOPdesk's ICT +Applicatienummer) stays the same when TOPdesk changes the Applicatie Code +(Middel-ID). Two municipalities can each have an APPID `101` without +colliding. + +- **New APPID**: a module and a usage are created. The module gets a + publication date (the moment the import started), so OpenCatalogi lists it. +- **Known APPID, values changed**: only the fields in the column table + are updated. Everything else on the module stays as it is, for example a + website an administrator added. The publication date and the depublication + date are never changed: a module an administrator depublished stays + depublished. The row is reported as *updated*. +- **Known APPID, nothing changed**: nothing is saved; the row is reported + as *unchanged*. Importing the same export twice creates nothing the second + time. +- **APPID missing from a newer export**: the application, its usage and + its contact persons are left as they are. They are not changed, depublished + or deleted. +- Each application keeps exactly one usage for the municipality. + +Rows are **skipped** when the APPID is empty (`missing APPID`), when the +Applicatie Naam is empty (`missing Applicatie Naam`), when an APPID appears a +second time in the same upload, also across the two sheets (`duplicate APPID +in file`; the first occurrence is imported), or, with **Update existing +records** off, when the application already exists (`exists`). + +An application that moves from `Onbeh Applicaties CMDB` to `Beheerde +Applicaties CMDB` keeps its module and usage (same APPID); its internal note +is not rewritten when it already has one. + +Every row is processed on its own. When one row fails, for example because +OpenRegister refuses to save it, that row is reported as *failed* with the +step that failed, and the other rows are imported. Importing again completes +the failed row. + +## Owners + +The owner becomes a **contact person of the municipality**, never a +Nextcloud user account. It comes from `Applicatie Eigenaar (Persoon)`; its +function (`Applicatie Eigenaar (Functie)`) is stored as the contact person's +role, and the department (`Applicatie Eigenaar (Afdeling)`) goes into the +usage's internal note. When TOPdesk has no owner, the CMDB sheet shows the +owner's function in the person column; the import then uses that function as +the contact's name. No technical owner is imported: the functional +administrator (FB contactpersoon) is not read. + +The identity is kept in **Nextcloud Contacts**, in the first writable +address book of the administrator who runs the import, the same as every +other stackiq contact. The CMDB sheets have no e-mail address, so a contact +is found by an exact match on the name, and created when there is none. The +stackiq contact person object only holds the link to that contact, the role +and the municipality. The same owner on several rows is one contact person. + +When the Contacts app is disabled, applications and usages are still +imported; the owners are skipped and each affected row carries a warning. + +**Never public.** Contact persons and usages have no public read rule, so an +anonymous visitor cannot read them through OpenRegister, and a published +module in an OpenCatalogi search result refers to them by id at most. The +import report and the Nextcloud log never contain owner names. + +## Errors and what to do + +When the file or the request cannot be imported at all, nothing is written +and the section shows the reason and the error code. + +| Error code | What it means | What to do | +|---|---|---| +| `NOT_XLSX` | The file is not an Excel workbook: wrong extension, or the content is not an `.xlsx` package. | Save the export as Excel workbook (`.xlsx`). | +| `FILE_TOO_LARGE` | The file is larger than 10 MB. | Remove sheets the import does not read, or split the export. | +| `NO_FILE_UPLOADED` | No file arrived. | Choose the file again. | +| `MUNICIPALITY_REQUIRED` | No municipality was chosen. | Pick or type a municipality. | +| `MUNICIPALITY_INVALID` | The chosen organisation does not exist or is not of type Municipality. | Pick an organisation of type Municipality, or type a new name. | +| `NO_SOURCE_SHEET` | Neither `Onbeh Applicaties CMDB` nor `Beheerde Applicaties CMDB` is in the workbook. | Check the sheet names; they must match exactly. | +| `MISSING_COLUMN` | A present CMDB sheet has no `APPID` or `Applicatie Naam` column. The message names the sheet and the column. | Add the column to that sheet. | +| `TOO_MANY_ROWS` | A CMDB sheet has more than 10,000 rows with data. | Split the export and import the parts one after the other. | +| `MISSING_RECORDS_UNSUPPORTED` | The request asked to mark or remove records missing from the export. Only keeping them is supported. | Not reachable from the section; reported for API callers. | +| `MAPPING_UNAVAILABLE` | OpenRegister's mapping engine is missing, or one of the mapping files is invalid. | Update OpenRegister. If you changed a mapping file, check it against the Nextcloud log. | +| `READER_UNAVAILABLE` | The Excel reader that ships with OpenRegister cannot be loaded. | Make sure OpenRegister is installed and enabled. | +| `NOT_CONFIGURED` | The stackiq register or its schemas cannot be found. | Run **Auto Configure** at the top of the stackiq admin settings. | +| `IMPORT_FAILED` | Something unexpected went wrong. | The Nextcloud log has the details. | + +A message that you are not signed in, not an administrator, or that your +session expired comes from Nextcloud itself: sign in again, use an +administrator account, or reload the page. + +## Adjusting the mapping + +The mapping from columns to fields is not in code. It is a set of JSON files +in `lib/Settings/cmdb-import/`, executed by OpenRegister's mapping engine: + +| File | What it maps | +|---|---| +| `topdesk-profile.json` | the sheets, the constant each sheet adds to its rows (`Beheer`) and the columns it is known to lack, the match column, the required, date and id columns, the placeholder values that mean empty, the limits, and which pack is used for which target | +| `topdesk-module.json` | a row to the module (hosting model and BBN lookups) | +| `topdesk-manufacturer.json` | "Vendor" to the Supplier organisation | +| `topdesk-municipality.json` | a typed municipality name to a new organisation | +| `topdesk-usage.json` | a row to the usage (status and TIME lookups, dates, annotation) | +| `topdesk-business-owner.json` | the owner columns | + +Each pack has a list of `fieldMappings`, one per column: `source` (the column +name in the export), `target` (the field), optionally `required`, and a +`transform` such as `trim`, `date` or a `lookup` with a `map` of export values +to stored values. For example, to accept a new "Applicatiesoort" value, add +it to the `map` of the hosting-model lookup in `topdesk-module.json`: + +```json +"Cloud": ["SaaS"] +``` + +To accept a new "Applicatie Status" value, add it to the `map` of the status +lookup in `topdesk-usage.json`. The packs are checked by OpenRegister when an import +starts; an invalid pack stops the import with `MAPPING_UNAVAILABLE` before +any row is read. A mapping file changed on the server is overwritten by the +next app update, so propose lasting changes to the app itself. diff --git a/l10n/en.js b/l10n/en.js index a2dd9df92..d125ec5cf 100644 --- a/l10n/en.js +++ b/l10n/en.js @@ -955,7 +955,121 @@ OC.L10N.register( "The currency of the costs, as a three-letter ISO 4217 code.": "The currency of the costs, as a three-letter ISO 4217 code.", "The organisation this contract or licence was bought from.": "The organisation this contract or licence was bought from.", "TOPdesk asset template id": "TOPdesk asset template id", - "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk." + "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.", + "{processed} of {total} rows processed": "{processed} of {total} rows processed", + "{size} KB": "{size} KB", + "{size} MB": "{size} MB", + "A new municipality \"{name}\" is created, unless one with this name already exists.": "A new municipality \"{name}\" is created, unless one with this name already exists.", + "A sheet has more rows than the import can process.": "A sheet has more rows than the import can process.", + "A source sheet may hold at most 10,000 rows. Split the export and import the parts one after the other.": "A source sheet may hold at most 10,000 rows. Split the export and import the parts one after the other.", + "All outcomes": "All outcomes", + "Check the connection and try again.": "Check the connection and try again.", + "Choose a municipality first.": "Choose a municipality first.", + "Choose or type a municipality": "Choose or type a municipality", + "Choose the TOPdesk export": "Choose the TOPdesk export", + "Choose the TOPdesk export and try again.": "Choose the TOPdesk export and try again.", + "CMDB import": "CMDB import", + "Created": "Created", + "Each application row of the export becomes or updates an application, its vendor, and a usage that links it to the chosen municipality. The application owner becomes a contact person of the municipality in Nextcloud Contacts; owners are never shown to the public.": "Each application row of the export becomes or updates an application, its vendor, and a usage that links it to the chosen municipality. The application owner becomes a contact person of the municipality in Nextcloud Contacts; owners are never shown to the public.", + "Error code: {code}": "Error code: {code}", + "Excel workbook (.xlsx), at most 10 MB, with the sheet \"Onbeh Applicaties CMDB\" or \"Beheerde Applicaties CMDB\".": "Excel workbook (.xlsx), at most 10 MB, with the sheet \"Onbeh Applicaties CMDB\" or \"Beheerde Applicaties CMDB\".", + "Existing municipalities could not be loaded. You can still type the name of a municipality.": "Existing municipalities could not be loaded. You can still type the name of a municipality.", + "Expected a sheet named \"{first}\" or \"{second}\". Sheet names must match exactly.": "Expected a sheet named \"{first}\" or \"{second}\". Sheet names must match exactly.", + "Failed": "Failed", + "Import": "Import", + "Import a TOPdesk CMDB export (.xlsx) as the applications one municipality uses": "Import a TOPdesk CMDB export (.xlsx) as the applications one municipality uses", + "Import finished. The municipality {name} was created. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import finished. The municipality {name} was created. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.", + "Import for {name} cancelled after {read} rows.": "Import for {name} cancelled after {read} rows.", + "Import for {name} finished. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import for {name} finished. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.", + "Import progress": "Import progress", + "Importing a newer export again updates the same applications, matched on APPID per municipality. Applications missing from it are left as they are.": "Importing a newer export again updates the same applications, matched on APPID per municipality. Applications missing from it are left as they are.", + "Importing the export…": "Importing the export…", + "Importing…": "Importing…", + "APPID": "APPID", + "Municipality": "Municipality", + "No file was uploaded.": "No file was uploaded.", + "No rows with this outcome": "No rows with this outcome", + "Nothing more is known on this page; the Nextcloud log has the details.": "Nothing more is known on this page; the Nextcloud log has the details.", + "Only Nextcloud administrators can import a CMDB export.": "Only Nextcloud administrators can import a CMDB export.", + "OpenRegister's mapping engine is missing or a mapping file is invalid. Update OpenRegister and check the Nextcloud log.": "OpenRegister's mapping engine is missing or a mapping file is invalid. Update OpenRegister and check the Nextcloud log.", + "Outcome": "Outcome", + "Pick an existing municipality or type the name of a new one.": "Pick an existing municipality or type the name of a new one.", + "Pick an existing organisation of type Municipality, or type a new name and press Enter.": "Pick an existing organisation of type Municipality, or type a new name and press Enter.", + "Pick an organisation of type Municipality, or type the name of a new one.": "Pick an organisation of type Municipality, or type the name of a new one.", + "Reasons and warnings": "Reasons and warnings", + "Records missing from the export can only be kept.": "Records missing from the export can only be kept.", + "Reload the page and try again.": "Reload the page and try again.", + "Remove sheets the import does not read, or split the export, and try again.": "Remove sheets the import does not read, or split the export, and try again.", + "Row": "Row", + "Row 1 holds the column names. \"APPID\" and \"Applicatie Naam\" are required; column order does not matter.": "Row 1 holds the column names. \"APPID\" and \"Applicatie Naam\" are required; column order does not matter.", + "Rows": "Rows", + "Rows read": "Rows read", + "Save the TOPdesk export as an Excel workbook (.xlsx). CSV, .xls and macro-enabled .xlsm files are not accepted.": "Save the TOPdesk export as an Excel workbook (.xlsx). CSV, .xls and macro-enabled .xlsm files are not accepted.", + "Sheet": "Sheet", + "Show rows with outcome": "Show rows with outcome", + "Sign in again and retry the import.": "Sign in again and retry the import.", + "Skipped": "Skipped", + "The chosen organisation is not a municipality.": "The chosen organisation is not a municipality.", + "The columns \"APPID\" and \"Applicatie Naam\" are required on every source sheet. Add the column to the export and try again. Nothing was imported.": "The columns \"APPID\" and \"Applicatie Naam\" are required on every source sheet. Add the column to the export and try again. Nothing was imported.", + "The Excel reader is not available.": "The Excel reader is not available.", + "The file": "The file", + "The file is larger than 10 MB.": "The file is larger than 10 MB.", + "The import failed unexpectedly.": "The import failed unexpectedly.", + "The import mapping cannot run.": "The import mapping cannot run.", + "The import reads workbooks with the spreadsheet library that ships with OpenRegister. Make sure OpenRegister is installed and enabled.": "The import reads workbooks with the spreadsheet library that ships with OpenRegister. Make sure OpenRegister is installed and enabled.", + "The import was cancelled. The rows processed before it stopped are kept.": "The import was cancelled. The rows processed before it stopped are kept.", + "The organisation register is not configured, so existing municipalities cannot be listed. You can still type the name of a municipality.": "The organisation register is not configured, so existing municipalities cannot be listed. You can still type the name of a municipality.", + "The server could not be reached.": "The server could not be reached.", + "The sheet \"{sheet}\" has no column \"{column}\".": "The sheet \"{sheet}\" has no column \"{column}\".", + "The sheets \"Onbeh Applicaties CMDB\" (applications without arranged maintenance) and \"Beheerde Applicaties CMDB\" (with arranged maintenance) are read; other sheets, including the \"Invoer\" sheets, are ignored.": "The sheets \"Onbeh Applicaties CMDB\" (applications without arranged maintenance) and \"Beheerde Applicaties CMDB\" (with arranged maintenance) are read; other sheets, including the \"Invoer\" sheets, are ignored.", + "The workbook has none of the sheets the import reads.": "The workbook has none of the sheets the import reads.", + "This file is not an Excel workbook (.xlsx).": "This file is not an Excel workbook (.xlsx).", + "This import is no longer running.": "This import is no longer running.", + "Unchanged": "Unchanged", + "Update existing records": "Update existing records", + "Updated": "Updated", + "Warnings": "Warnings", + "Warnings for the whole file": "Warnings for the whole file", + "When off, applications imported before are left as they are and reported as skipped.": "When off, applications imported before are left as they are and reported as skipped.", + "You are not signed in.": "You are not signed in.", + "Your session has expired.": "Your session has expired.", + "Stackiq is not configured for the import.": "Stackiq is not configured for the import.", + "The sheet \"{sheet}\" has more rows than the import can process.": "The sheet \"{sheet}\" has more rows than the import can process.", + "The stackiq register or its schemas cannot be found. Run Auto Configure at the top of this page, then try again.": "The stackiq register or its schemas cannot be found. Run Auto Configure at the top of this page, then try again.", + "Choose a municipality or enter the name of a new one.": "Choose a municipality or enter the name of a new one.", + "No running CMDB import has this id.": "No running CMDB import has this id.", + "Only keeping records that are missing from the export is supported.": "Only keeping records that are missing from the export is supported.", + "Sheet \"%1$s\" has more than %2$s rows.": "Sheet \"%1$s\" has more than %2$s rows.", + "Sheet \"%1$s\" has no column \"%2$s\".": "Sheet \"%1$s\" has no column \"%2$s\".", + "Stackiq is not configured: the register or its schemas cannot be found.": "Stackiq is not configured: the register or its schemas cannot be found.", + "The Excel reader is not available: OpenRegister is missing or incomplete.": "The Excel reader is not available: OpenRegister is missing or incomplete.", + "The file is larger than the maximum of %s MB.": "The file is larger than the maximum of %s MB.", + "The file is not an Excel workbook (.xlsx).": "The file is not an Excel workbook (.xlsx).", + "The import failed. The details are in the Nextcloud log.": "The import failed. The details are in the Nextcloud log.", + "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.": "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.", + "The workbook has neither of the sheets %s.": "The workbook has neither of the sheets %s.", + "Column \"%1$s\": %2$s": "Column \"%1$s\": %2$s", + "Optional column \"%s\" not found": "Optional column \"%s\" not found", + "Owner from column \"%s\" could not be resolved": "Owner from column \"%s\" could not be resolved", + "Owner from column \"%s\" could not be resolved in Nextcloud Contacts": "Owner from column \"%s\" could not be resolved in Nextcloud Contacts", + "Owners skipped: Nextcloud Contacts is unavailable": "Owners skipped: Nextcloud Contacts is unavailable", + "duplicate %s in file": "duplicate %s in file", + "exists": "exists", + "missing %s": "missing %s", + "step \"%1$s\" failed: %2$s": "step \"%1$s\" failed: %2$s", + "step \"%s\" failed": "step \"%s\" failed", + "Column \"%s\": formula without a cached value, read as empty": "Column \"%s\": formula without a cached value, read as empty", + "Formula cells are read as the value Excel saved with the workbook; formulas are never calculated. Save the workbook in Excel before importing it.": "Formula cells are read as the value Excel saved with the workbook; formulas are never calculated. Save the workbook in Excel before importing it.", + "Source id": "Source id", + "The code of the application in the source system it was imported from, such as the TOPdesk Applicatie Code (the Middel-ID). Shown for reference; it can change in the source, so it is not used to match records.": "The code of the application in the source system it was imported from, such as the TOPdesk Applicatie Code (the Middel-ID). Shown for reference; it can change in the source, so it is not used to match records.", + "Source number": "Source number", + "The application number in the source system, such as TOPdesk's APPID (ICT Applicatienummer). A repeated CMDB import matches on it, through the import key.": "The application number in the source system, such as TOPdesk's APPID (ICT Applicatienummer). A repeated CMDB import matches on it, through the import key.", + "Import key": "Import key", + "The key a repeated import matches this application on: topdesk::. Set by the CMDB import; do not edit.": "The key a repeated import matches this application on: topdesk::. Set by the CMDB import; do not edit.", + "Created in source": "Created in source", + "The date the application was registered in the source system.": "The date the application was registered in the source system.", + "Changed in source": "Changed in source", + "The date the application was last changed in the source system.": "The date the application was last changed in the source system." }, "nplurals=2; plural=(n != 1);" ) diff --git a/l10n/en.json b/l10n/en.json index fbee94c64..4ec00a0f9 100644 --- a/l10n/en.json +++ b/l10n/en.json @@ -954,6 +954,120 @@ "The currency of the costs, as a three-letter ISO 4217 code.": "The currency of the costs, as a three-letter ISO 4217 code.", "The organisation this contract or licence was bought from.": "The organisation this contract or licence was bought from.", "TOPdesk asset template id": "TOPdesk asset template id", - "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk." + "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.", + "{processed} of {total} rows processed": "{processed} of {total} rows processed", + "{size} KB": "{size} KB", + "{size} MB": "{size} MB", + "A new municipality \"{name}\" is created, unless one with this name already exists.": "A new municipality \"{name}\" is created, unless one with this name already exists.", + "A sheet has more rows than the import can process.": "A sheet has more rows than the import can process.", + "A source sheet may hold at most 10,000 rows. Split the export and import the parts one after the other.": "A source sheet may hold at most 10,000 rows. Split the export and import the parts one after the other.", + "All outcomes": "All outcomes", + "Check the connection and try again.": "Check the connection and try again.", + "Choose a municipality first.": "Choose a municipality first.", + "Choose or type a municipality": "Choose or type a municipality", + "Choose the TOPdesk export": "Choose the TOPdesk export", + "Choose the TOPdesk export and try again.": "Choose the TOPdesk export and try again.", + "CMDB import": "CMDB import", + "Created": "Created", + "Each application row of the export becomes or updates an application, its vendor, and a usage that links it to the chosen municipality. The application owner becomes a contact person of the municipality in Nextcloud Contacts; owners are never shown to the public.": "Each application row of the export becomes or updates an application, its vendor, and a usage that links it to the chosen municipality. The application owner becomes a contact person of the municipality in Nextcloud Contacts; owners are never shown to the public.", + "Error code: {code}": "Error code: {code}", + "Excel workbook (.xlsx), at most 10 MB, with the sheet \"Onbeh Applicaties CMDB\" or \"Beheerde Applicaties CMDB\".": "Excel workbook (.xlsx), at most 10 MB, with the sheet \"Onbeh Applicaties CMDB\" or \"Beheerde Applicaties CMDB\".", + "Existing municipalities could not be loaded. You can still type the name of a municipality.": "Existing municipalities could not be loaded. You can still type the name of a municipality.", + "Expected a sheet named \"{first}\" or \"{second}\". Sheet names must match exactly.": "Expected a sheet named \"{first}\" or \"{second}\". Sheet names must match exactly.", + "Failed": "Failed", + "Import": "Import", + "Import a TOPdesk CMDB export (.xlsx) as the applications one municipality uses": "Import a TOPdesk CMDB export (.xlsx) as the applications one municipality uses", + "Import finished. The municipality {name} was created. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import finished. The municipality {name} was created. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.", + "Import for {name} cancelled after {read} rows.": "Import for {name} cancelled after {read} rows.", + "Import for {name} finished. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import for {name} finished. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.", + "Import progress": "Import progress", + "Importing a newer export again updates the same applications, matched on APPID per municipality. Applications missing from it are left as they are.": "Importing a newer export again updates the same applications, matched on APPID per municipality. Applications missing from it are left as they are.", + "Importing the export…": "Importing the export…", + "Importing…": "Importing…", + "APPID": "APPID", + "Municipality": "Municipality", + "No file was uploaded.": "No file was uploaded.", + "No rows with this outcome": "No rows with this outcome", + "Nothing more is known on this page; the Nextcloud log has the details.": "Nothing more is known on this page; the Nextcloud log has the details.", + "Only Nextcloud administrators can import a CMDB export.": "Only Nextcloud administrators can import a CMDB export.", + "OpenRegister's mapping engine is missing or a mapping file is invalid. Update OpenRegister and check the Nextcloud log.": "OpenRegister's mapping engine is missing or a mapping file is invalid. Update OpenRegister and check the Nextcloud log.", + "Outcome": "Outcome", + "Pick an existing municipality or type the name of a new one.": "Pick an existing municipality or type the name of a new one.", + "Pick an existing organisation of type Municipality, or type a new name and press Enter.": "Pick an existing organisation of type Municipality, or type a new name and press Enter.", + "Pick an organisation of type Municipality, or type the name of a new one.": "Pick an organisation of type Municipality, or type the name of a new one.", + "Reasons and warnings": "Reasons and warnings", + "Records missing from the export can only be kept.": "Records missing from the export can only be kept.", + "Reload the page and try again.": "Reload the page and try again.", + "Remove sheets the import does not read, or split the export, and try again.": "Remove sheets the import does not read, or split the export, and try again.", + "Row": "Row", + "Row 1 holds the column names. \"APPID\" and \"Applicatie Naam\" are required; column order does not matter.": "Row 1 holds the column names. \"APPID\" and \"Applicatie Naam\" are required; column order does not matter.", + "Rows": "Rows", + "Rows read": "Rows read", + "Save the TOPdesk export as an Excel workbook (.xlsx). CSV, .xls and macro-enabled .xlsm files are not accepted.": "Save the TOPdesk export as an Excel workbook (.xlsx). CSV, .xls and macro-enabled .xlsm files are not accepted.", + "Sheet": "Sheet", + "Show rows with outcome": "Show rows with outcome", + "Sign in again and retry the import.": "Sign in again and retry the import.", + "Skipped": "Skipped", + "The chosen organisation is not a municipality.": "The chosen organisation is not a municipality.", + "The columns \"APPID\" and \"Applicatie Naam\" are required on every source sheet. Add the column to the export and try again. Nothing was imported.": "The columns \"APPID\" and \"Applicatie Naam\" are required on every source sheet. Add the column to the export and try again. Nothing was imported.", + "The Excel reader is not available.": "The Excel reader is not available.", + "The file": "The file", + "The file is larger than 10 MB.": "The file is larger than 10 MB.", + "The import failed unexpectedly.": "The import failed unexpectedly.", + "The import mapping cannot run.": "The import mapping cannot run.", + "The import reads workbooks with the spreadsheet library that ships with OpenRegister. Make sure OpenRegister is installed and enabled.": "The import reads workbooks with the spreadsheet library that ships with OpenRegister. Make sure OpenRegister is installed and enabled.", + "The import was cancelled. The rows processed before it stopped are kept.": "The import was cancelled. The rows processed before it stopped are kept.", + "The organisation register is not configured, so existing municipalities cannot be listed. You can still type the name of a municipality.": "The organisation register is not configured, so existing municipalities cannot be listed. You can still type the name of a municipality.", + "The server could not be reached.": "The server could not be reached.", + "The sheet \"{sheet}\" has no column \"{column}\".": "The sheet \"{sheet}\" has no column \"{column}\".", + "The sheets \"Onbeh Applicaties CMDB\" (applications without arranged maintenance) and \"Beheerde Applicaties CMDB\" (with arranged maintenance) are read; other sheets, including the \"Invoer\" sheets, are ignored.": "The sheets \"Onbeh Applicaties CMDB\" (applications without arranged maintenance) and \"Beheerde Applicaties CMDB\" (with arranged maintenance) are read; other sheets, including the \"Invoer\" sheets, are ignored.", + "The workbook has none of the sheets the import reads.": "The workbook has none of the sheets the import reads.", + "This file is not an Excel workbook (.xlsx).": "This file is not an Excel workbook (.xlsx).", + "This import is no longer running.": "This import is no longer running.", + "Unchanged": "Unchanged", + "Update existing records": "Update existing records", + "Updated": "Updated", + "Warnings": "Warnings", + "Warnings for the whole file": "Warnings for the whole file", + "When off, applications imported before are left as they are and reported as skipped.": "When off, applications imported before are left as they are and reported as skipped.", + "You are not signed in.": "You are not signed in.", + "Your session has expired.": "Your session has expired.", + "Stackiq is not configured for the import.": "Stackiq is not configured for the import.", + "The sheet \"{sheet}\" has more rows than the import can process.": "The sheet \"{sheet}\" has more rows than the import can process.", + "The stackiq register or its schemas cannot be found. Run Auto Configure at the top of this page, then try again.": "The stackiq register or its schemas cannot be found. Run Auto Configure at the top of this page, then try again.", + "Choose a municipality or enter the name of a new one.": "Choose a municipality or enter the name of a new one.", + "No running CMDB import has this id.": "No running CMDB import has this id.", + "Only keeping records that are missing from the export is supported.": "Only keeping records that are missing from the export is supported.", + "Sheet \"%1$s\" has more than %2$s rows.": "Sheet \"%1$s\" has more than %2$s rows.", + "Sheet \"%1$s\" has no column \"%2$s\".": "Sheet \"%1$s\" has no column \"%2$s\".", + "Stackiq is not configured: the register or its schemas cannot be found.": "Stackiq is not configured: the register or its schemas cannot be found.", + "The Excel reader is not available: OpenRegister is missing or incomplete.": "The Excel reader is not available: OpenRegister is missing or incomplete.", + "The file is larger than the maximum of %s MB.": "The file is larger than the maximum of %s MB.", + "The file is not an Excel workbook (.xlsx).": "The file is not an Excel workbook (.xlsx).", + "The import failed. The details are in the Nextcloud log.": "The import failed. The details are in the Nextcloud log.", + "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.": "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.", + "The workbook has neither of the sheets %s.": "The workbook has neither of the sheets %s.", + "Column \"%1$s\": %2$s": "Column \"%1$s\": %2$s", + "Optional column \"%s\" not found": "Optional column \"%s\" not found", + "Owner from column \"%s\" could not be resolved": "Owner from column \"%s\" could not be resolved", + "Owner from column \"%s\" could not be resolved in Nextcloud Contacts": "Owner from column \"%s\" could not be resolved in Nextcloud Contacts", + "Owners skipped: Nextcloud Contacts is unavailable": "Owners skipped: Nextcloud Contacts is unavailable", + "duplicate %s in file": "duplicate %s in file", + "exists": "exists", + "missing %s": "missing %s", + "step \"%1$s\" failed: %2$s": "step \"%1$s\" failed: %2$s", + "step \"%s\" failed": "step \"%s\" failed", + "Column \"%s\": formula without a cached value, read as empty": "Column \"%s\": formula without a cached value, read as empty", + "Formula cells are read as the value Excel saved with the workbook; formulas are never calculated. Save the workbook in Excel before importing it.": "Formula cells are read as the value Excel saved with the workbook; formulas are never calculated. Save the workbook in Excel before importing it.", + "Source id": "Source id", + "The code of the application in the source system it was imported from, such as the TOPdesk Applicatie Code (the Middel-ID). Shown for reference; it can change in the source, so it is not used to match records.": "The code of the application in the source system it was imported from, such as the TOPdesk Applicatie Code (the Middel-ID). Shown for reference; it can change in the source, so it is not used to match records.", + "Source number": "Source number", + "The application number in the source system, such as TOPdesk's APPID (ICT Applicatienummer). A repeated CMDB import matches on it, through the import key.": "The application number in the source system, such as TOPdesk's APPID (ICT Applicatienummer). A repeated CMDB import matches on it, through the import key.", + "Import key": "Import key", + "The key a repeated import matches this application on: topdesk::. Set by the CMDB import; do not edit.": "The key a repeated import matches this application on: topdesk::. Set by the CMDB import; do not edit.", + "Created in source": "Created in source", + "The date the application was registered in the source system.": "The date the application was registered in the source system.", + "Changed in source": "Changed in source", + "The date the application was last changed in the source system.": "The date the application was last changed in the source system." } } diff --git a/l10n/nl.js b/l10n/nl.js index ea47b67c6..563c72e95 100644 --- a/l10n/nl.js +++ b/l10n/nl.js @@ -1025,7 +1025,121 @@ OC.L10N.register( "The currency of the costs, as a three-letter ISO 4217 code.": "De valuta van de kosten, als ISO 4217-code van drie letters.", "The organisation this contract or licence was bought from.": "De organisatie waarvan dit contract of deze licentie is gekocht.", "TOPdesk asset template id": "Id van het TOPdesk-assetsjabloon", - "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk heeft een sjabloon nodig om een asset aan te maken. Kopieer het id van je sjabloon Applicatie uit TOPdesk." + "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk heeft een sjabloon nodig om een asset aan te maken. Kopieer het id van je sjabloon Applicatie uit TOPdesk.", + "{processed} of {total} rows processed": "{processed} van {total} rijen verwerkt", + "{size} KB": "{size} kB", + "{size} MB": "{size} MB", + "A new municipality \"{name}\" is created, unless one with this name already exists.": "Er wordt een nieuwe gemeente \"{name}\" aangemaakt, tenzij er al een gemeente met deze naam bestaat.", + "A sheet has more rows than the import can process.": "Een tabblad heeft meer rijen dan de import kan verwerken.", + "A source sheet may hold at most 10,000 rows. Split the export and import the parts one after the other.": "Een brontabblad mag hoogstens 10.000 rijen bevatten. Splits de export en importeer de delen na elkaar.", + "All outcomes": "Alle resultaten", + "Check the connection and try again.": "Controleer de verbinding en probeer het opnieuw.", + "Choose a municipality first.": "Kies eerst een gemeente.", + "Choose or type a municipality": "Kies of typ een gemeente", + "Choose the TOPdesk export": "Kies de TOPdesk-export", + "Choose the TOPdesk export and try again.": "Kies de TOPdesk-export en probeer het opnieuw.", + "CMDB import": "CMDB-import", + "Created": "Aangemaakt", + "Each application row of the export becomes or updates an application, its vendor, and a usage that links it to the chosen municipality. The application owner becomes a contact person of the municipality in Nextcloud Contacts; owners are never shown to the public.": "Elke applicatierij uit de export wordt een applicatie (of werkt die bij), met de leverancier van de software (Vendor) en een gebruik dat de applicatie aan de gekozen gemeente koppelt. De applicatie-eigenaar wordt een contactpersoon van de gemeente in Nextcloud Contacten; eigenaren worden nooit openbaar getoond.", + "Error code: {code}": "Foutcode: {code}", + "Excel workbook (.xlsx), at most 10 MB, with the sheet \"Onbeh Applicaties CMDB\" or \"Beheerde Applicaties CMDB\".": "Excel-werkmap (.xlsx), hoogstens 10 MB, met het tabblad \"Onbeh Applicaties CMDB\" of \"Beheerde Applicaties CMDB\".", + "Existing municipalities could not be loaded. You can still type the name of a municipality.": "Bestaande gemeenten konden niet worden geladen. U kunt nog steeds de naam van een gemeente typen.", + "Expected a sheet named \"{first}\" or \"{second}\". Sheet names must match exactly.": "Verwacht werd een tabblad met de naam \"{first}\" of \"{second}\". De naam van het tabblad moet precies overeenkomen.", + "Failed": "Mislukt", + "Import": "Importeren", + "Import a TOPdesk CMDB export (.xlsx) as the applications one municipality uses": "Importeer een TOPdesk CMDB-export (.xlsx) als de applicaties die één gemeente gebruikt", + "Import finished. The municipality {name} was created. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import voltooid. De gemeente {name} is aangemaakt. {read} rijen gelezen: {created} aangemaakt, {updated} bijgewerkt, {unchanged} ongewijzigd.", + "Import for {name} cancelled after {read} rows.": "Import voor {name} geannuleerd na {read} rijen.", + "Import for {name} finished. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import voor {name} voltooid. {read} rijen gelezen: {created} aangemaakt, {updated} bijgewerkt, {unchanged} ongewijzigd.", + "Import progress": "Voortgang van de import", + "Importing a newer export again updates the same applications, matched on APPID per municipality. Applications missing from it are left as they are.": "Een nieuwere export opnieuw importeren werkt dezelfde applicaties bij, herkend aan het APPID per gemeente. Applicaties die er niet meer in staan, blijven zoals ze zijn.", + "Importing the export…": "De export wordt geïmporteerd…", + "Importing…": "Importeren…", + "APPID": "APPID", + "Municipality": "Gemeente", + "No file was uploaded.": "Er is geen bestand geüpload.", + "No rows with this outcome": "Geen rijen met dit resultaat", + "Nothing more is known on this page; the Nextcloud log has the details.": "Op deze pagina is niet meer bekend; het Nextcloud-logboek bevat de details.", + "Only Nextcloud administrators can import a CMDB export.": "Alleen Nextcloud-beheerders kunnen een CMDB-export importeren.", + "OpenRegister's mapping engine is missing or a mapping file is invalid. Update OpenRegister and check the Nextcloud log.": "De mapping-engine van OpenRegister ontbreekt of een mappingbestand is ongeldig. Werk OpenRegister bij en bekijk het Nextcloud-logboek.", + "Outcome": "Resultaat", + "Pick an existing municipality or type the name of a new one.": "Kies een bestaande gemeente of typ de naam van een nieuwe.", + "Pick an existing organisation of type Municipality, or type a new name and press Enter.": "Kies een bestaande organisatie van het type Gemeente, of typ een nieuwe naam en druk op Enter.", + "Pick an organisation of type Municipality, or type the name of a new one.": "Kies een organisatie van het type Gemeente, of typ de naam van een nieuwe.", + "Reasons and warnings": "Redenen en waarschuwingen", + "Records missing from the export can only be kept.": "Records die in de export ontbreken, kunnen alleen worden behouden.", + "Reload the page and try again.": "Laad de pagina opnieuw en probeer het nog eens.", + "Remove sheets the import does not read, or split the export, and try again.": "Verwijder tabbladen die de import niet leest, of splits de export, en probeer het opnieuw.", + "Row": "Rij", + "Row 1 holds the column names. \"APPID\" and \"Applicatie Naam\" are required; column order does not matter.": "Rij 1 bevat de kolomnamen. \"APPID\" en \"Applicatie Naam\" zijn verplicht; de volgorde van de kolommen maakt niet uit.", + "Rows": "Rijen", + "Rows read": "Rijen gelezen", + "Save the TOPdesk export as an Excel workbook (.xlsx). CSV, .xls and macro-enabled .xlsm files are not accepted.": "Sla de TOPdesk-export op als Excel-werkmap (.xlsx). CSV-, .xls- en .xlsm-bestanden met macro's worden niet geaccepteerd.", + "Sheet": "Tabblad", + "Show rows with outcome": "Toon rijen met resultaat", + "Sign in again and retry the import.": "Meld u opnieuw aan en probeer de import nog eens.", + "Skipped": "Overgeslagen", + "The chosen organisation is not a municipality.": "De gekozen organisatie is geen gemeente.", + "The columns \"APPID\" and \"Applicatie Naam\" are required on every source sheet. Add the column to the export and try again. Nothing was imported.": "De kolommen \"APPID\" en \"Applicatie Naam\" zijn op elk brontabblad verplicht. Voeg de kolom toe aan de export en probeer het opnieuw. Er is niets geïmporteerd.", + "The Excel reader is not available.": "De Excel-lezer is niet beschikbaar.", + "The file": "Het bestand", + "The file is larger than 10 MB.": "Het bestand is groter dan 10 MB.", + "The import failed unexpectedly.": "De import is onverwacht mislukt.", + "The import mapping cannot run.": "De mapping van de import kan niet worden uitgevoerd.", + "The import reads workbooks with the spreadsheet library that ships with OpenRegister. Make sure OpenRegister is installed and enabled.": "De import leest werkmappen met de spreadsheetbibliotheek die met OpenRegister wordt meegeleverd. Zorg dat OpenRegister is geïnstalleerd en ingeschakeld.", + "The import was cancelled. The rows processed before it stopped are kept.": "De import is geannuleerd. De rijen die vóór het stoppen zijn verwerkt, blijven behouden.", + "The organisation register is not configured, so existing municipalities cannot be listed. You can still type the name of a municipality.": "Het organisatieregister is niet ingesteld, dus bestaande gemeenten kunnen niet worden getoond. U kunt nog steeds de naam van een gemeente typen.", + "The server could not be reached.": "De server is niet bereikbaar.", + "The sheet \"{sheet}\" has no column \"{column}\".": "Het tabblad \"{sheet}\" heeft geen kolom \"{column}\".", + "The sheets \"Onbeh Applicaties CMDB\" (applications without arranged maintenance) and \"Beheerde Applicaties CMDB\" (with arranged maintenance) are read; other sheets, including the \"Invoer\" sheets, are ignored.": "De tabbladen \"Onbeh Applicaties CMDB\" (applicaties zonder geregeld beheer) en \"Beheerde Applicaties CMDB\" (met geregeld beheer) worden gelezen; andere tabbladen, ook de \"Invoer\"-tabbladen, worden genegeerd.", + "The workbook has none of the sheets the import reads.": "De werkmap bevat geen van de tabbladen die de import leest.", + "This file is not an Excel workbook (.xlsx).": "Dit bestand is geen Excel-werkmap (.xlsx).", + "This import is no longer running.": "Deze import loopt niet meer.", + "Unchanged": "Ongewijzigd", + "Update existing records": "Bestaande records bijwerken", + "Updated": "Bijgewerkt", + "Warnings": "Waarschuwingen", + "Warnings for the whole file": "Waarschuwingen voor het hele bestand", + "When off, applications imported before are left as they are and reported as skipped.": "Als dit uit staat, blijven eerder geïmporteerde applicaties zoals ze zijn en worden ze als overgeslagen gemeld.", + "You are not signed in.": "U bent niet aangemeld.", + "Your session has expired.": "Uw sessie is verlopen.", + "Stackiq is not configured for the import.": "Stackiq is niet ingesteld voor de import.", + "The sheet \"{sheet}\" has more rows than the import can process.": "Het tabblad \"{sheet}\" heeft meer rijen dan de import kan verwerken.", + "The stackiq register or its schemas cannot be found. Run Auto Configure at the top of this page, then try again.": "Het stackiq-register of de schema's ervan zijn niet gevonden. Voer bovenaan deze pagina Auto Configure uit en probeer het daarna opnieuw.", + "Choose a municipality or enter the name of a new one.": "Kies een gemeente of voer de naam van een nieuwe in.", + "No running CMDB import has this id.": "Er loopt geen CMDB-import met deze id.", + "Only keeping records that are missing from the export is supported.": "Alleen het behouden van records die in de export ontbreken, wordt ondersteund.", + "Sheet \"%1$s\" has more than %2$s rows.": "Tabblad \"%1$s\" heeft meer dan %2$s rijen.", + "Sheet \"%1$s\" has no column \"%2$s\".": "Tabblad \"%1$s\" heeft geen kolom \"%2$s\".", + "Stackiq is not configured: the register or its schemas cannot be found.": "Stackiq is niet ingesteld: het register of de schema's ervan zijn niet gevonden.", + "The Excel reader is not available: OpenRegister is missing or incomplete.": "De Excel-lezer is niet beschikbaar: OpenRegister ontbreekt of is onvolledig.", + "The file is larger than the maximum of %s MB.": "Het bestand is groter dan het maximum van %s MB.", + "The file is not an Excel workbook (.xlsx).": "Het bestand is geen Excel-werkmap (.xlsx).", + "The import failed. The details are in the Nextcloud log.": "De import is mislukt. De details staan in het Nextcloud-logboek.", + "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.": "De mapping van de import kan niet worden uitgevoerd: OpenRegister ontbreekt of een mappingbestand is ongeldig.", + "The workbook has neither of the sheets %s.": "De werkmap bevat geen van de tabbladen %s.", + "Column \"%1$s\": %2$s": "Kolom \"%1$s\": %2$s", + "Optional column \"%s\" not found": "Optionele kolom \"%s\" niet gevonden", + "Owner from column \"%s\" could not be resolved": "Eigenaar uit kolom \"%s\" kon niet worden gevonden", + "Owner from column \"%s\" could not be resolved in Nextcloud Contacts": "Eigenaar uit kolom \"%s\" kon niet worden gevonden in Nextcloud Contacten", + "Owners skipped: Nextcloud Contacts is unavailable": "Eigenaren overgeslagen: Nextcloud Contacten is niet beschikbaar", + "duplicate %s in file": "dubbele %s in het bestand", + "exists": "bestaat al", + "missing %s": "%s ontbreekt", + "step \"%1$s\" failed: %2$s": "stap \"%1$s\" mislukt: %2$s", + "step \"%s\" failed": "stap \"%s\" mislukt", + "Column \"%s\": formula without a cached value, read as empty": "Kolom \"%s\": formule zonder opgeslagen waarde, gelezen als leeg", + "Formula cells are read as the value Excel saved with the workbook; formulas are never calculated. Save the workbook in Excel before importing it.": "Formulecellen worden gelezen als de waarde die Excel bij de werkmap heeft opgeslagen; formules worden nooit berekend. Sla de werkmap op in Excel voordat u hem importeert.", + "Source id": "Bron-id", + "The code of the application in the source system it was imported from, such as the TOPdesk Applicatie Code (the Middel-ID). Shown for reference; it can change in the source, so it is not used to match records.": "De code van de applicatie in het bronsysteem waaruit ze is geïmporteerd, zoals de TOPdesk Applicatie Code (het Middel-ID). Ter informatie; ze kan in de bron veranderen, dus records worden er niet op gekoppeld.", + "Source number": "Bronnummer", + "The application number in the source system, such as TOPdesk's APPID (ICT Applicatienummer). A repeated CMDB import matches on it, through the import key.": "Het applicatienummer in het bronsysteem, zoals het APPID van TOPdesk (ICT Applicatienummer). Een herhaalde CMDB-import koppelt erop, via de importsleutel.", + "Import key": "Importsleutel", + "The key a repeated import matches this application on: topdesk::. Set by the CMDB import; do not edit.": "De sleutel waarop een herhaalde import deze applicatie koppelt: topdesk::. Gezet door de CMDB-import; niet aanpassen.", + "Created in source": "Aangemaakt in de bron", + "The date the application was registered in the source system.": "De datum waarop de applicatie in het bronsysteem is geregistreerd.", + "Changed in source": "Gewijzigd in de bron", + "The date the application was last changed in the source system.": "De datum waarop de applicatie in het bronsysteem het laatst is gewijzigd." }, "nplurals=2; plural=(n != 1);" ) diff --git a/l10n/nl.json b/l10n/nl.json index f340f06b1..f26e94423 100644 --- a/l10n/nl.json +++ b/l10n/nl.json @@ -1024,6 +1024,120 @@ "The currency of the costs, as a three-letter ISO 4217 code.": "De valuta van de kosten, als ISO 4217-code van drie letters.", "The organisation this contract or licence was bought from.": "De organisatie waarvan dit contract of deze licentie is gekocht.", "TOPdesk asset template id": "Id van het TOPdesk-assetsjabloon", - "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk heeft een sjabloon nodig om een asset aan te maken. Kopieer het id van je sjabloon Applicatie uit TOPdesk." + "TOPdesk needs a template to create an asset. Copy the id of your Application template from TOPdesk.": "TOPdesk heeft een sjabloon nodig om een asset aan te maken. Kopieer het id van je sjabloon Applicatie uit TOPdesk.", + "{processed} of {total} rows processed": "{processed} van {total} rijen verwerkt", + "{size} KB": "{size} kB", + "{size} MB": "{size} MB", + "A new municipality \"{name}\" is created, unless one with this name already exists.": "Er wordt een nieuwe gemeente \"{name}\" aangemaakt, tenzij er al een gemeente met deze naam bestaat.", + "A sheet has more rows than the import can process.": "Een tabblad heeft meer rijen dan de import kan verwerken.", + "A source sheet may hold at most 10,000 rows. Split the export and import the parts one after the other.": "Een brontabblad mag hoogstens 10.000 rijen bevatten. Splits de export en importeer de delen na elkaar.", + "All outcomes": "Alle resultaten", + "Check the connection and try again.": "Controleer de verbinding en probeer het opnieuw.", + "Choose a municipality first.": "Kies eerst een gemeente.", + "Choose or type a municipality": "Kies of typ een gemeente", + "Choose the TOPdesk export": "Kies de TOPdesk-export", + "Choose the TOPdesk export and try again.": "Kies de TOPdesk-export en probeer het opnieuw.", + "CMDB import": "CMDB-import", + "Created": "Aangemaakt", + "Each application row of the export becomes or updates an application, its vendor, and a usage that links it to the chosen municipality. The application owner becomes a contact person of the municipality in Nextcloud Contacts; owners are never shown to the public.": "Elke applicatierij uit de export wordt een applicatie (of werkt die bij), met de leverancier van de software (Vendor) en een gebruik dat de applicatie aan de gekozen gemeente koppelt. De applicatie-eigenaar wordt een contactpersoon van de gemeente in Nextcloud Contacten; eigenaren worden nooit openbaar getoond.", + "Error code: {code}": "Foutcode: {code}", + "Excel workbook (.xlsx), at most 10 MB, with the sheet \"Onbeh Applicaties CMDB\" or \"Beheerde Applicaties CMDB\".": "Excel-werkmap (.xlsx), hoogstens 10 MB, met het tabblad \"Onbeh Applicaties CMDB\" of \"Beheerde Applicaties CMDB\".", + "Existing municipalities could not be loaded. You can still type the name of a municipality.": "Bestaande gemeenten konden niet worden geladen. U kunt nog steeds de naam van een gemeente typen.", + "Expected a sheet named \"{first}\" or \"{second}\". Sheet names must match exactly.": "Verwacht werd een tabblad met de naam \"{first}\" of \"{second}\". De naam van het tabblad moet precies overeenkomen.", + "Failed": "Mislukt", + "Import": "Importeren", + "Import a TOPdesk CMDB export (.xlsx) as the applications one municipality uses": "Importeer een TOPdesk CMDB-export (.xlsx) als de applicaties die één gemeente gebruikt", + "Import finished. The municipality {name} was created. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import voltooid. De gemeente {name} is aangemaakt. {read} rijen gelezen: {created} aangemaakt, {updated} bijgewerkt, {unchanged} ongewijzigd.", + "Import for {name} cancelled after {read} rows.": "Import voor {name} geannuleerd na {read} rijen.", + "Import for {name} finished. {read} rows read: {created} created, {updated} updated, {unchanged} unchanged.": "Import voor {name} voltooid. {read} rijen gelezen: {created} aangemaakt, {updated} bijgewerkt, {unchanged} ongewijzigd.", + "Import progress": "Voortgang van de import", + "Importing a newer export again updates the same applications, matched on APPID per municipality. Applications missing from it are left as they are.": "Een nieuwere export opnieuw importeren werkt dezelfde applicaties bij, herkend aan het APPID per gemeente. Applicaties die er niet meer in staan, blijven zoals ze zijn.", + "Importing the export…": "De export wordt geïmporteerd…", + "Importing…": "Importeren…", + "APPID": "APPID", + "Municipality": "Gemeente", + "No file was uploaded.": "Er is geen bestand geüpload.", + "No rows with this outcome": "Geen rijen met dit resultaat", + "Nothing more is known on this page; the Nextcloud log has the details.": "Op deze pagina is niet meer bekend; het Nextcloud-logboek bevat de details.", + "Only Nextcloud administrators can import a CMDB export.": "Alleen Nextcloud-beheerders kunnen een CMDB-export importeren.", + "OpenRegister's mapping engine is missing or a mapping file is invalid. Update OpenRegister and check the Nextcloud log.": "De mapping-engine van OpenRegister ontbreekt of een mappingbestand is ongeldig. Werk OpenRegister bij en bekijk het Nextcloud-logboek.", + "Outcome": "Resultaat", + "Pick an existing municipality or type the name of a new one.": "Kies een bestaande gemeente of typ de naam van een nieuwe.", + "Pick an existing organisation of type Municipality, or type a new name and press Enter.": "Kies een bestaande organisatie van het type Gemeente, of typ een nieuwe naam en druk op Enter.", + "Pick an organisation of type Municipality, or type the name of a new one.": "Kies een organisatie van het type Gemeente, of typ de naam van een nieuwe.", + "Reasons and warnings": "Redenen en waarschuwingen", + "Records missing from the export can only be kept.": "Records die in de export ontbreken, kunnen alleen worden behouden.", + "Reload the page and try again.": "Laad de pagina opnieuw en probeer het nog eens.", + "Remove sheets the import does not read, or split the export, and try again.": "Verwijder tabbladen die de import niet leest, of splits de export, en probeer het opnieuw.", + "Row": "Rij", + "Row 1 holds the column names. \"APPID\" and \"Applicatie Naam\" are required; column order does not matter.": "Rij 1 bevat de kolomnamen. \"APPID\" en \"Applicatie Naam\" zijn verplicht; de volgorde van de kolommen maakt niet uit.", + "Rows": "Rijen", + "Rows read": "Rijen gelezen", + "Save the TOPdesk export as an Excel workbook (.xlsx). CSV, .xls and macro-enabled .xlsm files are not accepted.": "Sla de TOPdesk-export op als Excel-werkmap (.xlsx). CSV-, .xls- en .xlsm-bestanden met macro's worden niet geaccepteerd.", + "Sheet": "Tabblad", + "Show rows with outcome": "Toon rijen met resultaat", + "Sign in again and retry the import.": "Meld u opnieuw aan en probeer de import nog eens.", + "Skipped": "Overgeslagen", + "The chosen organisation is not a municipality.": "De gekozen organisatie is geen gemeente.", + "The columns \"APPID\" and \"Applicatie Naam\" are required on every source sheet. Add the column to the export and try again. Nothing was imported.": "De kolommen \"APPID\" en \"Applicatie Naam\" zijn op elk brontabblad verplicht. Voeg de kolom toe aan de export en probeer het opnieuw. Er is niets geïmporteerd.", + "The Excel reader is not available.": "De Excel-lezer is niet beschikbaar.", + "The file": "Het bestand", + "The file is larger than 10 MB.": "Het bestand is groter dan 10 MB.", + "The import failed unexpectedly.": "De import is onverwacht mislukt.", + "The import mapping cannot run.": "De mapping van de import kan niet worden uitgevoerd.", + "The import reads workbooks with the spreadsheet library that ships with OpenRegister. Make sure OpenRegister is installed and enabled.": "De import leest werkmappen met de spreadsheetbibliotheek die met OpenRegister wordt meegeleverd. Zorg dat OpenRegister is geïnstalleerd en ingeschakeld.", + "The import was cancelled. The rows processed before it stopped are kept.": "De import is geannuleerd. De rijen die vóór het stoppen zijn verwerkt, blijven behouden.", + "The organisation register is not configured, so existing municipalities cannot be listed. You can still type the name of a municipality.": "Het organisatieregister is niet ingesteld, dus bestaande gemeenten kunnen niet worden getoond. U kunt nog steeds de naam van een gemeente typen.", + "The server could not be reached.": "De server is niet bereikbaar.", + "The sheet \"{sheet}\" has no column \"{column}\".": "Het tabblad \"{sheet}\" heeft geen kolom \"{column}\".", + "The sheets \"Onbeh Applicaties CMDB\" (applications without arranged maintenance) and \"Beheerde Applicaties CMDB\" (with arranged maintenance) are read; other sheets, including the \"Invoer\" sheets, are ignored.": "De tabbladen \"Onbeh Applicaties CMDB\" (applicaties zonder geregeld beheer) en \"Beheerde Applicaties CMDB\" (met geregeld beheer) worden gelezen; andere tabbladen, ook de \"Invoer\"-tabbladen, worden genegeerd.", + "The workbook has none of the sheets the import reads.": "De werkmap bevat geen van de tabbladen die de import leest.", + "This file is not an Excel workbook (.xlsx).": "Dit bestand is geen Excel-werkmap (.xlsx).", + "This import is no longer running.": "Deze import loopt niet meer.", + "Unchanged": "Ongewijzigd", + "Update existing records": "Bestaande records bijwerken", + "Updated": "Bijgewerkt", + "Warnings": "Waarschuwingen", + "Warnings for the whole file": "Waarschuwingen voor het hele bestand", + "When off, applications imported before are left as they are and reported as skipped.": "Als dit uit staat, blijven eerder geïmporteerde applicaties zoals ze zijn en worden ze als overgeslagen gemeld.", + "You are not signed in.": "U bent niet aangemeld.", + "Your session has expired.": "Uw sessie is verlopen.", + "Stackiq is not configured for the import.": "Stackiq is niet ingesteld voor de import.", + "The sheet \"{sheet}\" has more rows than the import can process.": "Het tabblad \"{sheet}\" heeft meer rijen dan de import kan verwerken.", + "The stackiq register or its schemas cannot be found. Run Auto Configure at the top of this page, then try again.": "Het stackiq-register of de schema's ervan zijn niet gevonden. Voer bovenaan deze pagina Auto Configure uit en probeer het daarna opnieuw.", + "Choose a municipality or enter the name of a new one.": "Kies een gemeente of voer de naam van een nieuwe in.", + "No running CMDB import has this id.": "Er loopt geen CMDB-import met deze id.", + "Only keeping records that are missing from the export is supported.": "Alleen het behouden van records die in de export ontbreken, wordt ondersteund.", + "Sheet \"%1$s\" has more than %2$s rows.": "Tabblad \"%1$s\" heeft meer dan %2$s rijen.", + "Sheet \"%1$s\" has no column \"%2$s\".": "Tabblad \"%1$s\" heeft geen kolom \"%2$s\".", + "Stackiq is not configured: the register or its schemas cannot be found.": "Stackiq is niet ingesteld: het register of de schema's ervan zijn niet gevonden.", + "The Excel reader is not available: OpenRegister is missing or incomplete.": "De Excel-lezer is niet beschikbaar: OpenRegister ontbreekt of is onvolledig.", + "The file is larger than the maximum of %s MB.": "Het bestand is groter dan het maximum van %s MB.", + "The file is not an Excel workbook (.xlsx).": "Het bestand is geen Excel-werkmap (.xlsx).", + "The import failed. The details are in the Nextcloud log.": "De import is mislukt. De details staan in het Nextcloud-logboek.", + "The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.": "De mapping van de import kan niet worden uitgevoerd: OpenRegister ontbreekt of een mappingbestand is ongeldig.", + "The workbook has neither of the sheets %s.": "De werkmap bevat geen van de tabbladen %s.", + "Column \"%1$s\": %2$s": "Kolom \"%1$s\": %2$s", + "Optional column \"%s\" not found": "Optionele kolom \"%s\" niet gevonden", + "Owner from column \"%s\" could not be resolved": "Eigenaar uit kolom \"%s\" kon niet worden gevonden", + "Owner from column \"%s\" could not be resolved in Nextcloud Contacts": "Eigenaar uit kolom \"%s\" kon niet worden gevonden in Nextcloud Contacten", + "Owners skipped: Nextcloud Contacts is unavailable": "Eigenaren overgeslagen: Nextcloud Contacten is niet beschikbaar", + "duplicate %s in file": "dubbele %s in het bestand", + "exists": "bestaat al", + "missing %s": "%s ontbreekt", + "step \"%1$s\" failed: %2$s": "stap \"%1$s\" mislukt: %2$s", + "step \"%s\" failed": "stap \"%s\" mislukt", + "Column \"%s\": formula without a cached value, read as empty": "Kolom \"%s\": formule zonder opgeslagen waarde, gelezen als leeg", + "Formula cells are read as the value Excel saved with the workbook; formulas are never calculated. Save the workbook in Excel before importing it.": "Formulecellen worden gelezen als de waarde die Excel bij de werkmap heeft opgeslagen; formules worden nooit berekend. Sla de werkmap op in Excel voordat u hem importeert.", + "Source id": "Bron-id", + "The code of the application in the source system it was imported from, such as the TOPdesk Applicatie Code (the Middel-ID). Shown for reference; it can change in the source, so it is not used to match records.": "De code van de applicatie in het bronsysteem waaruit ze is geïmporteerd, zoals de TOPdesk Applicatie Code (het Middel-ID). Ter informatie; ze kan in de bron veranderen, dus records worden er niet op gekoppeld.", + "Source number": "Bronnummer", + "The application number in the source system, such as TOPdesk's APPID (ICT Applicatienummer). A repeated CMDB import matches on it, through the import key.": "Het applicatienummer in het bronsysteem, zoals het APPID van TOPdesk (ICT Applicatienummer). Een herhaalde CMDB-import koppelt erop, via de importsleutel.", + "Import key": "Importsleutel", + "The key a repeated import matches this application on: topdesk::. Set by the CMDB import; do not edit.": "De sleutel waarop een herhaalde import deze applicatie koppelt: topdesk::. Gezet door de CMDB-import; niet aanpassen.", + "Created in source": "Aangemaakt in de bron", + "The date the application was registered in the source system.": "De datum waarop de applicatie in het bronsysteem is geregistreerd.", + "Changed in source": "Gewijzigd in de bron", + "The date the application was last changed in the source system.": "De datum waarop de applicatie in het bronsysteem het laatst is gewijzigd." } } diff --git a/lib/Controller/CmdbImportController.php b/lib/Controller/CmdbImportController.php new file mode 100644 index 000000000..821deaece --- /dev/null +++ b/lib/Controller/CmdbImportController.php @@ -0,0 +1,292 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Controller; + +use OCA\Stackiq\AppInfo\Application; +use OCA\Stackiq\Exception\CmdbImportException; +use OCA\Stackiq\Service\CmdbExportImportService; +use OCP\AppFramework\Controller; +use OCP\AppFramework\Http; +use OCP\AppFramework\Http\JSONResponse; +use OCP\IL10N; +use OCP\IRequest; +use Psr\Log\LoggerInterface; + +/** + * CMDB import and cancel, admin-only and CSRF-protected. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ +class CmdbImportController extends Controller { + /** + * The multipart field of the export. + */ + public const FILE_FIELD = 'cmdbFile'; + + /** + * Constructor. + * + * @param IRequest $request The request. + * @param CmdbExportImportService $importService The import service. + * @param IL10N $l10n Translations of the error messages. + * @param LoggerInterface $logger Logger. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + public function __construct( + IRequest $request, + private readonly CmdbExportImportService $importService, + private readonly IL10N $l10n, + private readonly LoggerInterface $logger, + ) { + parent::__construct(appName: Application::APP_ID, request: $request); + }//end __construct() + + /** + * Import a TOPdesk CMDB export for one municipality. + * + * Multipart fields: `cmdbFile`, `municipalityUuid` or `municipalityName`, + * `updateExisting` (default true), `missingRecords` (only `keep`) and + * `operationId` (pattern `cmdb-` plus 8 to 64 letters, digits or hyphens). + * + * @return JSONResponse The report (200), or an error envelope with the contract code. + * + * @auth admin-only importing a CMDB export rewrites the catalogue of a whole municipality, so only a Nextcloud admin runs it. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + public function import(): JSONResponse { + $validated = $this->validateRequest(); + if ($validated instanceof JSONResponse) { + return $validated; + } + + try { + $report = $this->importService->import(path: $validated['path'], options: $validated['options']); + } catch (CmdbImportException $e) { + $this->logger->info( + 'CmdbImportController: import refused', + ['error' => $e->getErrorCode(), 'details' => $e->getDetails(), 'reason' => $e->getMessage()] + ); + return $this->fromException(e: $e); + } catch (\Exception $e) { + $this->logger->error('CmdbImportController: import failed', ['exception' => $e]); + return $this->error(code: 'IMPORT_FAILED', status: Http::STATUS_INTERNAL_SERVER_ERROR); + } + + return new JSONResponse(data: $report, statusCode: Http::STATUS_OK); + }//end import() + + /** + * Check the request in the order of design D10, before anything is parsed. + * + * Present, size, xlsx, `missingRecords`, municipality. + * + * @return array{path: string, options: array}|JSONResponse The import input, or the first error. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + private function validateRequest(): array|JSONResponse { + $upload = $this->uploadedFile(); + if ($upload === null) { + return $this->error(code: 'NO_FILE_UPLOADED', status: Http::STATUS_BAD_REQUEST); + } + + $maxBytes = $this->importService->maxFileBytes(); + if ($upload['tooLarge'] === true || $upload['size'] > $maxBytes) { + return $this->error(code: 'FILE_TOO_LARGE', status: Http::STATUS_REQUEST_ENTITY_TOO_LARGE, details: ['maxBytes' => $maxBytes]); + } + + try { + $this->importService->assertXlsx(path: $upload['tmpName'], fileName: $upload['name']); + } catch (CmdbImportException $e) { + return $this->fromException(e: $e); + } + + $missingRecords = (string)$this->request->getParam('missingRecords', 'keep'); + if ($this->importService->supportsMissingRecords(mode: $missingRecords) === false) { + return $this->error(code: 'MISSING_RECORDS_UNSUPPORTED', status: Http::STATUS_UNPROCESSABLE_ENTITY, details: ['accepted' => ['keep']]); + } + + $municipalityUuid = trim((string)$this->request->getParam('municipalityUuid', '')); + $municipalityName = trim((string)$this->request->getParam('municipalityName', '')); + if ($municipalityUuid === '' && $municipalityName === '') { + return $this->error(code: CmdbImportException::MUNICIPALITY_REQUIRED, status: Http::STATUS_UNPROCESSABLE_ENTITY); + } + + return [ + 'path' => $upload['tmpName'], + 'options' => [ + 'municipalityUuid' => $municipalityUuid, + 'municipalityName' => $municipalityName, + 'updateExisting' => $this->booleanParam(name: 'updateExisting', default: true), + 'operationId' => $this->request->getParam('operationId'), + ], + ]; + }//end validateRequest() + + /** + * Ask a running CMDB import to stop between rows. + * + * @param string $operationId The operation id. + * + * @return JSONResponse `{success, cancelRequested}`, or 404 OPERATION_NOT_FOUND. + * + * @auth admin-only cancelling an import is part of running it, so only a Nextcloud admin may do it (no NoAdminRequired, CSRF checked). + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function cancel(string $operationId): JSONResponse { + if ($this->importService->requestCancel(operationId: $operationId) === false) { + return $this->error(code: 'OPERATION_NOT_FOUND', status: Http::STATUS_NOT_FOUND); + } + + return new JSONResponse(data: ['success' => true, 'cancelRequested' => true], statusCode: Http::STATUS_OK); + }//end cancel() + + /** + * Translate a CmdbImportException into its contract response. + * + * @param CmdbImportException $e The exception. + * + * @return JSONResponse + */ + private function fromException(CmdbImportException $e): JSONResponse { + return $this->error(code: $e->getErrorCode(), status: $e->getHttpStatus(), details: $e->getDetails()); + }//end fromException() + + /** + * The error envelope of contract.md. + * + * @param string $code The machine error code. + * @param int $status The HTTP status. + * @param array $details Details, e.g. sheet and column. + * + * @return JSONResponse + */ + private function error(string $code, int $status, array $details = []): JSONResponse { + return new JSONResponse( + data: [ + 'success' => false, + 'error' => $code, + 'message' => $this->message(code: $code, details: $details), + 'details' => (object)$details, + ], + statusCode: $status + ); + }//end error() + + /** + * The translated message of an error code. + * + * @param string $code The machine error code. + * @param array $details The details. + * + * @return string + * + * @SuppressWarnings(PHPMD.CyclomaticComplexity) One branch per contract error code. + */ + private function message(string $code, array $details): string { + $megabytes = (string)intdiv($this->importService->maxFileBytes(), 1048576); + $expected = implode(', ', array_map('strval', ($details['expected'] ?? []))); + + return match ($code) { + 'NO_FILE_UPLOADED' => $this->l10n->t('No file was uploaded.'), + 'NOT_XLSX' => $this->l10n->t('The file is not an Excel workbook (.xlsx).'), + 'FILE_TOO_LARGE' => $this->l10n->t('The file is larger than the maximum of %s MB.', [$megabytes]), + 'MISSING_RECORDS_UNSUPPORTED' => $this->l10n->t('Only keeping records that are missing from the export is supported.'), + 'MUNICIPALITY_REQUIRED' => $this->l10n->t('Choose a municipality or enter the name of a new one.'), + 'MUNICIPALITY_INVALID' => $this->l10n->t('The chosen organisation is not a municipality.'), + 'NO_SOURCE_SHEET' => $this->l10n->t('The workbook has neither of the sheets %s.', [$expected]), + 'MISSING_COLUMN' => $this->l10n->t('Sheet "%1$s" has no column "%2$s".', [(string)($details['sheet'] ?? ''), (string)($details['column'] ?? '')]), + 'TOO_MANY_ROWS' => $this->l10n->t('Sheet "%1$s" has more than %2$s rows.', [(string)($details['sheet'] ?? ''), (string)($details['limit'] ?? '')]), + 'MAPPING_UNAVAILABLE' => $this->l10n->t('The import mapping cannot run: OpenRegister is missing or a mapping file is invalid.'), + 'READER_UNAVAILABLE' => $this->l10n->t('The Excel reader is not available: OpenRegister is missing or incomplete.'), + 'NOT_CONFIGURED' => $this->l10n->t('Stackiq is not configured: the register or its schemas cannot be found.'), + 'OPERATION_NOT_FOUND' => $this->l10n->t('No running CMDB import has this id.'), + default => $this->l10n->t('The import failed. The details are in the Nextcloud log.'), + }; + }//end message() + + /** + * A boolean form field (`true`/`false`, `1`/`0`). + * + * @param string $name The field. + * @param bool $default The value when absent. + * + * @return bool + */ + private function booleanParam(string $name, bool $default): bool { + $value = $this->request->getParam($name); + if ($value === null || $value === '') { + return $default; + } + + if (is_bool($value) === true) { + return $value; + } + + return in_array(strtolower((string)$value), ['false', '0', 'no', 'off'], true) === false; + }//end booleanParam() + + /** + * The uploaded export, or null when none was sent. + * + * @return array{tmpName: string, name: string, size: int, tooLarge: bool}|null + */ + private function uploadedFile(): ?array { + $file = $this->request->getUploadedFile(self::FILE_FIELD); + if (is_array($file) === false || $file === []) { + return null; + } + + $error = (int)($file['error'] ?? UPLOAD_ERR_OK); + if ($error === UPLOAD_ERR_INI_SIZE || $error === UPLOAD_ERR_FORM_SIZE) { + return ['tmpName' => '', 'name' => (string)($file['name'] ?? ''), 'size' => 0, 'tooLarge' => true]; + } + + $tmpName = (string)($file['tmp_name'] ?? ''); + if ($error !== UPLOAD_ERR_OK || $tmpName === '') { + return null; + } + + $size = (int)($file['size'] ?? 0); + if ($size === 0 && is_file($tmpName) === true) { + $size = (int)filesize($tmpName); + } + + return ['tmpName' => $tmpName, 'name' => (string)($file['name'] ?? ''), 'size' => $size, 'tooLarge' => false]; + }//end uploadedFile() +}//end class diff --git a/lib/Exception/CmdbImportException.php b/lib/Exception/CmdbImportException.php new file mode 100644 index 000000000..4e96756e9 --- /dev/null +++ b/lib/Exception/CmdbImportException.php @@ -0,0 +1,118 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Exception; + +use RuntimeException; +use Throwable; + +/** + * A CMDB import failure with a contract error code and an HTTP status. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ +class CmdbImportException extends RuntimeException { + public const NOT_XLSX = 'NOT_XLSX'; + public const NO_SOURCE_SHEET = 'NO_SOURCE_SHEET'; + public const MISSING_COLUMN = 'MISSING_COLUMN'; + public const TOO_MANY_ROWS = 'TOO_MANY_ROWS'; + public const MUNICIPALITY_REQUIRED = 'MUNICIPALITY_REQUIRED'; + public const MUNICIPALITY_INVALID = 'MUNICIPALITY_INVALID'; + public const MAPPING_UNAVAILABLE = 'MAPPING_UNAVAILABLE'; + public const READER_UNAVAILABLE = 'READER_UNAVAILABLE'; + public const NOT_CONFIGURED = 'NOT_CONFIGURED'; + + /** + * HTTP status per error code. + * + * @var array + */ + private const STATUS = [ + self::NOT_XLSX => 400, + self::NO_SOURCE_SHEET => 422, + self::MISSING_COLUMN => 422, + self::TOO_MANY_ROWS => 422, + self::MUNICIPALITY_REQUIRED => 422, + self::MUNICIPALITY_INVALID => 422, + self::MAPPING_UNAVAILABLE => 503, + self::READER_UNAVAILABLE => 503, + self::NOT_CONFIGURED => 503, + ]; + + /** + * Constructor. + * + * @param string $errorCode One of the class constants. + * @param string $message English log message, no person data. + * @param array $details Contract details, e.g. sheet and column. + * @param Throwable|null $previous The cause, if any. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function __construct( + private readonly string $errorCode, + string $message, + private readonly array $details = [], + ?Throwable $previous = null, + ) { + parent::__construct(message: $message, code: 0, previous: $previous); + }//end __construct() + + /** + * The contract error code, e.g. `MISSING_COLUMN`. + * + * @return string + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function getErrorCode(): string { + return $this->errorCode; + }//end getErrorCode() + + /** + * The HTTP status the controller answers with. + * + * @return int + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function getHttpStatus(): int { + return (self::STATUS[$this->errorCode] ?? 500); + }//end getHttpStatus() + + /** + * The contract details of the error. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function getDetails(): array { + return $this->details; + }//end getDetails() +}//end class diff --git a/lib/Service/Cmdb/CmdbImportProfile.php b/lib/Service/Cmdb/CmdbImportProfile.php new file mode 100644 index 000000000..ff67accc6 --- /dev/null +++ b/lib/Service/Cmdb/CmdbImportProfile.php @@ -0,0 +1,603 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Service\Cmdb; + +use OCA\Stackiq\Exception\CmdbImportException; +use Psr\Container\ContainerInterface; +use Throwable; + +/** + * The validated import profile plus its packs. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + * + * @SuppressWarnings(PHPMD.TooManyPublicMethods) One small accessor per profile setting, so + * callers never read the raw JSON. + * @SuppressWarnings(PHPMD.TooManyMethods) The same accessors, plus the loader's small private helpers. + * @SuppressWarnings(PHPMD.ExcessiveClassComplexity) The accessors each guard against a + * malformed profile value; the sum passes the threshold, no single method is complex. + */ +class CmdbImportProfile { + /** + * OpenRegister's pack validator (not a public contract). + */ + public const VALIDATOR_CLASS = 'OCA\OpenRegister\Service\MigrationPack\PackDefinitionValidator'; + + /** + * The targets every profile must name a pack for. + * + * @var array + */ + public const TARGETS = ['module', 'manufacturer', 'municipality', 'usage', 'businessOwner']; + + /** + * Default upload limit when the profile file cannot be read (10 MB). + */ + public const DEFAULT_MAX_FILE_BYTES = 10485760; + + /** + * Sources of the municipality pack that come from the request, not from a sheet. + * + * @var array + */ + private const OPTION_SOURCES = ['municipalityName']; + + /** + * The decoded profile, once loaded. + * + * @var array|null + */ + private ?array $profile = null; + + /** + * The validated packs per target, once loaded. + * + * @var array> + */ + private array $packs = []; + + /** + * Constructor. + * + * @param ContainerInterface $container Resolves OpenRegister's validator. + * @param string|null $directory Directory of the profile and packs; null is the shipped one. + * @param string $profileFile File name of the profile inside the directory. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function __construct( + private readonly ContainerInterface $container, + private ?string $directory = null, + private readonly string $profileFile = 'topdesk-profile.json', + ) { + if ($this->directory === null) { + $this->directory = __DIR__ . '/../../Settings/cmdb-import'; + } + }//end __construct() + + /** + * Load the profile and validate every pack it names. + * + * @return void + * + * @throws CmdbImportException MAPPING_UNAVAILABLE when the validator is missing, + * or the profile or a pack is unreadable or invalid. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function load(): void { + $validator = $this->resolveValidator(); + $profile = $this->decodeFile(fileName: $this->profileFile); + + $packs = []; + foreach (self::TARGETS as $target) { + $fileName = $profile['packs'][$target] ?? null; + if (is_string($fileName) === false || $fileName === '' || basename($fileName) !== $fileName) { + throw new CmdbImportException( + errorCode: CmdbImportException::MAPPING_UNAVAILABLE, + message: 'CMDB import profile names no pack for target ' . $target + ); + } + + $pack = $this->decodeFile(fileName: $fileName); + $errors = $validator->validate($pack); + if (is_array($errors) === true && $errors !== []) { + throw new CmdbImportException( + errorCode: CmdbImportException::MAPPING_UNAVAILABLE, + message: 'CMDB mapping pack ' . $fileName . ' is invalid: ' . implode('; ', array_map('strval', $errors)) + ); + } + + $packs[$target] = $pack; + } + + $this->profile = $profile; + $this->packs = $packs; + }//end load() + + /** + * The upload limit in bytes, readable without validating the packs. + * + * The controller checks the size before anything else, so this must not + * depend on OpenRegister being available. + * + * @return int + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + public function maxFileBytes(): int { + try { + $profile = $this->profile ?? $this->decodeFile(fileName: $this->profileFile); + } catch (CmdbImportException $e) { + return self::DEFAULT_MAX_FILE_BYTES; + } + + $limit = $profile['maxFileBytes'] ?? null; + if (is_int($limit) === true && $limit > 0) { + return $limit; + } + + return self::DEFAULT_MAX_FILE_BYTES; + }//end maxFileBytes() + + /** + * The maximum number of non-empty rows per source sheet. + * + * @return int + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function maxRowsPerSheet(): int { + $limit = $this->profile()['maxRowsPerSheet'] ?? 10000; + if (is_int($limit) === false || $limit < 0) { + return 10000; + } + + return $limit; + }//end maxRowsPerSheet() + + /** + * The source sheets, each with the constants it adds to its rows and the + * pack columns it is known not to have. + * + * @return array, absentColumns: array}> + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function sheets(): array { + $sheets = []; + foreach (($this->profile()['sheets'] ?? []) as $sheet) { + if (is_array($sheet) === false || is_string($sheet['name'] ?? null) === false) { + continue; + } + + $constants = []; + if (is_array($sheet['constants'] ?? null) === true) { + foreach ($sheet['constants'] as $column => $value) { + if (is_scalar($value) === true) { + $constants[(string)$column] = (string)$value; + } + } + } + + $absent = []; + if (is_array($sheet['absentColumns'] ?? null) === true) { + $absent = array_values(array_map('strval', $sheet['absentColumns'])); + } + + $sheets[] = ['name' => $sheet['name'], 'constants' => $constants, 'absentColumns' => $absent]; + }//end foreach + + return $sheets; + }//end sheets() + + /** + * The names of the source sheets. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function sheetNames(): array { + return array_column($this->sheets(), 'name'); + }//end sheetNames() + + /** + * The constants a sheet adds to each of its rows, as column => value. + * + * A constant is mapped like a column (the usage pack reads "Beheer"), but + * it is never looked up in the sheet. + * + * @param string $sheetName The sheet name. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public function sheetConstants(string $sheetName): array { + foreach ($this->sheets() as $sheet) { + if ($sheet['name'] === $sheetName) { + return $sheet['constants']; + } + } + + return []; + }//end sheetConstants() + + /** + * The pack columns a sheet is known not to have; their absence is no warning. + * + * @param string $sheetName The sheet name. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function absentColumns(string $sheetName): array { + foreach ($this->sheets() as $sheet) { + if ($sheet['name'] === $sheetName) { + return $sheet['absentColumns']; + } + } + + return []; + }//end absentColumns() + + /** + * The names of every sheet constant; these are never read from a sheet. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function constantColumns(): array { + $columns = []; + foreach ($this->sheets() as $sheet) { + $columns = array_merge($columns, array_keys($sheet['constants'])); + } + + return array_values(array_unique(array_map('strval', $columns))); + }//end constantColumns() + + /** + * Values that mean "empty" per column, such as the "NB" a CMDB sheet + * writes for an unknown BNN classification. + * + * @return array> + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function emptyValues(): array { + $value = $this->profile()['emptyValues'] ?? []; + if (is_array($value) === false) { + return []; + } + + $empty = []; + foreach ($value as $column => $values) { + if (is_array($values) === true) { + $empty[(string)$column] = array_values(array_map('strval', $values)); + } + } + + return $empty; + }//end emptyValues() + + /** + * The match key column ("APPID"). + * + * @return string + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public function keyColumn(): string { + return (string)($this->profile()['keyColumn'] ?? 'APPID'); + }//end keyColumn() + + /** + * The application name column ("Applicatie Naam"). + * + * @return string + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function nameColumn(): string { + return (string)($this->profile()['nameColumn'] ?? 'Applicatie Naam'); + }//end nameColumn() + + /** + * Columns whose absence stops the import with MISSING_COLUMN. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function requiredColumns(): array { + return $this->stringList(key: 'requiredColumns'); + }//end requiredColumns() + + /** + * Columns that hold Excel serial dates. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function dateColumns(): array { + return $this->stringList(key: 'dateColumns'); + }//end dateColumns() + + /** + * Columns that hold identifiers which must not carry a decimal part. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function idColumns(): array { + return $this->stringList(key: 'idColumns'); + }//end idColumns() + + /** + * The prefix of the module match key ("topdesk"). + * + * @return string + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public function externalKeyPrefix(): string { + return (string)($this->profile()['externalKeyPrefix'] ?? 'topdesk'); + }//end externalKeyPrefix() + + /** + * The validated pack for a target. + * + * @param string $target One of self::TARGETS. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function pack(string $target): array { + $this->profile(); + return ($this->packs[$target] ?? []); + }//end pack() + + /** + * Values set on create only, per target, as field => value. + * + * @param string $target The target. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public function createOnlyDefaults(string $target): array { + $value = $this->profile()['createOnly'][$target] ?? []; + if (is_array($value) === false || array_is_list($value) === true) { + return []; + } + + return $value; + }//end createOnlyDefaults() + + /** + * Mapped fields written on create, or on update only when the stored value is empty. + * + * @param string $target The target. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public function createOnlyFields(string $target): array { + $value = $this->profile()['createOnly'][$target] ?? []; + if (is_array($value) === false) { + return []; + } + + if (array_is_list($value) === true) { + return array_values(array_map('strval', $value)); + } + + return array_map('strval', array_keys($value)); + }//end createOnlyFields() + + /** + * Fields the import never writes on an existing object. + * + * @param string $target The target. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public function neverWrittenOnUpdate(string $target): array { + $value = $this->profile()['neverWritten'][$target] ?? []; + if (is_array($value) === false) { + return []; + } + + return array_values(array_map('strval', $value)); + }//end neverWrittenOnUpdate() + + /** + * The accepted values of the missingRecords option. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + public function missingRecordsModes(): array { + $modes = $this->stringList(key: 'missingRecords'); + if ($modes === []) { + return ['keep']; + } + + return $modes; + }//end missingRecordsModes() + + /** + * Every column the profile or a pack references: the read allowlist. + * + * The municipality pack maps the request options, not a sheet, and the + * sheet constants are added by the import, so both are left out. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + public function referencedColumns(): array { + $columns = array_merge( + [$this->keyColumn(), $this->nameColumn()], + $this->requiredColumns(), + $this->dateColumns(), + $this->idColumns() + ); + + foreach (self::TARGETS as $target) { + foreach (($this->pack(target: $target)['fieldMappings'] ?? []) as $mapping) { + $columns[] = (string)($mapping['source'] ?? ''); + foreach (($mapping['transform']['fields'] ?? []) as $extra) { + $columns[] = (string)$extra; + } + } + } + + $excluded = array_merge(self::OPTION_SOURCES, $this->constantColumns()); + $columns = array_filter( + $columns, + fn (string $column): bool => $column !== '' && $column[0] !== '/' && in_array($column, $excluded, true) === false + ); + + return array_values(array_unique($columns)); + }//end referencedColumns() + + /** + * The loaded profile. + * + * @return array + * + * @throws CmdbImportException MAPPING_UNAVAILABLE when load() failed. + */ + private function profile(): array { + if ($this->profile === null) { + $this->load(); + } + + return ($this->profile ?? []); + }//end profile() + + /** + * A list of strings from the profile. + * + * @param string $key The profile key. + * + * @return array + */ + private function stringList(string $key): array { + $value = $this->profile()[$key] ?? []; + if (is_array($value) === false) { + return []; + } + + return array_values(array_map('strval', $value)); + }//end stringList() + + /** + * Resolve OpenRegister's pack validator. + * + * @return object The validator, with a `validate(array): array` method. + * + * @throws CmdbImportException MAPPING_UNAVAILABLE when it is not available. + */ + private function resolveValidator(): object { + $class = static::VALIDATOR_CLASS; + + try { + if ($this->container->has($class) === true) { + $validator = $this->container->get($class); + if (is_object($validator) === true && method_exists($validator, 'validate') === true) { + return $validator; + } + } + } catch (Throwable $e) { + // Fall through to the class check below. + $validator = null; + } + + if (class_exists($class) === true) { + $validator = new $class(); + if (method_exists($validator, 'validate') === true) { + return $validator; + } + } + + throw new CmdbImportException( + errorCode: CmdbImportException::MAPPING_UNAVAILABLE, + message: 'OpenRegister PackDefinitionValidator is not available' + ); + }//end resolveValidator() + + /** + * Read and decode one JSON file from the profile directory. + * + * @param string $fileName The file name. + * + * @return array + * + * @throws CmdbImportException MAPPING_UNAVAILABLE when the file is missing or not a JSON object. + */ + private function decodeFile(string $fileName): array { + $path = $this->directory . '/' . $fileName; + $content = false; + if (is_readable($path) === true) { + $content = file_get_contents($path); + } + + $decoded = null; + if (is_string($content) === true) { + $decoded = json_decode($content, true); + } + + if (is_array($decoded) === false) { + throw new CmdbImportException( + errorCode: CmdbImportException::MAPPING_UNAVAILABLE, + message: 'CMDB import file ' . $fileName . ' is missing or not valid JSON' + ); + } + + return $decoded; + }//end decodeFile() +}//end class diff --git a/lib/Service/Cmdb/CmdbImportReport.php b/lib/Service/Cmdb/CmdbImportReport.php new file mode 100644 index 000000000..091b0a903 --- /dev/null +++ b/lib/Service/Cmdb/CmdbImportReport.php @@ -0,0 +1,220 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Service\Cmdb; + +/** + * The per-row report of one CMDB import run. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ +class CmdbImportReport { + public const CREATED = 'created'; + public const UPDATED = 'updated'; + public const UNCHANGED = 'unchanged'; + public const SKIPPED = 'skipped'; + public const FAILED = 'failed'; + + /** + * The row entries, in processing order. + * + * @var array> + */ + private array $rows = []; + + /** + * Import-level warnings. + * + * @var array + */ + private array $importWarnings = []; + + /** + * Whether the run stopped on a cancel. + * + * @var bool + */ + private bool $cancelled = false; + + /** + * The consuming municipality. + * + * @var array{uuid: string, name: string, created: bool}|null + */ + private ?array $municipality = null; + + /** + * Constructor. + * + * @param string $operationId The progress operation id. + * @param int $rowsRead Non-empty rows read from the workbook. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function __construct( + private readonly string $operationId, + private readonly int $rowsRead, + ) { + }//end __construct() + + /** + * Add one row outcome. + * + * @param string $sheet The sheet name. + * @param int $row The 1-based sheet row number. + * @param string $appId The APPID ('' when missing). + * @param string $name The application name ('' when missing). + * @param string $outcome One of the outcome constants. + * @param array $reasons Why the row was skipped or failed. + * @param array $warnings Row warnings. + * @param string|null $moduleUuid The module, when there is one. + * @param string|null $usageUuid The usage, when there is one. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function addRow( + string $sheet, + int $row, + string $appId, + string $name, + string $outcome, + array $reasons = [], + array $warnings = [], + ?string $moduleUuid = null, + ?string $usageUuid = null, + ): void { + $this->rows[] = [ + 'sheet' => $sheet, + 'row' => $row, + 'appId' => $appId, + 'name' => $name, + 'outcome' => $outcome, + 'reasons' => array_values($reasons), + 'warnings' => array_values($warnings), + 'moduleUuid' => $moduleUuid, + 'usageUuid' => $usageUuid, + ]; + }//end addRow() + + /** + * Add import-level warnings, such as a missing optional column. + * + * @param array $warnings The warnings. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function addImportWarnings(array $warnings): void { + foreach ($warnings as $warning) { + $this->importWarnings[] = ['sheet' => (string)$warning['sheet'], 'message' => (string)$warning['message']]; + } + }//end addImportWarnings() + + /** + * Record the consuming municipality. + * + * @param string $uuid The organisation uuid. + * @param string $name Its name. + * @param bool $created Whether this run created it. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function setMunicipality(string $uuid, string $name, bool $created): void { + $this->municipality = ['uuid' => $uuid, 'name' => $name, 'created' => $created]; + }//end setMunicipality() + + /** + * Mark the run as stopped on a cancel. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function markCancelled(): void { + $this->cancelled = true; + }//end markCancelled() + + /** + * The number of rows processed so far. + * + * @return int + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function processed(): int { + return count($this->rows); + }//end processed() + + /** + * The summary counts. + * + * @return array{rowsRead: int, processed: int, created: int, updated: int, unchanged: int, skipped: int, failed: int, warnings: int} + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function summary(): array { + $summary = [ + 'rowsRead' => $this->rowsRead, + 'processed' => count($this->rows), + self::CREATED => 0, + self::UPDATED => 0, + self::UNCHANGED => 0, + self::SKIPPED => 0, + self::FAILED => 0, + 'warnings' => 0, + ]; + + foreach ($this->rows as $row) { + $summary[$row['outcome']]++; + $summary['warnings'] += count($row['warnings']); + } + + return $summary; + }//end summary() + + /** + * The report in the contract shape. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function toArray(): array { + return [ + 'success' => true, + 'operationId' => $this->operationId, + 'cancelled' => $this->cancelled, + 'municipality' => $this->municipality, + 'summary' => $this->summary(), + 'importWarnings' => $this->importWarnings, + 'rows' => $this->rows, + ]; + }//end toArray() +}//end class diff --git a/lib/Service/Cmdb/CmdbRowNormaliser.php b/lib/Service/Cmdb/CmdbRowNormaliser.php new file mode 100644 index 000000000..80c71f080 --- /dev/null +++ b/lib/Service/Cmdb/CmdbRowNormaliser.php @@ -0,0 +1,209 @@ + string` row OpenRegister's `MappingEngine` expects (design D4): + * + * - Date columns: an Excel serial number becomes `Y-m-d` (1900 date system, + * or 1904 when the workbook says so). A non-numeric value stays as it is, so + * the pack's `date` transform accepts it or reports a warning. + * - Id columns: a whole number becomes a string without a decimal part + * (`1234.0` becomes `"1234"`). + * - Every value is trimmed; an empty value becomes the empty string. + * - A value the profile lists as "empty" for its column (such as the "NB" a + * CMDB sheet writes for an unknown BNN classification) becomes the empty + * string, compared case-insensitively before any conversion. + * + * The conversion lives here and not in the packs, so every pack stays a plain + * OpenRegister migration pack. + * + * @category Service + * @package OCA\Stackiq\Service\Cmdb + * @author Conduction b.v. + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Service\Cmdb; + +use DateInterval; +use DateTimeImmutable; +use DateTimeZone; + +/** + * Normalises reader rows into string rows for the mapping engine. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + * + * @SuppressWarnings(PHPMD.BooleanArgumentFlag) The date system (1900 or 1904) is a property + * of the workbook that the reader reports; it is data, not a mode switch. + */ +class CmdbRowNormaliser { + /** + * Highest serial Excel accepts (9999-12-31). + */ + private const MAX_SERIAL = 2958465; + + /** + * Normalise one row. + * + * @param array $cells Column name => raw cell value. + * @param array $dateColumns Columns holding Excel serial dates. + * @param array $idColumns Columns holding identifiers. + * @param bool $date1904 Whether the workbook uses the 1904 date system. + * @param array> $emptyValues Values that mean empty, per column. + * + * @return array Column name => normalised value. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function normalise(array $cells, array $dateColumns, array $idColumns, bool $date1904 = false, array $emptyValues = []): array { + $row = []; + foreach ($cells as $column => $value) { + $column = (string)$column; + if (self::meansEmpty(text: $this->toText(value: $value), empty: ($emptyValues[$column] ?? [])) === true) { + $row[$column] = ''; + continue; + } + + if (in_array($column, $dateColumns, true) === true) { + $row[$column] = $this->normaliseDate(value: $value, date1904: $date1904); + continue; + } + + if (in_array($column, $idColumns, true) === true) { + $row[$column] = $this->normaliseId(value: $value); + continue; + } + + $row[$column] = $this->toText(value: $value); + } + + return $row; + }//end normalise() + + /** + * An Excel serial date to `Y-m-d`; any other value trimmed as text. + * + * @param mixed $value The raw value. + * @param bool $date1904 Whether the workbook uses the 1904 date system. + * + * @return string + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function normaliseDate(mixed $value, bool $date1904 = false): string { + $text = $this->toText(value: $value); + if (is_numeric($text) === false) { + return $text; + } + + $serial = (float)$text; + if ($serial < 1 || $serial > self::MAX_SERIAL) { + return $text; + } + + $days = (int)floor($serial); + // 1900 system: 1899-12-30 plus the serial, which absorbs Excel's + // phantom 1900-02-29 for every serial after it. Before it (serial < 61) + // the base is one day later. 1904 system: serial 0 is 1904-01-01. + $base = '1899-12-30'; + if ($days < 61) { + $base = '1899-12-31'; + } + + if ($date1904 === true) { + $base = '1904-01-01'; + } + + $base = new DateTimeImmutable($base, new DateTimeZone('UTC')); + + return $base->add(new DateInterval('P' . $days . 'D'))->format('Y-m-d'); + }//end normaliseDate() + + /** + * A numeric identifier without a decimal part, as a string. + * + * @param mixed $value The raw value. + * + * @return string + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function normaliseId(mixed $value): string { + if (is_float($value) === true && floor($value) === $value && abs($value) < PHP_INT_MAX) { + return (string)(int)$value; + } + + $text = $this->toText(value: $value); + if (preg_match('/^(\d+)\.0+$/', $text, $matches) === 1) { + return $matches[1]; + } + + return $text; + }//end normaliseId() + + /** + * Whether a value is one of the column's "empty" values. + * + * @param string $text The trimmed value. + * @param array $empty The column's empty values. + * + * @return bool + */ + private static function meansEmpty(string $text, array $empty): bool { + if ($text === '' || $empty === []) { + return false; + } + + $needle = mb_strtolower($text); + foreach ($empty as $candidate) { + if (mb_strtolower(trim($candidate)) === $needle) { + return true; + } + } + + return false; + }//end meansEmpty() + + /** + * Any scalar as trimmed text; null as the empty string. + * + * @param mixed $value The raw value. + * + * @return string + */ + private function toText(mixed $value): string { + if ($value === null) { + return ''; + } + + if (is_bool($value) === true) { + if ($value === true) { + return 'TRUE'; + } + + return 'FALSE'; + } + + if (is_float($value) === true && floor($value) === $value && abs($value) < PHP_INT_MAX) { + return (string)(int)$value; + } + + if (is_scalar($value) === false) { + return ''; + } + + return trim((string)$value); + }//end toText() +}//end class diff --git a/lib/Service/Cmdb/CmdbWorkbookReader.php b/lib/Service/Cmdb/CmdbWorkbookReader.php new file mode 100644 index 000000000..1ec5cac07 --- /dev/null +++ b/lib/Service/Cmdb/CmdbWorkbookReader.php @@ -0,0 +1,497 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Service\Cmdb; + +use OCA\Stackiq\Exception\CmdbImportException; +use Throwable; +use ZipArchive; + +/** + * Reads the allowlisted columns of the profile's source sheets. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + * + * @SuppressWarnings(PHPMD.ExcessiveClassComplexity) The file checks before parsing and the + * header resolution are each a chain of small guards; together they pass the threshold. + */ +class CmdbWorkbookReader { + /** + * PhpSpreadsheet's Xlsx reader, shipped in OpenRegister's vendor directory. + */ + public const READER_CLASS = 'PhpOffice\PhpSpreadsheet\Reader\Xlsx'; + + /** + * The ZIP local-file-header signature every xlsx package starts with. + */ + private const ZIP_SIGNATURE = "PK\x03\x04"; + + /** + * Check that an upload is an xlsx workbook, without parsing it. + * + * @param string $path The uploaded temporary file. + * @param string $fileName The original file name. + * + * @return void + * + * @throws CmdbImportException NOT_XLSX when the name, signature or package is wrong. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function assertXlsx(string $path, string $fileName): void { + if (strtolower((string)pathinfo($fileName, PATHINFO_EXTENSION)) !== 'xlsx') { + throw new CmdbImportException(errorCode: CmdbImportException::NOT_XLSX, message: 'The file name does not end in .xlsx'); + } + + $head = false; + if (is_file($path) === true && is_readable($path) === true) { + $head = file_get_contents($path, false, null, 0, 4); + } + + if ($head !== self::ZIP_SIGNATURE) { + throw new CmdbImportException(errorCode: CmdbImportException::NOT_XLSX, message: 'The file is not a ZIP package'); + } + + $zip = new ZipArchive(); + if ($zip->open($path, ZipArchive::RDONLY) !== true) { + throw new CmdbImportException(errorCode: CmdbImportException::NOT_XLSX, message: 'The ZIP package cannot be opened'); + } + + $hasWorkbook = ($zip->locateName('xl/workbook.xml') !== false); + $zip->close(); + + if ($hasWorkbook === false) { + throw new CmdbImportException(errorCode: CmdbImportException::NOT_XLSX, message: 'The package holds no xl/workbook.xml'); + } + }//end assertXlsx() + + /** + * Whether PhpSpreadsheet's Xlsx reader can be loaded. + * + * @return bool + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function isAvailable(): bool { + return class_exists(static::READER_CLASS) === true; + }//end isAvailable() + + /** + * Read the source sheets of an xlsx workbook. + * + * @param string $path The xlsx file, already checked by assertXlsx(). + * @param CmdbImportProfile $profile The import profile. + * + * @return array `rows` (list of {sheet, row, cells, uncached}), `importWarnings` + * (list of {sheet, message}) and `date1904` (bool). + * + * @throws CmdbImportException READER_UNAVAILABLE, NOT_XLSX, NO_SOURCE_SHEET, MISSING_COLUMN or TOO_MANY_ROWS. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public function read(string $path, CmdbImportProfile $profile): array { + if ($this->isAvailable() === false) { + throw new CmdbImportException( + errorCode: CmdbImportException::READER_UNAVAILABLE, + message: 'PhpSpreadsheet Xlsx reader is not available' + ); + } + + $readerClass = static::READER_CLASS; + $reader = new $readerClass(); + + try { + $available = $reader->listWorksheetNames($path); + } catch (Throwable $e) { + throw new CmdbImportException( + errorCode: CmdbImportException::NOT_XLSX, + message: 'The workbook cannot be read: ' . get_class($e), + previous: $e + ); + } + + $expected = $profile->sheetNames(); + $present = array_values(array_intersect($expected, $available)); + if ($present === []) { + throw new CmdbImportException( + errorCode: CmdbImportException::NO_SOURCE_SHEET, + message: 'The workbook holds none of the source sheets', + details: ['expected' => $expected] + ); + } + + $reader->setReadDataOnly(true); + $reader->setReadEmptyCells(false); + $reader->setLoadSheetsOnly($present); + + try { + $spreadsheet = $reader->load($path); + } catch (Throwable $e) { + throw new CmdbImportException( + errorCode: CmdbImportException::NOT_XLSX, + message: 'The workbook cannot be loaded: ' . get_class($e), + previous: $e + ); + } + + try { + $result = $this->readSheets(spreadsheet: $spreadsheet, sheetNames: $present, profile: $profile); + } finally { + $spreadsheet->disconnectWorksheets(); + } + + return $result; + }//end read() + + /** + * Normalise a header or column name for matching. + * + * @param string $header The raw header. + * + * @return string The normalised name. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + */ + public static function normaliseHeader(string $header): string { + $header = (string)preg_replace('/\s+/u', ' ', trim($header)); + $header = (string)preg_replace('/\s*(?::|⚡)+$/u', '', $header); + + return mb_strtolower(trim($header)); + }//end normaliseHeader() + + /** + * Read every present source sheet. + * + * @param object $spreadsheet The loaded PhpSpreadsheet workbook. + * @param array $sheetNames The present source sheets, in profile order. + * @param CmdbImportProfile $profile The import profile. + * + * @return array `rows` (list of {sheet, row, cells, uncached}), `importWarnings` + * (list of {sheet, message}) and `date1904` (bool). + * + * @throws CmdbImportException MISSING_COLUMN or TOO_MANY_ROWS. + */ + private function readSheets(object $spreadsheet, array $sheetNames, CmdbImportProfile $profile): array { + $referenced = $profile->referencedColumns(); + $mapped = $this->packSources(profile: $profile); + $required = $profile->requiredColumns(); + + // Resolve every sheet's columns first, so a missing required column + // stops the import before a single row is read. + $columnsPerSheet = []; + $warnings = []; + foreach ($sheetNames as $sheetName) { + $worksheet = $spreadsheet->getSheetByName($sheetName); + $columns = $this->resolveColumns(worksheet: $worksheet, referenced: $referenced); + + foreach ($required as $column) { + if (in_array($column, $columns, true) === false) { + throw new CmdbImportException( + errorCode: CmdbImportException::MISSING_COLUMN, + message: 'A source sheet lacks a required column', + details: ['sheet' => $sheetName, 'column' => $column] + ); + } + } + + $skip = array_merge($required, $profile->absentColumns(sheetName: $sheetName)); + array_push($warnings, ...self::missingOptionalColumns(sheetName: $sheetName, mapped: $mapped, columns: $columns, skip: $skip)); + $columnsPerSheet[$sheetName] = $columns; + } + + $rows = []; + $limit = $profile->maxRowsPerSheet(); + foreach ($sheetNames as $sheetName) { + $worksheet = $spreadsheet->getSheetByName($sheetName); + $sheetRows = $this->readRows(worksheet: $worksheet, columns: $columnsPerSheet[$sheetName], sheetName: $sheetName, limit: $limit); + array_push($rows, ...$sheetRows); + } + + $date1904 = false; + if (method_exists($spreadsheet, 'getExcelCalendar') === true) { + $date1904 = ((int)$spreadsheet->getExcelCalendar() === 1904); + } + + return ['rows' => $rows, 'importWarnings' => $warnings, 'date1904' => $date1904]; + }//end readSheets() + + /** + * Map the header row to the referenced column names. + * + * @param object $worksheet The worksheet. + * @param array $referenced The allowlisted column names. + * + * @return array Column letter => referenced column name. + */ + private function resolveColumns(object $worksheet, array $referenced): array { + $wanted = []; + foreach ($referenced as $column) { + $wanted[self::normaliseHeader(header: $column)] = $column; + } + + $columns = []; + $lastColumn = self::columnIndex(letters: (string)$worksheet->getHighestDataColumn(1)); + for ($index = 1; $index <= $lastColumn; $index++) { + $letters = self::columnLetters(index: $index); + $coordinate = $letters . '1'; + if ($worksheet->cellExists($coordinate) === false) { + continue; + } + + $header = $this->cellValue(cell: $worksheet->getCell($coordinate)); + if (is_scalar($header) === false) { + continue; + } + + $name = $wanted[self::normaliseHeader(header: (string)$header)] ?? null; + // The first column with a matching header wins. + if ($name !== null && in_array($name, $columns, true) === false) { + $columns[$letters] = $name; + } + } + + return $columns; + }//end resolveColumns() + + /** + * Read the non-empty data rows of one sheet. + * + * @param object $worksheet The worksheet. + * @param array $columns Column letter => column name. + * @param string $sheetName The sheet name. + * @param int $limit The maximum number of non-empty rows. + * + * @return array, uncached: array}> + * + * @throws CmdbImportException TOO_MANY_ROWS. + */ + private function readRows(object $worksheet, array $columns, string $sheetName, int $limit): array { + $rows = []; + $lastRow = (int)$worksheet->getHighestDataRow(); + for ($rowNumber = 2; $rowNumber <= $lastRow; $rowNumber++) { + ['cells' => $cells, 'uncached' => $uncached, 'empty' => $empty] = $this->readRow( + worksheet: $worksheet, + columns: $columns, + rowNumber: $rowNumber + ); + if ($empty === true) { + continue; + } + + if (count($rows) >= $limit) { + throw new CmdbImportException( + errorCode: CmdbImportException::TOO_MANY_ROWS, + message: 'A source sheet has more rows than the profile allows', + details: ['sheet' => $sheetName, 'limit' => $limit] + ); + } + + $rows[] = ['sheet' => $sheetName, 'row' => $rowNumber, 'cells' => $cells, 'uncached' => $uncached]; + }//end for + + return $rows; + }//end readRows() + + /** + * The kept cells of one row, the columns whose formula has no cached value, + * and whether every kept cell is empty. + * + * @param object $worksheet The worksheet. + * @param array $columns Column letter => column name. + * @param int $rowNumber The 1-based row number. + * + * @return array{cells: array, uncached: array, empty: bool} + */ + private function readRow(object $worksheet, array $columns, int $rowNumber): array { + $cells = []; + $uncached = []; + $empty = true; + foreach ($columns as $letters => $name) { + $value = null; + $coordinate = $letters . $rowNumber; + if ($worksheet->cellExists($coordinate) === true) { + $cell = $worksheet->getCell($coordinate); + $value = $this->cellValue(cell: $cell); + if (self::isUncachedFormula(cell: $cell) === true) { + $uncached[] = $name; + } + } + + $cells[$name] = $value; + if ($value !== null && (is_string($value) === false || trim($value) !== '')) { + $empty = false; + } + } + + return ['cells' => $cells, 'uncached' => $uncached, 'empty' => $empty]; + }//end readRow() + + /** + * One import warning per pack column a sheet lacks, except the ones to skip. + * + * @param string $sheetName The sheet name. + * @param array $mapped The sheet-mapped pack sources. + * @param array $columns The sheet's resolved columns. + * @param array $skip Required columns and the columns the sheet is known to lack. + * + * @return array + */ + private static function missingOptionalColumns(string $sheetName, array $mapped, array $columns, array $skip): array { + $warnings = []; + foreach ($mapped as $column) { + if (in_array($column, $columns, true) === false && in_array($column, $skip, true) === false) { + $warnings[] = ['sheet' => $sheetName, 'column' => $column, 'message' => sprintf('Optional column "%s" not found', $column)]; + } + } + + return $warnings; + }//end missingOptionalColumns() + + /** + * Whether a cell holds a formula without a cached value. + * + * @param object $cell The PhpSpreadsheet cell. + * + * @return bool + */ + private static function isUncachedFormula(object $cell): bool { + return $cell->getDataType() === 'f' && $cell->getOldCalculatedValue() === null; + }//end isUncachedFormula() + + /** + * The stored value of a cell; for a formula, the value Excel cached. + * + * A formula without a cached value, and a formula whose cached value is + * the number 0 (Excel's result for a reference to an empty cell), yield + * null. + * + * @param object $cell The PhpSpreadsheet cell. + * + * @return mixed A scalar or null. + */ + private function cellValue(object $cell): mixed { + $value = $cell->getValue(); + if ($cell->getDataType() === 'f') { + // The value Excel cached; the formula itself is never evaluated. + $value = $cell->getOldCalculatedValue(); + if ((is_int($value) === true || is_float($value) === true) && (float)$value === 0.0) { + return null; + } + } + + if (is_object($value) === true && method_exists($value, 'getPlainText') === true) { + return (string)$value->getPlainText(); + } + + if (is_scalar($value) === false) { + return null; + } + + return $value; + }//end cellValue() + + /** + * The sources of every sheet-mapped pack field. + * + * @param CmdbImportProfile $profile The import profile. + * + * @return array + */ + private function packSources(CmdbImportProfile $profile): array { + $constants = $profile->constantColumns(); + $sources = []; + foreach (CmdbImportProfile::TARGETS as $target) { + if ($target === 'municipality') { + continue; + } + + foreach (($profile->pack(target: $target)['fieldMappings'] ?? []) as $mapping) { + $sources[] = (string)($mapping['source'] ?? ''); + } + } + + return array_values( + array_unique( + array_filter( + $sources, + fn (string $source): bool => $source !== '' && in_array($source, $constants, true) === false + ) + ) + ); + }//end packSources() + + /** + * Column letters to a 1-based index ("A" = 1, "AA" = 27). + * + * @param string $letters The column letters. + * + * @return int + */ + private static function columnIndex(string $letters): int { + $index = 0; + foreach (str_split(strtoupper($letters)) as $char) { + $index = (($index * 26) + (ord($char) - 64)); + } + + return $index; + }//end columnIndex() + + /** + * A 1-based column index to its letters. + * + * @param int $index The column index. + * + * @return string + */ + private static function columnLetters(int $index): string { + $letters = ''; + while ($index > 0) { + $remainder = (($index - 1) % 26); + $letters = chr(65 + $remainder) . $letters; + $index = intdiv(($index - 1), 26); + } + + return $letters; + }//end columnLetters() +}//end class diff --git a/lib/Service/CmdbExportImportService.php b/lib/Service/CmdbExportImportService.php new file mode 100644 index 000000000..793e9b836 --- /dev/null +++ b/lib/Service/CmdbExportImportService.php @@ -0,0 +1,1411 @@ +:`, so a + * second import of a newer export updates the same records. + * + * What each column becomes is declarative: the migration packs under + * `lib/Settings/cmdb-import/`, executed by OpenRegister's + * `MigrationPack\MappingEngine` (ADR-031). This class is the imperative glue + * around the file: reading it, splitting a row over four linked objects, + * resolving contacts, progress and cancel. Every read and write goes through + * OpenRegister's `ObjectServiceInterface` (ADR-022). + * + * Rules stated once and enforced here: + * - A module matches on `externalKey`; a usage on (consumer, module); a + * supplier on its normalised name and type Supplier; a contact person on + * (contactsUid, organization). + * - `publicationDate` is set to the import's start on create and never + * written on update; neither is `depublicationDate`. + * - Records missing from a newer export are left untouched. + * - Owners become contact persons, never Nextcloud user accounts, and no + * report entry or log line carries an owner name or e-mail address. + * + * @category Service + * @package OCA\Stackiq\Service + * @author Conduction b.v. + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Service; + +use DateTimeImmutable; +use DateTimeZone; +use OCA\OpenRegister\Contract\ObjectServiceInterface; +use OCA\Stackiq\Exception\CmdbImportException; +use OCA\Stackiq\Service\Cmdb\CmdbImportProfile; +use OCA\Stackiq\Service\Cmdb\CmdbImportReport; +use OCA\Stackiq\Service\Cmdb\CmdbRowNormaliser; +use OCA\Stackiq\Service\Cmdb\CmdbWorkbookReader; +use OCP\IL10N; +use Psr\Container\ContainerInterface; +use Psr\Log\LoggerInterface; +use Throwable; + +/** + * Imports a TOPdesk CMDB export for one municipality. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + * + * @SuppressWarnings(PHPMD.ExcessiveClassComplexity) One import run resolves four linked + * object kinds per row (supplier, module, owners, usage), each with its own match rule, + * create-only fields and run cache. Splitting them over several classes would hand the + * same run state from class to class without making any one rule simpler to read. + * @SuppressWarnings(PHPMD.TooManyFields) The run caches are one field per matched kind. + * @SuppressWarnings(PHPMD.CouplingBetweenObjects) Reader, normaliser, profile, report, + * contacts, progress and OpenRegister are the import's collaborators by design. + * @SuppressWarnings(PHPMD.TooManyMethods) Each match rule and each step of a row is its own + * small method; merging them back would only make the steps longer. + * @SuppressWarnings(PHPMD.ExcessiveClassLength) Most of the length is docblocks that state + * the matching rules; the code itself is under the threshold. + */ +class CmdbExportImportService { + /** + * The ProgressTracker operation type of an import. + */ + public const OPERATION_TYPE = 'cmdb_import'; + + /** + * Operation ids a client may choose; anything else gets a generated id. + */ + public const OPERATION_ID_PATTERN = '/^cmdb-[A-Za-z0-9-]{8,64}$/'; + + /** + * OpenRegister's migration-pack mapping engine (not a public contract). + */ + public const ENGINE_CLASS = 'OCA\OpenRegister\Service\MigrationPack\MappingEngine'; + + /** + * Page size for loading the organisations a name may match. + */ + private const PAGE_SIZE = 500; + + /** + * Separator of the concat mapping for the internal note. + */ + private const NOTE_SEPARATOR = ' / '; + + /** + * The mapping engine of the current run. + * + * @var object|null + */ + private ?object $engine = null; + + /** + * OpenRegister and the register/schema ids of the current run. + * + * @var array{objectService: ObjectServiceInterface, register: int, module: int, organization: int, usage: int, contactPerson: int}|null + */ + private ?array $coordinates = null; + + /** + * Suppliers by normalised name, loaded once per run. + * + * @var array|null + */ + private ?array $suppliers = null; + + /** + * APPIDs seen in this upload. + * + * @var array + */ + private array $seenKeys = []; + + /** + * Contact UIDs by e-mail or display name, per run. + * + * @var array + */ + private array $contactUids = []; + + /** + * Contact person uuids by contactsUid and organisation, per run. + * + * @var array + */ + private array $contactPersons = []; + + /** + * Constructor. + * + * @param ContainerInterface $container Resolves OpenRegister services. + * @param SettingsService $settingsService Register and schema ids. + * @param StackiqContactSyncService $contactSync Nextcloud Contacts bridge. + * @param ProgressTracker $progressTracker Progress and cancel. + * @param CmdbImportProfile $profile The import profile and packs. + * @param CmdbWorkbookReader $reader The xlsx reader. + * @param CmdbRowNormaliser $normaliser Dates and ids to strings. + * @param IL10N $l10n Translates report reasons and warnings. + * @param LoggerInterface $logger Logger; never handed person data. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public function __construct( + private readonly ContainerInterface $container, + private readonly SettingsService $settingsService, + private readonly StackiqContactSyncService $contactSync, + private readonly ProgressTracker $progressTracker, + private readonly CmdbImportProfile $profile, + private readonly CmdbWorkbookReader $reader, + private readonly CmdbRowNormaliser $normaliser, + private readonly IL10N $l10n, + private readonly LoggerInterface $logger, + ) { + }//end __construct() + + /** + * The upload limit in bytes (the profile's `maxFileBytes`). + * + * @return int + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + public function maxFileBytes(): int { + return $this->profile->maxFileBytes(); + }//end maxFileBytes() + + /** + * Check that an upload is an xlsx workbook, without parsing it. + * + * @param string $path The uploaded temporary file. + * @param string $fileName The original file name. + * + * @return void + * + * @throws CmdbImportException NOT_XLSX. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + public function assertXlsx(string $path, string $fileName): void { + $this->reader->assertXlsx(path: $path, fileName: $fileName); + }//end assertXlsx() + + /** + * Whether a `missingRecords` value is supported (only `keep`). + * + * @param string $mode The requested value. + * + * @return bool + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + */ + public function supportsMissingRecords(string $mode): bool { + return $mode === 'keep'; + }//end supportsMissingRecords() + + /** + * Ask a running import to stop between rows. + * + * @param string $operationId The operation id. + * + * @return bool False when no `cmdb_import` operation has this id. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function requestCancel(string $operationId): bool { + if (preg_match(self::OPERATION_ID_PATTERN, $operationId) !== 1) { + return false; + } + + $progress = $this->progressTracker->getProgress(operationId: $operationId); + if (is_array($progress) === false || ($progress['operation_type'] ?? null) !== self::OPERATION_TYPE) { + return false; + } + + $this->progressTracker->setCancelRequested(operationId: $operationId); + return true; + }//end requestCancel() + + /** + * Import an export for one municipality. + * + * Validation that can fail the whole import runs before any object is + * written: the packs and the engine, the configuration, the workbook and + * the municipality uuid. After that every row is processed in its own + * error boundary. + * + * @param string $path The xlsx file, already checked by assertXlsx(). + * @param array $options municipalityUuid, municipalityName, updateExisting, operationId. + * + * @return array The report (contract.md). + * + * @throws CmdbImportException MAPPING_UNAVAILABLE, NOT_CONFIGURED, READER_UNAVAILABLE, NOT_XLSX, + * NO_SOURCE_SHEET, MISSING_COLUMN, TOO_MANY_ROWS, MUNICIPALITY_REQUIRED + * or MUNICIPALITY_INVALID. + * @throws \Exception An unexpected OpenRegister error outside a row, such as + * creating the municipality; rows catch their own. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + public function import(string $path, array $options): array { + $this->resetRun(); + $startedAt = (new DateTimeImmutable('now', new DateTimeZone('UTC')))->format(DATE_ATOM); + + $this->profile->load(); + $this->engine = $this->resolveEngine(); + $this->coordinates = $this->resolveCoordinates(); + + $workbook = $this->reader->read(path: $path, profile: $this->profile); + $municipality = $this->resolveMunicipality(options: $options); + + $operationId = $this->operationIdFrom(options: $options); + $rows = $workbook['rows']; + $report = new CmdbImportReport(operationId: $operationId, rowsRead: count($rows)); + $report->setMunicipality(uuid: $municipality['uuid'], name: $municipality['name'], created: $municipality['created']); + $report->addImportWarnings(warnings: $this->translateImportWarnings(warnings: $workbook['importWarnings'])); + + $this->progressTracker->startOperation( + operationType: self::OPERATION_TYPE, + options: ['total_items' => count($rows)], + operationId: $operationId + ); + $this->progressTracker->setPhase(phase: 'processing_elements', data: ['total_items' => count($rows)]); + + $updateExisting = (($options['updateExisting'] ?? true) !== false); + foreach ($rows as $index => $row) { + if ($this->progressTracker->isCancelRequested(operationId: $operationId) === true) { + $report->markCancelled(); + break; + } + + $this->processRow( + row: $row, + municipalityUuid: $municipality['uuid'], + updateExisting: $updateExisting, + startedAt: $startedAt, + date1904: $workbook['date1904'], + report: $report + ); + $this->progressTracker->updateProgress(processedItems: ($index + 1)); + } + + $result = $report->toArray(); + $this->finishOperation(report: $result); + + $this->logger->info( + 'CmdbExportImportService: import finished', + ['operationId' => $operationId, 'summary' => $result['summary'], 'cancelled' => $result['cancelled']] + ); + + return $result; + }//end import() + + /** + * Process one row in its own error boundary and add its outcome. + * + * @param array{sheet: string, row: int, cells: array, uncached?: array} $row The reader row. + * @param string $municipalityUuid The consumer. + * @param bool $updateExisting Whether matched rows are updated. + * @param string $startedAt ISO start time of the import. + * @param bool $date1904 The workbook's date system. + * @param CmdbImportReport $report The report. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + private function processRow( + array $row, + string $municipalityUuid, + bool $updateExisting, + string $startedAt, + bool $date1904, + CmdbImportReport $report, + ): void { + $sheet = $row['sheet']; + $values = $this->normaliser->normalise( + cells: array_merge($row['cells'], $this->profile->sheetConstants(sheetName: $sheet)), + dateColumns: $this->profile->dateColumns(), + idColumns: $this->profile->idColumns(), + date1904: $date1904, + emptyValues: $this->profile->emptyValues() + ); + $rowNumber = $row['row']; + $appId = ($values[$this->profile->keyColumn()] ?? ''); + $name = ($values[$this->profile->nameColumn()] ?? ''); + + $entry = ['sheet' => $sheet, 'row' => $rowNumber, 'appId' => $appId, 'name' => $name]; + + $warnings = []; + foreach (($row['uncached'] ?? []) as $column) { + $warnings[] = $this->l10n->t('Column "%s": formula without a cached value, read as empty', [(string)$column]); + } + + $skipReason = $this->skipReason(appId: $appId); + if ($skipReason !== null) { + $this->addRow(report: $report, entry: $entry, outcome: CmdbImportReport::SKIPPED, reasons: [$skipReason], warnings: $warnings); + return; + } + + $step = 'mapping'; + $moduleUuid = null; + $usageUuid = null; + + try { + $module = $this->map(target: 'module', values: $values, rowNumber: $rowNumber, warnings: $warnings); + if ($module['missing'] !== []) { + $reasons = array_map(fn (string $column): string => $this->l10n->t('missing %s', [$column]), $module['missing']); + $this->addRow(report: $report, entry: $entry, outcome: CmdbImportReport::SKIPPED, reasons: $reasons, warnings: $warnings); + return; + } + + $step = 'manufacturer'; + $providerUuid = $this->resolveManufacturer(values: $values, rowNumber: $rowNumber); + + $step = 'module'; + $externalKey = $this->profile->externalKeyPrefix() . ':' . $municipalityUuid . ':' . $appId; + $moduleResult = $this->upsertModule( + data: $module['data'], + externalKey: $externalKey, + providerUuid: $providerUuid, + startedAt: $startedAt, + updateExisting: $updateExisting + ); + $moduleUuid = $moduleResult['uuid']; + if ($moduleResult['outcome'] === 'exists') { + $this->addRow( + report: $report, + entry: $entry, + outcome: CmdbImportReport::SKIPPED, + reasons: [$this->l10n->t('exists')], + warnings: $warnings, + moduleUuid: $moduleUuid + ); + return; + } + + $step = 'owners'; + $owners = $this->resolveOwners(values: $values, rowNumber: $rowNumber, municipalityUuid: $municipalityUuid, warnings: $warnings); + + $step = 'usage'; + $usage = $this->map(target: 'usage', values: $values, rowNumber: $rowNumber, warnings: $warnings); + $usageResult = $this->upsertUsage( + data: $usage['data'], + municipalityUuid: $municipalityUuid, + moduleUuid: $moduleUuid, + providerUuid: $providerUuid, + owners: $owners + ); + $usageUuid = $usageResult['uuid']; + } catch (Throwable $e) { + $this->failRow(report: $report, entry: $entry, step: $step, e: $e, warnings: $warnings, uuids: [$moduleUuid, $usageUuid]); + return; + }//end try + + $outcome = self::rowOutcome(module: $moduleResult['outcome'], usage: $usageResult['outcome']); + $this->addRow(report: $report, entry: $entry, outcome: $outcome, warnings: $warnings, moduleUuid: $moduleUuid, usageUuid: $usageUuid); + }//end processRow() + + /** + * Report a row as failed at a step, and log it without person data. + * + * @param CmdbImportReport $report The report. + * @param array{sheet: string, row: int, appId: string, name: string} $entry Where the row is. + * @param string $step The step that failed. + * @param Throwable $e The cause. + * @param array $warnings Row warnings so far. + * @param array{0: string|null, 1: string|null} $uuids Module and usage, when saved. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + private function failRow(CmdbImportReport $report, array $entry, string $step, Throwable $e, array $warnings, array $uuids): void { + $detail = $this->safeMessage(step: $step, e: $e); + $this->logger->warning( + 'CmdbExportImportService: row failed', + array_merge( + ['sheet' => $entry['sheet'], 'row' => $entry['row'], 'appId' => $entry['appId']], + ['step' => $step, 'exception' => get_class($e), 'error' => $detail] + ) + ); + + $reason = $this->l10n->t('step "%s" failed', [$step]); + if ($detail !== '') { + $reason = $this->l10n->t('step "%1$s" failed: %2$s', [$step, $detail]); + } + + $this->addRow( + report: $report, + entry: $entry, + outcome: CmdbImportReport::FAILED, + reasons: [$reason], + warnings: $warnings, + moduleUuid: $uuids[0], + usageUuid: $uuids[1] + ); + }//end failRow() + + /** + * Translate the reader's import-level warnings. + * + * @param array $warnings The reader warnings. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + private function translateImportWarnings(array $warnings): array { + $translated = []; + foreach ($warnings as $warning) { + $message = $warning['message']; + if (isset($warning['column']) === true) { + $message = $this->l10n->t('Optional column "%s" not found', [$warning['column']]); + } + + $translated[] = ['sheet' => $warning['sheet'], 'message' => $message]; + } + + return $translated; + }//end translateImportWarnings() + + /** + * The row outcome from the module and usage outcomes. + * + * @param string $module The module outcome. + * @param string $usage The usage outcome. + * + * @return string created when the module was created, updated when anything was saved, else unchanged. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + private static function rowOutcome(string $module, string $usage): string { + if ($module === CmdbImportReport::CREATED) { + return CmdbImportReport::CREATED; + } + + if ($module !== CmdbImportReport::UNCHANGED || $usage !== CmdbImportReport::UNCHANGED) { + return CmdbImportReport::UPDATED; + } + + return CmdbImportReport::UNCHANGED; + }//end rowOutcome() + + /** + * Store the report with the operation and close it, as completed or cancelled. + * + * @param array $report The report. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + private function finishOperation(array $report): void { + if ($report['cancelled'] === true) { + $this->progressTracker->updateStatistics(statistics: ['report' => $report]); + $this->progressTracker->cancelOperation(); + return; + } + + $this->progressTracker->completeOperation(finalStatistics: ['report' => $report]); + }//end finishOperation() + + /** + * Add a row outcome to the report. + * + * @param CmdbImportReport $report The report. + * @param array{sheet: string, row: int, appId: string, name: string} $entry Where the row is. + * @param string $outcome The outcome. + * @param array $reasons Reasons. + * @param array $warnings Warnings. + * @param string|null $moduleUuid The module. + * @param string|null $usageUuid The usage. + * + * @return void + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + private function addRow( + CmdbImportReport $report, + array $entry, + string $outcome, + array $reasons = [], + array $warnings = [], + ?string $moduleUuid = null, + ?string $usageUuid = null, + ): void { + $report->addRow( + sheet: $entry['sheet'], + row: $entry['row'], + appId: $entry['appId'], + name: $entry['name'], + outcome: $outcome, + reasons: $reasons, + warnings: $warnings, + moduleUuid: $moduleUuid, + usageUuid: $usageUuid + ); + }//end addRow() + + /** + * Why a row is skipped before mapping, or null when it is imported. + * + * An APPID seen earlier in the same upload, on either sheet, is a duplicate. + * + * @param string $appId The APPID. + * + * @return string|null + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-7 + */ + private function skipReason(string $appId): ?string { + if ($appId === '') { + return $this->l10n->t('missing %s', [$this->profile->keyColumn()]); + } + + if (isset($this->seenKeys[$appId]) === true) { + return $this->l10n->t('duplicate %s in file', [$this->profile->keyColumn()]); + } + + $this->seenKeys[$appId] = true; + + return null; + }//end skipReason() + + /** + * Map a row through a target's pack. + * + * Errors on required mappings are returned as missing columns. Other + * errors drop the field and become a warning naming the column and the + * value, except for the owner and manufacturer packs, whose errors are + * silent: such a row simply has no owner or no manufacturer. + * + * @param string $target The pack target. + * @param array $values The normalised row. + * @param int $rowNumber The sheet row number, for the engine's errors. + * @param array $warnings Row warnings, appended to. + * + * @return array{data: array, missing: array} + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function map(string $target, array $values, int $rowNumber, array &$warnings): array { + $pack = $this->profile->pack(target: $target); + $result = $this->engine->mapRow($pack, $values, $rowNumber); + + $required = []; + foreach (($pack['fieldMappings'] ?? []) as $mapping) { + if (($mapping['required'] ?? false) === true) { + $required[] = (string)($mapping['source'] ?? ''); + } + } + + $silent = in_array($target, ['manufacturer', 'businessOwner', 'municipality'], true); + $missing = []; + foreach (($result['errors'] ?? []) as $error) { + $source = (string)($error['source'] ?? ''); + if (in_array($source, $required, true) === true) { + $missing[] = $source; + continue; + } + + if ($silent === false) { + $warnings[] = $this->l10n->t('Column "%1$s": %2$s', [$source, (string)($error['message'] ?? '')]); + } + } + + $data = ($result['data'] ?? []); + unset($data['id']); + + return ['data' => $data, 'missing' => array_values(array_unique($missing))]; + }//end map() + + /** + * Find or create the Supplier organisation for the row's manufacturer. + * + * @param array $values The normalised row. + * @param int $rowNumber The sheet row number. + * + * @return string|null The supplier uuid, or null when the row names no manufacturer. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function resolveManufacturer(array $values, int $rowNumber): ?string { + $warnings = []; + $mapped = $this->map(target: 'manufacturer', values: $values, rowNumber: $rowNumber, warnings: $warnings); + $name = trim((string)($mapped['data']['name'] ?? '')); + if ($mapped['missing'] !== [] || $name === '') { + return null; + } + + $key = self::normaliseName(name: $name); + $suppliers = $this->suppliers(); + if (isset($suppliers[$key]) === true) { + return $suppliers[$key]; + } + + $data = $mapped['data']; + $data['name'] = (string)preg_replace('/\s+/u', ' ', $name); + $uuid = $this->save(schemaKey: 'organization', data: $data, uuid: null); + $this->suppliers[$key] = $uuid; + + return $uuid; + }//end resolveManufacturer() + + /** + * Create, update, or leave the module matched on its external key. + * + * @param array $data The mapped module fields. + * @param string $externalKey The match key. + * @param string|null $providerUuid The supplier, when there is one. + * @param string $startedAt ISO start time of the import. + * @param bool $updateExisting Whether a match is updated. + * + * @return array{uuid: string, outcome: string} Outcome created, updated, unchanged or exists. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function upsertModule(array $data, string $externalKey, ?string $providerUuid, string $startedAt, bool $updateExisting): array { + $data['externalKey'] = $externalKey; + if ($providerUuid !== null) { + $data['provider'] = $providerUuid; + } + + $existing = $this->findOne(schemaKey: 'module', filters: ['externalKey' => $externalKey]); + if ($existing === null) { + $create = array_merge($this->profile->createOnlyDefaults(target: 'module'), $data); + $create['publicationDate'] = $startedAt; + return ['uuid' => $this->save(schemaKey: 'module', data: $create, uuid: null), 'outcome' => CmdbImportReport::CREATED]; + } + + $uuid = (string)$existing->getUuid(); + if ($updateExisting === false) { + return ['uuid' => $uuid, 'outcome' => 'exists']; + } + + $merged = $this->merge(target: 'module', stored: $existing->getObject(), mapped: $data); + if ($merged === null) { + return ['uuid' => $uuid, 'outcome' => CmdbImportReport::UNCHANGED]; + } + + $this->save(schemaKey: 'module', data: $merged, uuid: $uuid); + return ['uuid' => $uuid, 'outcome' => CmdbImportReport::UPDATED]; + }//end upsertModule() + + /** + * Create, update, or leave the usage of the module for the municipality. + * + * @param array $data The mapped usage fields. + * @param string $municipalityUuid The consumer. + * @param string $moduleUuid The module. + * @param string|null $providerUuid The supplier, when there is one. + * @param array{businessOwner: string|null} $owners The owner contact person. + * + * @return array{uuid: string, outcome: string} + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function upsertUsage(array $data, string $municipalityUuid, string $moduleUuid, ?string $providerUuid, array $owners): array { + if (isset($data['interneAnnotation']) === true && is_string($data['interneAnnotation']) === true) { + // The concat mapping leaves an empty part for every empty column; drop those. + $parts = array_filter( + array_map('trim', explode(self::NOTE_SEPARATOR, $data['interneAnnotation'])), + fn (string $part): bool => $part !== '' + ); + $data['interneAnnotation'] = implode(self::NOTE_SEPARATOR, $parts); + } + + $data['consumer'] = $municipalityUuid; + $data['module'] = $moduleUuid; + if ($providerUuid !== null) { + $data['provider'] = $providerUuid; + } + + foreach ($owners as $field => $contactPersonUuid) { + if ($contactPersonUuid !== null) { + $data[$field] = $contactPersonUuid; + } + } + + $existing = $this->findOne(schemaKey: 'usage', filters: ['consumer' => $municipalityUuid, 'module' => $moduleUuid]); + if ($existing === null) { + return ['uuid' => $this->save(schemaKey: 'usage', data: $data, uuid: null), 'outcome' => CmdbImportReport::CREATED]; + } + + $uuid = (string)$existing->getUuid(); + $merged = $this->merge(target: 'usage', stored: $existing->getObject(), mapped: $data); + if ($merged === null) { + return ['uuid' => $uuid, 'outcome' => CmdbImportReport::UNCHANGED]; + } + + $this->save(schemaKey: 'usage', data: $merged, uuid: $uuid); + return ['uuid' => $uuid, 'outcome' => CmdbImportReport::UPDATED]; + }//end upsertUsage() + + /** + * Merge mapped fields onto a stored object. + * + * Fields the pack does not map stay as they are. Create-only fields are + * written only when the stored value is empty; never-written fields are + * never touched. + * + * @param string $target The profile target. + * @param array $stored The stored object data. + * @param array $mapped The mapped fields. + * + * @return array|null The merged object, or null when nothing changes. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function merge(string $target, array $stored, array $mapped): ?array { + $createOnly = $this->profile->createOnlyFields(target: $target); + $never = $this->profile->neverWrittenOnUpdate(target: $target); + $merged = $stored; + unset($merged['@self']); + $changed = false; + + foreach ($mapped as $field => $value) { + if (in_array($field, $never, true) === true) { + continue; + } + + $current = ($stored[$field] ?? null); + if (in_array($field, $createOnly, true) === true && self::isEmptyValue(value: $current) === false) { + continue; + } + + if (self::sameValue(stored: $current, value: $value) === true) { + continue; + } + + $merged[$field] = $value; + $changed = true; + } + + if ($changed === false) { + return null; + } + + return $merged; + }//end merge() + + /** + * Resolve the owner of a row (Applicatie Eigenaar) as the usage's business owner. + * + * No technical owner is read: the functional administrator columns are + * not part of the mapping. + * + * @param array $values The normalised row. + * @param int $rowNumber The sheet row number. + * @param string $municipalityUuid The municipality the contact person belongs to. + * @param array $warnings Row warnings, appended to. + * + * @return array{businessOwner: string|null} + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-6 + */ + private function resolveOwners(array $values, int $rowNumber, string $municipalityUuid, array &$warnings): array { + $owners = ['businessOwner' => null]; + $identities = []; + foreach (array_keys($owners) as $target) { + $silent = []; + $mapped = $this->map(target: $target, values: $values, rowNumber: $rowNumber, warnings: $silent); + $name = trim((string)($mapped['data']['name'] ?? '')); + if ($mapped['missing'] === [] && $name !== '') { + $identities[$target] = $mapped['data']; + } + } + + if ($identities === []) { + return $owners; + } + + if ($this->contactSync->isAvailable() === false) { + $warnings[] = $this->l10n->t('Owners skipped: Nextcloud Contacts is unavailable'); + return $owners; + } + + foreach ($identities as $target => $identity) { + $column = $this->ownerColumn(target: $target); + try { + $contactsUid = $this->resolveContactUid(identity: $identity); + if ($contactsUid === null) { + $warnings[] = $this->l10n->t('Owner from column "%s" could not be resolved in Nextcloud Contacts', [$column]); + continue; + } + + $owners[$target] = $this->resolveContactPerson( + contactsUid: $contactsUid, + municipalityUuid: $municipalityUuid, + role: trim((string)($identity['role'] ?? '')) + ); + } catch (Throwable $e) { + $this->logger->warning( + 'CmdbExportImportService: owner could not be resolved', + ['row' => $rowNumber, 'column' => $column, 'exception' => get_class($e)] + ); + $warnings[] = $this->l10n->t('Owner from column "%s" could not be resolved', [$column]); + } + }//end foreach + + return $owners; + }//end resolveOwners() + + /** + * Resolve the Nextcloud contact of an owner identity. + * + * With an e-mail address, StackiqContactSyncService matches on it or + * creates the contact. Without one, only a contact whose display name is + * exactly the owner's name (case-insensitive) is reused, so an owner + * known by name alone is not created again on every import. + * + * @param array $identity name, email and role from the owner pack. + * + * @return string|null The contact UID, or null. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-6 + */ + private function resolveContactUid(array $identity): ?string { + $parts = self::splitPersonName(name: (string)$identity['name']); + $displayName = trim($parts['voornaam'] . ' ' . $parts['achternaam']); + $email = trim((string)($identity['email'] ?? '')); + + $cacheKey = 'name:' . mb_strtolower($displayName); + if ($email !== '') { + $cacheKey = 'email:' . mb_strtolower($email); + } + + if (array_key_exists($cacheKey, $this->contactUids) === true) { + return $this->contactUids[$cacheKey]; + } + + $uid = null; + if ($email === '') { + $uid = $this->contactByDisplayName(displayName: $displayName); + } + + if ($uid === null) { + $record = ['voornaam' => $parts['voornaam'], 'achternaam' => $parts['achternaam']]; + if ($email !== '') { + $record['email'] = $email; + } + + $role = trim((string)($identity['role'] ?? '')); + if ($role !== '') { + $record['role'] = $role; + } + + $uid = $this->contactSync->syncToContacts(objectType: 'contactPerson', record: $record); + if ($uid === '') { + $uid = null; + } + } + + $this->contactUids[$cacheKey] = $uid; + return $uid; + }//end resolveContactUid() + + /** + * The contact whose display name is exactly this one, case-insensitive. + * + * @param string $displayName The display name. + * + * @return string|null The contact UID, or null. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-6 + */ + private function contactByDisplayName(string $displayName): ?string { + $needle = mb_strtolower($displayName); + foreach ($this->contactSync->searchContacts(query: $displayName) as $contact) { + if (mb_strtolower(trim((string)($contact['name'] ?? ''))) === $needle) { + return (string)$contact['uid']; + } + } + + return null; + }//end contactByDisplayName() + + /** + * Find or create the contact person of a contact for the municipality. + * + * The object carries only `contactsUid`, `organization` and `role`: no + * e-mail and no username, so neither the contact-person listener nor + * OrganizationSyncService::performUserSync() provisions a user for it. + * + * @param string $contactsUid The Nextcloud contact UID. + * @param string $municipalityUuid The organisation. + * @param string $role The owner's function, or ''. + * + * @return string The contact person uuid. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-6 + */ + private function resolveContactPerson(string $contactsUid, string $municipalityUuid, string $role): string { + $cacheKey = $contactsUid . '|' . $municipalityUuid; + if (isset($this->contactPersons[$cacheKey]) === true) { + return $this->contactPersons[$cacheKey]; + } + + $existing = $this->findOne(schemaKey: 'contactPerson', filters: ['contactsUid' => $contactsUid, 'organization' => $municipalityUuid]); + $data = ['contactsUid' => $contactsUid, 'organization' => $municipalityUuid]; + if ($role !== '') { + $data['role'] = $role; + } + + $uuid = (string)$existing?->getUuid(); + if ($existing === null) { + $uuid = $this->save(schemaKey: 'contactPerson', data: $data, uuid: null); + } + + $this->contactPersons[$cacheKey] = $uuid; + return $uuid; + }//end resolveContactPerson() + + /** + * Resolve the consuming municipality from the options. + * + * @param array $options municipalityUuid or municipalityName. + * + * @return array{uuid: string, name: string, created: bool} + * + * @throws CmdbImportException MUNICIPALITY_REQUIRED or MUNICIPALITY_INVALID. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function resolveMunicipality(array $options): array { + $uuid = trim((string)($options['municipalityUuid'] ?? '')); + if ($uuid !== '') { + return $this->municipalityByUuid(uuid: $uuid); + } + + $warnings = []; + $values = ['municipalityName' => trim((string)($options['municipalityName'] ?? ''))]; + $mapped = $this->map(target: 'municipality', values: $values, rowNumber: 0, warnings: $warnings); + $name = trim((string)preg_replace('/\s+/u', ' ', (string)($mapped['data']['name'] ?? ''))); + if ($mapped['missing'] !== [] || $name === '') { + throw new CmdbImportException(errorCode: CmdbImportException::MUNICIPALITY_REQUIRED, message: 'No municipality given'); + } + + $key = self::normaliseName(name: $name); + foreach ($this->organisationsOfType(type: 'Municipality') as $organisation) { + if (self::normaliseName(name: (string)($organisation['name'] ?? '')) === $key) { + return ['uuid' => $organisation['uuid'], 'name' => (string)$organisation['name'], 'created' => false]; + } + } + + $data = $mapped['data']; + $data['name'] = $name; + $created = $this->save(schemaKey: 'organization', data: $data, uuid: null); + + return ['uuid' => $created, 'name' => $name, 'created' => true]; + }//end resolveMunicipality() + + /** + * Resolve a municipality uuid, which must be an organisation of type Municipality. + * + * @param string $uuid The organisation uuid. + * + * @return array{uuid: string, name: string, created: bool} + * + * @throws CmdbImportException MUNICIPALITY_INVALID. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function municipalityByUuid(string $uuid): array { + $coordinates = $this->coordinates(); + $organisation = null; + try { + $organisation = $coordinates['objectService']->find( + id: $uuid, + register: $coordinates['register'], + schema: $coordinates['organization'], + _rbac: false, + _multitenancy: false + ); + } catch (Throwable $e) { + // OpenRegister throws DoesNotExistException for an unknown uuid. + $organisation = null; + } + + $data = []; + if ($organisation !== null) { + $data = $organisation->getObject(); + } + + if ($organisation === null || ($data['type'] ?? null) !== 'Municipality') { + throw new CmdbImportException( + errorCode: CmdbImportException::MUNICIPALITY_INVALID, + message: 'The municipality uuid is not an organisation of type Municipality' + ); + } + + return ['uuid' => (string)$organisation->getUuid(), 'name' => (string)($data['name'] ?? ''), 'created' => false]; + }//end municipalityByUuid() + + /** + * Suppliers by normalised name, loaded once per run. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function suppliers(): array { + if ($this->suppliers === null) { + $this->suppliers = []; + foreach ($this->organisationsOfType(type: 'Supplier') as $organisation) { + $key = self::normaliseName(name: (string)($organisation['name'] ?? '')); + if ($key !== '' && isset($this->suppliers[$key]) === false) { + $this->suppliers[$key] = $organisation['uuid']; + } + } + } + + return $this->suppliers; + }//end suppliers() + + /** + * Every organisation of a type, as uuid and name. + * + * @param string $type Municipality or Supplier. + * + * @return array + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function organisationsOfType(string $type): array { + $coordinates = $this->coordinates(); + $found = []; + $offset = 0; + do { + $page = $coordinates['objectService']->searchObjects( + query: [ + '@self' => ['register' => $coordinates['register'], 'schema' => $coordinates['organization']], + 'type' => $type, + '_limit' => self::PAGE_SIZE, + '_offset' => $offset, + ], + _rbac: false, + _multitenancy: false + ); + if (is_array($page) === false) { + $page = []; + } + + foreach ($page as $entity) { + $data = $entity->getObject(); + // The filter is checked again: a filter OpenRegister cannot apply must not widen the match. + if (($data['type'] ?? null) === $type && $entity->getUuid() !== null) { + $found[] = ['uuid' => (string)$entity->getUuid(), 'name' => ($data['name'] ?? '')]; + } + } + + $offset += self::PAGE_SIZE; + $pageSize = count($page); + } while ($pageSize === self::PAGE_SIZE); + + return $found; + }//end organisationsOfType() + + /** + * The one object matching every filter, or null. + * + * @param string $schemaKey The coordinates key of the schema. + * @param array $filters Field => exact value. + * + * @return object|null The entity (ObjectEntityInterface). + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function findOne(string $schemaKey, array $filters): ?object { + $coordinates = $this->coordinates(); + $results = $coordinates['objectService']->searchObjects( + query: array_merge( + ['@self' => ['register' => $coordinates['register'], 'schema' => $coordinates[$schemaKey]], '_limit' => 10], + $filters + ), + _rbac: false, + _multitenancy: false + ); + if (is_array($results) === false) { + return null; + } + + foreach ($results as $entity) { + $data = $entity->getObject(); + $matches = true; + foreach ($filters as $field => $value) { + if (self::relationUuid(value: ($data[$field] ?? null)) !== $value) { + $matches = false; + break; + } + } + + if ($matches === true) { + return $entity; + } + } + + return null; + }//end findOne() + + /** + * Save an object through OpenRegister and return its uuid. + * + * @param string $schemaKey The coordinates key of the schema. + * @param array $data The object data. + * @param string|null $uuid The uuid to update, or null to create. + * + * @return string The uuid. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function save(string $schemaKey, array $data, ?string $uuid): string { + $coordinates = $this->coordinates(); + $entity = $coordinates['objectService']->saveObject( + object: $data, + register: $coordinates['register'], + schema: $coordinates[$schemaKey], + uuid: $uuid, + _rbac: false, + _multitenancy: false + ); + + return (string)$entity->getUuid(); + }//end save() + + /** + * Resolve OpenRegister's mapping engine. + * + * @return object The engine, with `mapRow(array, array, int): array`. + * + * @throws CmdbImportException MAPPING_UNAVAILABLE. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + */ + private function resolveEngine(): object { + $class = static::ENGINE_CLASS; + try { + if ($this->container->has($class) === true) { + $engine = $this->container->get($class); + if (is_object($engine) === true && method_exists($engine, 'mapRow') === true) { + return $engine; + } + } + } catch (Throwable $e) { + // Fall through to the class check below. + $engine = null; + } + + if (class_exists($class) === true) { + $engine = new $class(); + if (method_exists($engine, 'mapRow') === true) { + return $engine; + } + } + + throw new CmdbImportException(errorCode: CmdbImportException::MAPPING_UNAVAILABLE, message: 'OpenRegister MappingEngine is not available'); + }//end resolveEngine() + + /** + * Resolve OpenRegister and the register and schema ids, failing closed. + * + * @return array{objectService: ObjectServiceInterface, register: int, module: int, organization: int, usage: int, contactPerson: int} + * + * @throws CmdbImportException NOT_CONFIGURED when OpenRegister or a schema is not configured. + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + private function resolveCoordinates(): array { + try { + $objectService = $this->container->get(ObjectServiceInterface::class); + } catch (Throwable $e) { + $objectService = null; + } + + $register = (int)($this->settingsService->getVoorzieningenConfig()['register'] ?? 0); + $schemas = []; + foreach (['module', 'organization', 'usage', 'contactPerson'] as $type) { + $schemas[$type] = (int)($this->settingsService->getSchemaIdForObjectType($type) ?? 0); + } + + if ($objectService instanceof ObjectServiceInterface === false || $register <= 0 || in_array(0, $schemas, true) === true) { + throw new CmdbImportException( + errorCode: CmdbImportException::NOT_CONFIGURED, + message: 'OpenRegister or the stackiq register and schemas are not configured' + ); + } + + return array_merge(['objectService' => $objectService, 'register' => $register], $schemas); + }//end resolveCoordinates() + + /** + * The coordinates of the current run. + * + * @return array{objectService: ObjectServiceInterface, register: int, module: int, organization: int, usage: int, contactPerson: int} + * + * @throws CmdbImportException NOT_CONFIGURED. + */ + private function coordinates(): array { + if ($this->coordinates === null) { + $this->coordinates = $this->resolveCoordinates(); + } + + return $this->coordinates; + }//end coordinates() + + /** + * The operation id the client chose, or a new one. + * + * @param array $options The import options. + * + * @return string + */ + private function operationIdFrom(array $options): string { + $operationId = $options['operationId'] ?? null; + if (is_string($operationId) === true && preg_match(self::OPERATION_ID_PATTERN, $operationId) === 1) { + return $operationId; + } + + return 'cmdb-' . bin2hex(random_bytes(16)); + }//end operationIdFrom() + + /** + * Clear the run caches. + * + * @return void + */ + private function resetRun(): void { + $this->engine = null; + $this->coordinates = null; + $this->suppliers = null; + $this->seenKeys = []; + $this->contactUids = []; + $this->contactPersons = []; + }//end resetRun() + + /** + * The source column of an owner target, for warnings. + * + * @param string $target businessOwner. + * + * @return string + */ + private function ownerColumn(string $target): string { + foreach (($this->profile->pack(target: $target)['fieldMappings'] ?? []) as $mapping) { + if (($mapping['target'] ?? null) === 'name') { + return (string)($mapping['source'] ?? $target); + } + } + + return $target; + }//end ownerColumn() + + /** + * An exception message that is safe for the report. + * + * Owner steps get no detail, so no contact data can leak into the report. + * + * @param string $step The step that failed. + * @param Throwable $e The exception. + * + * @return string + */ + private function safeMessage(string $step, Throwable $e): string { + if ($step === 'owners') { + return ''; + } + + return mb_substr(trim($e->getMessage()), 0, 300); + }//end safeMessage() + + /** + * Split a TOPdesk person name ("Achternaam, Voornaam"). + * + * @param string $name The name. + * + * @return array{voornaam: string, achternaam: string} + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-6 + */ + public static function splitPersonName(string $name): array { + $name = trim((string)preg_replace('/\s+/u', ' ', $name)); + if (str_contains($name, ',') === true) { + [$last, $first] = array_map('trim', explode(',', $name, 2)); + return ['voornaam' => $first, 'achternaam' => $last]; + } + + return ['voornaam' => '', 'achternaam' => $name]; + }//end splitPersonName() + + /** + * Normalise an organisation name for matching: trim, collapse whitespace, lower case. + * + * @param string $name The name. + * + * @return string + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + */ + public static function normaliseName(string $name): string { + return mb_strtolower(trim((string)preg_replace('/\s+/u', ' ', $name))); + }//end normaliseName() + + /** + * A relation value (uuid string, or array/object with uuid or id) as a string. + * + * @param mixed $value The stored value. + * + * @return string|null + */ + private static function relationUuid(mixed $value): ?string { + if (is_scalar($value) === true) { + return (string)$value; + } + + if (is_array($value) === true) { + $uuid = ($value['uuid'] ?? ($value['id'] ?? null)); + if (is_scalar($uuid) === true) { + return (string)$uuid; + } + } + + return null; + }//end relationUuid() + + /** + * Whether a stored value equals a mapped value. + * + * @param mixed $stored The stored value. + * @param mixed $value The mapped value. + * + * @return bool + */ + private static function sameValue(mixed $stored, mixed $value): bool { + if (is_scalar($value) === true && (is_scalar($stored) === true || is_array($stored) === true)) { + return self::relationUuid(value: $stored) === (string)$value; + } + + return $stored === $value; + }//end sameValue() + + /** + * Whether a stored value counts as empty. + * + * @param mixed $value The stored value. + * + * @return bool + */ + private static function isEmptyValue(mixed $value): bool { + return $value === null || $value === '' || $value === []; + }//end isEmptyValue() +}//end class diff --git a/lib/Settings/cmdb-import/topdesk-business-owner.json b/lib/Settings/cmdb-import/topdesk-business-owner.json new file mode 100644 index 000000000..f987d51ee --- /dev/null +++ b/lib/Settings/cmdb-import/topdesk-business-owner.json @@ -0,0 +1,12 @@ +{ + "id": "stackiq-topdesk-business-owner", + "name": "TOPdesk CMDB export to business owner identity", + "description": "The application owner (Applicatie Eigenaar (Persoon)); the value may be a function instead of a name and is used as the display name either way. The import resolves it in Nextcloud Contacts by exact display name and links a contactPerson of the municipality as usage.businessOwner, with the owner's function as its role. No other person column is read.", + "sourceFormat": "excel", + "version": "2.0.0", + "fieldMappings": [ + { "source": "Applicatie Eigenaar (Persoon)", "target": "name", "required": true, "transform": { "type": "trim" } }, + { "source": "Applicatie Eigenaar (Functie)", "target": "role", "transform": { "type": "trim" } } + ], + "idStrategy": { "type": "generate" } +} diff --git a/lib/Settings/cmdb-import/topdesk-manufacturer.json b/lib/Settings/cmdb-import/topdesk-manufacturer.json new file mode 100644 index 000000000..49fd3ad0c --- /dev/null +++ b/lib/Settings/cmdb-import/topdesk-manufacturer.json @@ -0,0 +1,12 @@ +{ + "id": "stackiq-topdesk-manufacturer", + "name": "TOPdesk CMDB export to stackiq supplier organisation", + "description": "The Vendor column (the maker of the software) becomes one organisation of type Supplier per distinct name. An empty Vendor means the row has no provider. Leverancier and Hostingpartij are not read.", + "sourceFormat": "excel", + "version": "2.0.0", + "fieldMappings": [ + { "source": "Vendor", "target": "name", "required": true, "transform": { "type": "trim" } } + ], + "defaults": { "type": "Supplier", "status": "Active" }, + "idStrategy": { "type": "generate" } +} diff --git a/lib/Settings/cmdb-import/topdesk-module.json b/lib/Settings/cmdb-import/topdesk-module.json new file mode 100644 index 000000000..e5580d973 --- /dev/null +++ b/lib/Settings/cmdb-import/topdesk-module.json @@ -0,0 +1,43 @@ +{ + "id": "stackiq-topdesk-module", + "name": "TOPdesk CMDB export to stackiq module", + "description": "One application row of a CMDB sheet becomes a stackiq module. A mapping marked required that fails skips the row; any other failing mapping drops that field with a warning. Nickname and Roepnaam both feed shortDescription; Roepnaam wins when both are filled.", + "sourceFormat": "excel", + "version": "2.0.0", + "fieldMappings": [ + { "source": "Applicatie Naam", "target": "name", "required": true, "transform": { "type": "trim" } }, + { "source": "APPID", "target": "externalNumber", "required": true, "transform": { "type": "trim" } }, + { "source": "Applicatie Code", "target": "externalId", "transform": { "type": "trim" } }, + { "source": "Nickname", "target": "shortDescription", "transform": { "type": "trim" } }, + { "source": "Roepnaam", "target": "shortDescription", "transform": { "type": "trim" } }, + { "source": "Functionele Omschrijving", "target": "longDescription", "transform": { "type": "trim" } }, + { + "source": "Applicatiesoort", + "target": "cloudDienstverleningsmodel", + "transform": { + "type": "lookup", + "map": { + "Saas": ["SaaS"], "SaaS": ["SaaS"], "SAAS": ["SaaS"], "saas": ["SaaS"], + "Paas": ["PaaS"], "PaaS": ["PaaS"], "PAAS": ["PaaS"], + "Iaas": ["IaaS"], "IaaS": ["IaaS"], "IAAS": ["IaaS"], + "On-premise": ["On-premises (self-managed)"], "On-premises": ["On-premises (self-managed)"], "On premise": ["On-premises (self-managed)"] + } + } + }, + { + "source": "BNN Classificatie", + "target": "bbnLevel", + "transform": { + "type": "lookup", + "map": { + "BBN1": "BBN1", "BBN 1": "BBN1", "bbn1": "BBN1", "bbn 1": "BBN1", "BNN1": "BBN1", "BNN 1": "BBN1", + "BBN2": "BBN2", "BBN 2": "BBN2", "bbn2": "BBN2", "bbn 2": "BBN2", "BNN2": "BBN2", "BNN 2": "BBN2", + "BBN3": "BBN3", "BBN 3": "BBN3", "bbn3": "BBN3", "bbn 3": "BBN3", "BNN3": "BBN3", "BNN 3": "BBN3" + } + } + }, + { "source": "Datum", "target": "externalCreatedAt", "transform": { "type": "date", "sourceFormat": "!Y-m-d", "targetFormat": "Y-m-d" } }, + { "source": "Referentie datum wijziging", "target": "externalModifiedAt", "transform": { "type": "date", "sourceFormat": "!Y-m-d", "targetFormat": "Y-m-d" } } + ], + "idStrategy": { "type": "generate" } +} diff --git a/lib/Settings/cmdb-import/topdesk-municipality.json b/lib/Settings/cmdb-import/topdesk-municipality.json new file mode 100644 index 000000000..eb16fe829 --- /dev/null +++ b/lib/Settings/cmdb-import/topdesk-municipality.json @@ -0,0 +1,12 @@ +{ + "id": "stackiq-topdesk-municipality", + "name": "CMDB import options to stackiq municipality", + "description": "Maps the import options row (municipalityName), not a sheet row, to the consuming organisation of type Municipality.", + "sourceFormat": "excel", + "version": "1.0.0", + "fieldMappings": [ + { "source": "municipalityName", "target": "name", "required": true, "transform": { "type": "trim" } } + ], + "defaults": { "type": "Municipality", "status": "Active" }, + "idStrategy": { "type": "generate" } +} diff --git a/lib/Settings/cmdb-import/topdesk-profile.json b/lib/Settings/cmdb-import/topdesk-profile.json new file mode 100644 index 000000000..88ddf79de --- /dev/null +++ b/lib/Settings/cmdb-import/topdesk-profile.json @@ -0,0 +1,44 @@ +{ + "id": "topdesk-cmdb", + "name": "TOPdesk CMDB export", + "version": "2.0.0", + "description": "How stackiq reads a TOPdesk CMDB export (xlsx): the two CMDB sheets, the key and required columns, date and id columns, values that mean empty, the pack per target and the limits. Columns that neither this profile nor a pack names are never read.", + "maxFileBytes": 10485760, + "maxRowsPerSheet": 10000, + "sheets": [ + { + "name": "Onbeh Applicaties CMDB", + "constants": { "Beheer": "Beheer geregeld: nee" }, + "absentColumns": ["Nickname"] + }, + { + "name": "Beheerde Applicaties CMDB", + "constants": { "Beheer": "Beheer geregeld: ja" } + } + ], + "keyColumn": "APPID", + "nameColumn": "Applicatie Naam", + "requiredColumns": ["APPID", "Applicatie Naam"], + "dateColumns": ["Datum", "Referentie datum wijziging", "End-of-Life Functioneel"], + "idColumns": ["APPID"], + "emptyValues": { + "BNN Classificatie": ["NB"], + "End-of-Life Functioneel": ["49675"] + }, + "externalKeyPrefix": "topdesk", + "packs": { + "module": "topdesk-module.json", + "manufacturer": "topdesk-manufacturer.json", + "municipality": "topdesk-municipality.json", + "usage": "topdesk-usage.json", + "businessOwner": "topdesk-business-owner.json" + }, + "createOnly": { + "module": { "type": "Application" }, + "usage": ["interneAnnotation"] + }, + "neverWritten": { + "module": ["publicationDate", "depublicationDate"] + }, + "missingRecords": ["keep"] +} diff --git a/lib/Settings/cmdb-import/topdesk-usage.json b/lib/Settings/cmdb-import/topdesk-usage.json new file mode 100644 index 000000000..12ae459fa --- /dev/null +++ b/lib/Settings/cmdb-import/topdesk-usage.json @@ -0,0 +1,39 @@ +{ + "id": "stackiq-topdesk-usage", + "name": "TOPdesk CMDB export to stackiq usage", + "description": "The usage that links the application to the municipality. The internal note records whether maintenance is arranged (the sheet the row came from), the cluster and the owner's department. An unknown status or TIME value drops the field with a warning.", + "sourceFormat": "excel", + "version": "2.0.0", + "fieldMappings": [ + { + "source": "Applicatie Status", + "target": "status", + "transform": { + "type": "lookup", + "map": { + "In productie": "In production", + "In voorraad": "Planned", + "In ontwikkeling": "Acquisition", + "Uit te faseren": "To be phased out", + "Uitgefaseerd": "Phased out" + } + } + }, + { + "source": "Classificatie", + "target": "timeClassification", + "transform": { + "type": "lookup", + "map": { + "Tolerate": "Tolerate", "Tolereren": "Tolerate", + "Invest": "Invest", "Investeren": "Invest", + "Migrate": "Migrate", "Migreren": "Migrate", + "Eliminate": "Eliminate", "Elimineren": "Eliminate" + } + } + }, + { "source": "End-of-Life Functioneel", "target": "startDateOutPhased", "transform": { "type": "date", "sourceFormat": "!Y-m-d", "targetFormat": "Y-m-d" } }, + { "source": "Beheer", "target": "interneAnnotation", "transform": { "type": "concat", "fields": ["Cluster", "Applicatie Eigenaar (Afdeling)"], "separator": " / " } } + ], + "idStrategy": { "type": "generate" } +} diff --git a/lib/Settings/register.d/topdesk-cmdb-import.json b/lib/Settings/register.d/topdesk-cmdb-import.json new file mode 100644 index 000000000..1f226ed50 --- /dev/null +++ b/lib/Settings/register.d/topdesk-cmdb-import.json @@ -0,0 +1,109 @@ +{ + "components": { + "schemas": { + "module": { + "version": "0.3.5", + "properties": { + "externalId": { + "type": "string", + "title": "Source id", + "description": "The code of the application in the source system it was imported from, such as the TOPdesk Applicatie Code (the Middel-ID). Shown for reference; it can change in the source, so it is not used to match records.", + "maxLength": 100, + "visible": true, + "facetable": false, + "order": 60 + }, + "externalNumber": { + "type": "string", + "title": "Source number", + "description": "The application number in the source system, such as TOPdesk's APPID (ICT Applicatienummer). A repeated CMDB import matches on it, through the import key.", + "maxLength": 50, + "visible": true, + "facetable": false, + "order": 61 + }, + "externalKey": { + "type": "string", + "title": "Import key", + "description": "The key a repeated import matches this application on: topdesk::. Set by the CMDB import; do not edit.", + "maxLength": 200, + "visible": true, + "facetable": false, + "table": { + "default": false + }, + "order": 62 + }, + "externalCreatedAt": { + "type": "string", + "format": "date", + "title": "Created in source", + "description": "The date the application was registered in the source system.", + "visible": true, + "facetable": false, + "order": 63 + }, + "externalModifiedAt": { + "type": "string", + "format": "date", + "title": "Changed in source", + "description": "The date the application was last changed in the source system.", + "visible": true, + "facetable": false, + "order": 64 + } + } + } + }, + "objects": [ + { + "@self": { + "register": "stackiq", + "schema": "module", + "slug": "voorbeeld-zaaksysteem", + "version": "0.0.1" + }, + "name": "Voorbeeld Zaaksysteem", + "type": "Application", + "longDescription": "Registreert en volgt zaken van intake tot archivering.", + "externalId": "APP-00001", + "externalNumber": "101", + "externalCreatedAt": "2023-07-04", + "externalModifiedAt": "2026-07-29", + "bbnLevel": "BBN2" + }, + { + "@self": { + "register": "stackiq", + "schema": "module", + "slug": "voorbeeld-afsprakenplanner", + "version": "0.0.1" + }, + "name": "Voorbeeld Afsprakenplanner", + "type": "Application", + "longDescription": "Laat inwoners online een afspraak maken bij de balie.", + "externalId": "APP-00002", + "externalNumber": "102", + "externalCreatedAt": "2022-03-16", + "externalModifiedAt": "2026-09-01", + "bbnLevel": "BBN1" + }, + { + "@self": { + "register": "stackiq", + "schema": "module", + "slug": "voorbeeld-belastingapplicatie", + "version": "0.0.1" + }, + "name": "Voorbeeld Belastingapplicatie", + "type": "Application", + "longDescription": "Berekent en verstuurt gemeentelijke belastingaanslagen.", + "externalId": "AIA-00003", + "externalNumber": "103", + "externalCreatedAt": "2024-01-15", + "externalModifiedAt": "2026-05-20", + "bbnLevel": "BBN2" + } + ] + } +} diff --git a/openapi.json b/openapi.json index 9746b7596..f0d63eddc 100644 --- a/openapi.json +++ b/openapi.json @@ -7,5 +7,390 @@ "license": { "name": "agpl" } + }, + "paths": { + "/index.php/apps/stackiq/api/cmdb-import": { + "post": { + "operationId": "cmdbImport-import", + "summary": "Import a TOPdesk CMDB export (xlsx) for one municipality", + "description": "Admin-only, CSRF-protected (requesttoken header or OCS-APIRequest: true). Creates or updates modules, supplier organisations, usages and owner contact persons, matched on topdesk::. See openspec/changes/cmdb-export-import/contract.md.", + "tags": [ + "cmdb-import" + ], + "requestBody": { + "required": true, + "content": { + "multipart/form-data": { + "schema": { + "type": "object", + "required": [ + "cmdbFile" + ], + "properties": { + "cmdbFile": { + "type": "string", + "format": "binary", + "description": "The TOPdesk export, .xlsx, at most 10 MB" + }, + "municipalityUuid": { + "type": "string", + "format": "uuid", + "description": "An existing organization of type Municipality; wins over municipalityName" + }, + "municipalityName": { + "type": "string", + "description": "Name of a Municipality to reuse (same normalised name) or create" + }, + "updateExisting": { + "type": "string", + "enum": [ + "true", + "false" + ], + "default": "true", + "description": "false reports matched rows as skipped (exists)" + }, + "missingRecords": { + "type": "string", + "enum": [ + "keep" + ], + "default": "keep", + "description": "Only keep is accepted; mark and remove are reserved" + }, + "operationId": { + "type": "string", + "pattern": "^cmdb-[A-Za-z0-9-]{8,64}$", + "description": "Progress operation id, readable through GET /api/progress/{operationId}" + } + } + } + } + } + }, + "responses": { + "200": { + "description": "The import report", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CmdbImportReport" + } + } + } + }, + "400": { + "description": "NO_FILE_UPLOADED or NOT_XLSX", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CmdbImportError" + } + } + } + }, + "401": { + "description": "Not signed in" + }, + "403": { + "description": "Not a Nextcloud admin" + }, + "412": { + "description": "Missing or invalid CSRF token" + }, + "413": { + "description": "FILE_TOO_LARGE", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CmdbImportError" + } + } + } + }, + "422": { + "description": "MISSING_RECORDS_UNSUPPORTED, MUNICIPALITY_REQUIRED, MUNICIPALITY_INVALID, NO_SOURCE_SHEET, MISSING_COLUMN or TOO_MANY_ROWS", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CmdbImportError" + } + } + } + }, + "500": { + "description": "IMPORT_FAILED", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CmdbImportError" + } + } + } + }, + "503": { + "description": "MAPPING_UNAVAILABLE, READER_UNAVAILABLE or NOT_CONFIGURED", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CmdbImportError" + } + } + } + } + } + } + }, + "/index.php/apps/stackiq/api/cmdb-import/{operationId}/cancel": { + "post": { + "operationId": "cmdbImport-cancel", + "summary": "Ask a running CMDB import to stop between rows", + "description": "Admin-only, CSRF-protected. No body.", + "tags": [ + "cmdb-import" + ], + "parameters": [ + { + "name": "operationId", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Cancel requested", + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "success", + "cancelRequested" + ], + "properties": { + "success": { + "type": "boolean" + }, + "cancelRequested": { + "type": "boolean" + } + } + } + } + } + }, + "403": { + "description": "Not a Nextcloud admin" + }, + "404": { + "description": "OPERATION_NOT_FOUND", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CmdbImportError" + } + } + } + }, + "412": { + "description": "Missing or invalid CSRF token" + } + } + } + } + }, + "components": { + "schemas": { + "CmdbImportReport": { + "type": "object", + "required": [ + "success", + "operationId", + "cancelled", + "municipality", + "summary", + "importWarnings", + "rows" + ], + "properties": { + "success": { + "type": "boolean" + }, + "operationId": { + "type": "string" + }, + "cancelled": { + "type": "boolean" + }, + "municipality": { + "type": "object", + "required": [ + "uuid", + "name", + "created" + ], + "properties": { + "uuid": { + "type": "string" + }, + "name": { + "type": "string" + }, + "created": { + "type": "boolean" + } + } + }, + "summary": { + "type": "object", + "required": [ + "rowsRead", + "processed", + "created", + "updated", + "unchanged", + "skipped", + "failed", + "warnings" + ], + "properties": { + "rowsRead": { + "type": "integer" + }, + "processed": { + "type": "integer" + }, + "created": { + "type": "integer" + }, + "updated": { + "type": "integer" + }, + "unchanged": { + "type": "integer" + }, + "skipped": { + "type": "integer" + }, + "failed": { + "type": "integer" + }, + "warnings": { + "type": "integer" + } + } + }, + "importWarnings": { + "type": "array", + "items": { + "type": "object", + "required": [ + "sheet", + "message" + ], + "properties": { + "sheet": { + "type": "string" + }, + "message": { + "type": "string" + } + } + } + }, + "rows": { + "type": "array", + "items": { + "type": "object", + "required": [ + "sheet", + "row", + "appId", + "name", + "outcome", + "reasons", + "warnings", + "moduleUuid", + "usageUuid" + ], + "properties": { + "sheet": { + "type": "string" + }, + "row": { + "type": "integer" + }, + "appId": { + "type": "string" + }, + "name": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "created", + "updated", + "unchanged", + "skipped", + "failed" + ] + }, + "reasons": { + "type": "array", + "items": { + "type": "string" + } + }, + "warnings": { + "type": "array", + "items": { + "type": "string" + } + }, + "moduleUuid": { + "type": "string", + "nullable": true + }, + "usageUuid": { + "type": "string", + "nullable": true + } + } + } + } + } + }, + "CmdbImportError": { + "type": "object", + "required": [ + "success", + "error", + "message", + "details" + ], + "properties": { + "success": { + "type": "boolean", + "enum": [ + false + ] + }, + "error": { + "type": "string" + }, + "message": { + "type": "string" + }, + "details": { + "type": "object", + "additionalProperties": true + } + } + } + } } -} \ No newline at end of file +} diff --git a/openspec/changes/cmdb-export-import/.openspec.yaml b/openspec/changes/cmdb-export-import/.openspec.yaml new file mode 100644 index 000000000..67206a856 --- /dev/null +++ b/openspec/changes/cmdb-export-import/.openspec.yaml @@ -0,0 +1,2 @@ +schema: conduction +created: 2026-10-01 diff --git a/openspec/changes/cmdb-export-import/contract.md b/openspec/changes/cmdb-export-import/contract.md new file mode 100644 index 000000000..628cc39a4 --- /dev/null +++ b/openspec/changes/cmdb-export-import/contract.md @@ -0,0 +1,116 @@ +# Contract: cmdb-export-import + +## Consumers + +- `stackiq` frontend: the "CMDB import" admin-settings section (`src/views/settings/sections/CmdbImport.vue`) is the only caller of the two new endpoints. +- `opencatalogi` and `portaliq` call no new endpoint. They read the objects the import writes through their existing OpenRegister paths. Their interface is the data shape below: `module.publicationDate` for OpenCatalogi, and `usage.consumer` / `usage.module` for Portaliq. Neither gets owner data anonymously: `usage` and `contactPerson` have no public read rule, and a public `module` refers to them by id only. + +Paths are relative to `/index.php/apps/stackiq`. + +## Endpoints + +### `POST /api/cmdb-import` +**Auth**: Nextcloud session of a Nextcloud admin, plus CSRF `requesttoken` (header or form field). No `NoAdminRequired`, no `NoCSRFRequired`. + +**Request:** `multipart/form-data` + +| Field | Type | Required | Default | Meaning | +|---|---|---|---|---| +| `cmdbFile` | file | yes | | the TOPdesk export, `.xlsx`, at most 10 MB | +| `municipalityUuid` | string (uuid) | one of the two | | an existing `organization` of type Municipality | +| `municipalityName` | string | one of the two | | name of a Municipality to reuse (same normalised name) or create | +| `updateExisting` | `true`/`false` | no | `true` | `false` reports matched rows as skipped (`exists`) | +| `missingRecords` | string | no | `keep` | only `keep` is accepted; `mark` and `remove` are reserved | +| `operationId` | string | no | generated | progress operation id, readable through `GET /api/progress/{operationId}`; `cmdb-` followed by 8 to 64 letters, digits or hyphens (for example `cmdb-` plus a uuid v4). Any other value is replaced by a generated id, returned as `operationId` | + +**Response (200):** +```json +{ + "success": true, + "operationId": "cmdb-00000000-0000-0000-0000-000000000000", + "cancelled": false, + "municipality": { "uuid": "00000000-0000-0000-0000-000000000001", "name": "Gemeente Voorbeeldstad", "created": false }, + "summary": { "rowsRead": 2, "processed": 2, "created": 2, "updated": 0, "unchanged": 0, "skipped": 0, "failed": 0, "warnings": 1 }, + "importWarnings": [], + "rows": [ + { + "sheet": "Onbeh Applicaties CMDB", + "row": 2, + "appId": "1234", + "name": "Aangetekend Mailen", + "outcome": "created", + "reasons": [], + "warnings": ["Column \"Applicatiesoort\": Value \"Webapplicatie\" has no mapping and no default is configured"], + "moduleUuid": "00000000-0000-0000-0000-000000000004", + "usageUuid": "00000000-0000-0000-0000-000000000005" + } + ] +} +``` + +`appId` is the row's APPID, the match key (`''` when the row has none). `outcome` is one of `created`, `updated`, `unchanged`, `skipped`, `failed`. `reasons` and `warnings` are translated strings that name columns and values; a formula cell without a cached value gives the warning `Column "": formula without a cached value, read as empty`. They never contain owner names, e-mail addresses or other person data. `summary.rowsRead` counts the non-empty rows in the workbook; `summary.processed` counts the rows in `rows`, which is lower than `rowsRead` only after a cancel. `summary.warnings` counts row warnings; `importWarnings` are not included. + +**Errors:** +| Code | Condition | +|------|-----------| +| 400 | `NO_FILE_UPLOADED`, `NOT_XLSX` | +| 401 | not signed in (Nextcloud) | +| 403 | not a Nextcloud admin (Nextcloud) | +| 412 | missing or invalid CSRF token (Nextcloud) | +| 413 | `FILE_TOO_LARGE` | +| 422 | `MISSING_RECORDS_UNSUPPORTED`, `MUNICIPALITY_REQUIRED`, `MUNICIPALITY_INVALID`, `NO_SOURCE_SHEET`, `MISSING_COLUMN`, `TOO_MANY_ROWS` | +| 500 | `IMPORT_FAILED` (unexpected; generic message, details only in the log) | +| 503 | `MAPPING_UNAVAILABLE`, `READER_UNAVAILABLE`, `NOT_CONFIGURED` | + +Error body: `{"success": false, "error": "", "message": "", "details": {...}}`. `details` is always an object, empty when the code has none. For `MISSING_COLUMN`, `details` is `{"sheet": "...", "column": "..."}`. For `NO_SOURCE_SHEET`, it is `{"expected": ["Onbeh Applicaties CMDB", "Beheerde Applicaties CMDB"]}`. For `TOO_MANY_ROWS`, it is `{"sheet": "...", "limit": 10000}`. For `FILE_TOO_LARGE`, it is `{"maxBytes": 10485760}`. For `MISSING_RECORDS_UNSUPPORTED`, it is `{"accepted": ["keep"]}`. + +### `POST /api/cmdb-import/{operationId}/cancel` +**Auth**: Nextcloud admin session plus CSRF token. + +**Request:** no body. + +**Response (200):** +```json +{ "success": true, "cancelRequested": true } +``` + +**Errors:** +| Code | Condition | +|------|-----------| +| 403 | not a Nextcloud admin | +| 404 | `OPERATION_NOT_FOUND`: no `cmdb_import` operation with this id | +| 412 | missing or invalid CSRF token | + +### `GET /api/progress/{operationId}` (existing, unchanged) +Returns the `ProgressTracker` snapshot for the `cmdb_import` operation: `progress.total_items` is the number of non-empty rows read and `progress.processed_items` the rows done so far, updated after every row. `progress.status` is `running`, `completed` or `cancelled`. After completion, `progress.statistics.report` holds the report from the 200 response above, for as long as the tracker keeps the entry (one hour). + +## Error Codes + +| Code | Meaning | Condition | +|------|---------|-----------| +| `NO_FILE_UPLOADED` | no file | `cmdbFile` missing | +| `NOT_XLSX` | not an xlsx workbook | extension is not `.xlsx`, no ZIP signature, or no `xl/workbook.xml` | +| `FILE_TOO_LARGE` | too large | larger than the profile's `maxFileBytes` (10 MB) | +| `MISSING_RECORDS_UNSUPPORTED` | option not supported | `missingRecords` is not `keep` | +| `MUNICIPALITY_REQUIRED` | no consumer | neither `municipalityUuid` nor `municipalityName` given | +| `MUNICIPALITY_INVALID` | wrong consumer | uuid unknown, or the organisation is not of type Municipality | +| `NO_SOURCE_SHEET` | nothing to read | neither "Onbeh Applicaties CMDB" nor "Beheerde Applicaties CMDB" present | +| `MISSING_COLUMN` | required column absent | a present source sheet lacks "APPID" or "Applicatie Naam" | +| `TOO_MANY_ROWS` | file too large to process | a source sheet has more non-empty rows than `maxRowsPerSheet` (10,000) | +| `MAPPING_UNAVAILABLE` | mapping cannot run | OpenRegister's `MappingEngine`/`PackDefinitionValidator` missing, or a shipped pack is invalid | +| `READER_UNAVAILABLE` | xlsx reader missing | PhpSpreadsheet's Xlsx reader cannot be loaded | +| `NOT_CONFIGURED` | stackiq not configured (503) | OpenRegister's object service, the stackiq register, or the `module`, `organization`, `usage` or `contactPerson` schema cannot be resolved; checked before the file is read | +| `OPERATION_NOT_FOUND` | unknown operation | cancel for an id without a `cmdb_import` operation | +| `IMPORT_FAILED` | unexpected error | anything not listed above | + +## Versioning + +Internal app API, unversioned like the other stackiq settings endpoints. The report fields above are additive-only: new fields MAY be added, and existing fields keep their meaning. (Before the first release the row field `middelId` was renamed to `appId`, together with the switch of the match key to the APPID.) The `module` properties `externalId`, `externalNumber`, `externalKey`, `externalCreatedAt` and `externalModifiedAt` are part of the register schema and follow the register's versioning (`module` 0.3.5). + +## Breaking Change Policy + +A breaking change to the endpoints only affects stackiq's own settings section and ships in the same release. A change to the meaning of `externalKey` (the matching rule) is breaking for repeat imports. It requires a new OpenSpec change with a migration that rewrites the stored keys. + +## SLA + +Synchronous request. An unchanged 1,100-row export SHALL finish within PHP's default execution limits on the local rig. Progress is readable while the request runs. No availability promise beyond the Nextcloud instance itself. diff --git a/openspec/changes/cmdb-export-import/design.md b/openspec/changes/cmdb-export-import/design.md new file mode 100644 index 000000000..5d6200808 --- /dev/null +++ b/openspec/changes/cmdb-export-import/design.md @@ -0,0 +1,432 @@ +# Design: cmdb-export-import + +## Context + +A municipality delivers its application landscape as a TOPdesk export (xlsx). The anonymised test export has ten sheets. The municipality's application manager exports AIA and APP from TOPdesk into the raw "Invoer AIA data" and "Invoer APP data" sheets; the CMDB sheets next to them are the overviews the municipality itself uses as "the CMDB", built from the raw sheets with formulas. Decided with the municipality on 2026-10-01: the import reads the two CMDB sheets, "Onbeh Applicaties CMDB" (35 columns, from AIA: applications **without** arranged maintenance) and "Beheerde Applicaties CMDB" (42 columns, from APP: **with** arranged maintenance). The "Invoer" sheets are not read; "Gearchiveerde Applicaties" is a follow-up (missing records). Both CMDB sheets carry formatted but empty rows below the data, and "Beheerde" also formula rows that reference empty "Invoer" rows. Every cell is a formula; the reader uses the cached values. Dates are Excel serial numbers. The file also contains document metadata, a SharePoint sensitivity label, an embedded Power Query package and an external data connection (`xl/connections.xml`). + +The chain baseline on the local rig (OpenRegister 2.1.34-unstable, OpenCatalogi 2.1.17-unstable, Portaliq 0.2.8-unstable, stackiq 0.2.4-unstable) fixed what the import has to produce: + +- OpenRegister's `POST /api/registers/{id}/import` cannot take this file. It maps sheet names to schema slugs and stops at the first unknown sheet. Migration packs work on CSV and JSON only, and one pack targets one schema. +- OpenCatalogi lists a stackiq `module` only through a catalogue that includes the stackiq register and `module` schema, and only when the module's `publicationDate` is set and not in the future. +- Portaliq shows an application to a municipality only through a `usage` whose `consumer` is the organisation in the account's `stackiq.organisationId` claim. + +Stackiq already has two upload imports: `SbomController` with `SbomImportService`, and `SettingsController::importArchiMate` with `ArchiMateImportService`. Both write through `ObjectServiceInterface` and report progress through `ProgressTracker`. This change follows the same pattern. + +## Goals / Non-Goals + +**Goals** + +- One admin action turns a TOPdesk export into modules, vendors, usages and owners for one municipality. +- Owner data is kept in stackiq and Nextcloud Contacts but is never publicly readable. +- Re-importing a newer export updates the same records and creates no duplicates. +- The mapping is data (JSON), not code, and runs through OpenRegister's mapping engine. +- An untrusted spreadsheet is read safely, and one bad row never breaks the import. + +**Non-Goals** + +- Connections, suites, hosting parties ("Hostingpartij"), "Leverancier", the archive sheet, the functional administrator as technical owner. +- Marking or removing records that disappeared from the export. +- A background job, a dry run, an `occ` command, a live TOPdesk connection. +- Writing OpenCatalogi catalogues or Portaliq accounts. + +## Architecture Overview + +``` +Admin settings, "CMDB import" section (CmdbImport.vue) + │ multipart: cmdbFile, municipalityUuid | municipalityName, + │ updateExisting, missingRecords, operationId (+ requesttoken) + ▼ +CmdbImportController::import() admin-only, CSRF, size/type checks + ▼ +CmdbExportImportService::import() + ├─ CmdbImportProfile lib/Settings/cmdb-import/topdesk-profile.json + 5 packs + │ (packs checked with OR PackDefinitionValidator) + ├─ CmdbWorkbookReader PhpSpreadsheet Xlsx, read-data-only, profile sheets only, + │ header-name columns, allowlisted columns, empty rows dropped + ├─ CmdbRowNormaliser trim, placeholder → empty, Excel serial → Y-m-d, numeric ids → string + ├─ OR MappingEngine::mapRow() once per pack per row + ├─ resolve per row, in order: + │ municipality (once) → manufacturer → module → owners → usage + │ via ObjectServiceInterface::searchObjects()/saveObject() + │ and StackiqContactSyncService (OCP\Contacts\IManager) + ├─ ProgressTracker operation `cmdb_import`, per-row progress, cancel + └─ report summary + one entry per counted row + ▼ +OpenRegister, register `stackiq`: module, organization, usage, contactPerson + ├─▶ OpenCatalogi search (module with publicationDate, via its catalogue) + └─▶ Portaliq (usage.consumer = the account's organisation) +``` + +## Decisions + +### D1. A stackiq service, not OpenRegister's import endpoint + +The import is a stackiq service plus controller (route A in the WOO-586 plan). + +- **Alternative: OpenRegister `/api/registers/{id}/import` with a migration pack.** Rejected. It does not accept a pack on xlsx, maps one sheet to one schema, and cannot link the objects it creates (usage.module, usage.consumer, module.provider). +- **Alternative: stackiq splits the file into one CSV per schema and runs four OpenRegister imports.** Rejected. Stackiq still has to split and link the rows, so the four pack runs add moving parts without taking work away. + +### D2. The mapping is a set of migration packs executed by OpenRegister's MappingEngine + +Each target has one pack in OpenRegister's migration-pack format (`id`, `name`, `sourceFormat: excel`, `version`, `fieldMappings`, `idStrategy: {type: generate}`, optional `defaults`). The service validates each pack with `OCA\OpenRegister\Service\MigrationPack\PackDefinitionValidator` when an import starts, and maps rows with `MappingEngine::mapRow($pack, $row, $rowNumber)`. The engine supplies `trim`, `date`, `lookup`, `concat` and `const`, the "required" rule, the guard that an unmapped lookup value never passes through, and errors that name the row, the column and the transform. + +Files, all in `lib/Settings/cmdb-import/`: + +| File | Target | Notes | +|---|---|---| +| `topdesk-profile.json` | none | The two sheets, each with its `constants` (the `Beheer` value added to every row) and `absentColumns` (pack columns the sheet is known not to have), key column, required columns, date and id columns, `emptyValues` (placeholders that mean empty), the pack per target, create-only fields, size and row limits | +| `topdesk-module.json` | `module` | Mapping errors on `required` mappings skip the row; lookups for hosting model and BBN level | +| `topdesk-manufacturer.json` | `organization` (Supplier) | Empty "Vendor" means no provider | +| `topdesk-municipality.json` | `organization` (Municipality) | Maps the options row `{municipalityName}`, not a sheet row | +| `topdesk-usage.json` | `usage` | Lookups for status and TIME class; the maintenance note | +| `topdesk-business-owner.json` | owner identity | `name`, `role`; the service turns it into a contact and a `contactPerson` | + +There is no technical-owner pack: the functional administrator (FB contactpersoon) is not imported (decided 2026-10-01). + +Stackiq-specific settings live in the profile, not in the packs, so every pack stays a valid OpenRegister pack. + +`MappingEngine` and `PackDefinitionValidator` are not part of OpenRegister's `Contract` namespace. The service resolves them from the container inside a guard. If either is missing, the import answers 503 `MAPPING_UNAVAILABLE` before it reads the file. + +- **Alternative: OpenRegister's Twig-based `MappingService::executeMapping()` with `Mapping` entities shipped in the register's `components.mappings`.** Those mappings would be editable in OpenRegister's UI. Rejected for now: lookups and "required" would have to be written as Twig templates, and errors would come without row and column. Kept as an option if admins need to edit the mapping in a UI. +- **Follow-up (not built here):** before using the shipped file, look up a pack with the same `id` in OpenRegister's migration-pack store (`MigrationPackService::findByPackSlug()`). An admin could then override the mapping through `POST /api/migration-packs/import`, without a release. + +### D3. Reading the workbook + +`CmdbWorkbookReader` checks the upload, then reads it: + +1. Before PhpSpreadsheet: the name ends in `.xlsx`, the first bytes are the ZIP signature `PK\x03\x04`, and `ZipArchive` lists `xl/workbook.xml`. Otherwise 400 `NOT_XLSX`. +2. `new \PhpOffice\PhpSpreadsheet\Reader\Xlsx()`, then `setReadDataOnly(true)` and `setLoadSheetsOnly([...profile sheet names that exist])`. The sheet names come from `listWorksheetNames()`. The class comes from OpenRegister's vendor directory, which is loaded whenever OpenRegister is enabled. It is checked with `class_exists`; if absent, 503 `READER_UNAVAILABLE`. +3. Row 1 holds the headers. Each header is normalised (trim, collapse whitespace, drop a trailing `:` or `⚡`, lower case) and matched to the column names the profile and the packs reference. Only those columns are kept. Every other cell, such as Personeelsnummer, phone numbers and group mailboxes, is never copied out of the reader. +4. For each cell the reader takes `getValue()`. For a formula cell (data type `f`) it takes `getOldCalculatedValue()`, the value Excel cached. It never calls `getCalculatedValue()` or `toArray()` with formula calculation. Every cell of the CMDB sheets is a formula, so this is the normal path. A formula without a cached value (no `` in the file, for example a workbook written by a tool that does not calculate) is read as empty and its column is listed in the row's `uncached`; the service turns that into the row warning `Column "…": formula without a cached value, read as empty`. It never fails the row. A cached number `0` is what Excel stores for a reference to an empty cell, and is read as empty. +5. A row whose kept cells are all empty is dropped and not counted. With rule 4 this also drops the formula rows that reference empty "Invoer" rows. +6. Columns are resolved per sheet. A required column missing on a present sheet stops the import; an optional one gives one import-level warning, unless the profile lists it in that sheet's `absentColumns` (`Nickname` exists only on "Beheerde"). A source sheet with more than `maxRowsPerSheet` (10,000) non-empty rows stops the import with 422 `TOO_MANY_ROWS`. + +External connections, the Power Query package and hyperlinks are never resolved: PhpSpreadsheet does not follow them, and the reader gets no HTTP client. + +### D4. Normalising a row before mapping + +`CmdbRowNormaliser` turns reader output into the flat `column => string` row the engine expects: + +- Values listed in the profile's `emptyValues` for their column become empty, compared case-insensitively before any conversion: `NB` in "BNN Classificatie" (the CMDB sheet's "niet bekend") and `49675` (2036-01-01) in "End-of-Life Functioneel" (the CMDB sheet's placeholder for "no end-of-life date"; its formula turns an empty date, or TOPdesk's 2099-12-31, into 49675). +- Columns listed in the profile's `dateColumns` ("Datum", "Referentie datum wijziging", "End-of-Life Functioneel"): a numeric value is converted with `PhpOffice\PhpSpreadsheet\Shared\Date::excelToDateTimeObject()` in UTC and written as `Y-m-d`. For example, `45111.38…` becomes `2023-07-04` and `53359` becomes `2046-02-01`. A non-numeric value stays as it is, so the pack's `date` transform (`sourceFormat: Y-m-d`) either accepts it or reports a warning. +- Columns listed in `idColumns` ("APPID"): a whole number becomes a string without a decimal part (`1234.0` becomes `"1234"`). +- The constants of the row's sheet are added before mapping (`Beheer` = `Beheer geregeld: nee` on "Onbeh", `ja` on "Beheerde"), so the usage pack can map the sheet like a column. +- Every value is trimmed. An empty string counts as empty. + +The engine's `date` transform only parses formatted strings. Doing the serial conversion in the normaliser keeps the packs plain OpenRegister packs. + +### D5. Matching key and upsert + +The key is the TOPdesk APPID (the ICT Applicatienummer), scoped to the municipality: `externalKey = "topdesk:" + municipalityUuid + ":" + APPID`. Decided with the municipality on 2026-10-01: the Middel-ID (CMDB column "Applicatie Code") can be changed in TOPdesk, the APPID cannot; the first real import also showed the Middel-ID prefix in two spellings (`APP-` and `App-`). The Applicatie Code is stored as `externalId` for reference and updated like any mapped field. APPIDs are unique within one TOPdesk instance, not across municipalities; with the scope, two municipalities can each import an APPID `101` without colliding. + +Per row: + +1. A row without an APPID is skipped (`missing APPID`). An APPID already seen in this upload, on either sheet, is skipped (`duplicate APPID in file`). The CMDB sheets have no "Soort" column, so there is no row-kind filter. +2. Look up the module with `searchObjects` on the configured register and module schema, filtered on `externalKey`, with `_rbac: false` and `_multitenancy: false` (as `SbomImportService` does; the caller is an admin). The result is cached for the run. +3. No match: create the module from the mapped data, plus `externalKey`, the create-only defaults (`type: Application`), and `publicationDate` (D6). +4. Match and `updateExisting=false`: skip with reason `exists`. +5. Match: merge the mapped fields onto the stored object. Every field the pack does not map stays as it is. Create-only fields stay as they are, unless the stored value is empty. If the merged object equals the stored one, do not save, and report `unchanged`. Otherwise save, and report `updated`. + +The APPID is also stored as `externalNumber`, so it is visible on the module. + +- **Alternative: the Middel-ID ("Applicatie Code") as the key**, as in the first version of this change. Rejected on 2026-10-01: it can change in the source. +- **Alternative: OpenRegister's `idStrategy: sourceField` (APPID as the object id).** Rejected. Object ids are global uuids, and the APPID is neither a uuid nor unique across municipalities. +- **Alternative: put the key on `usage` (per municipality by nature).** Rejected for this change: the key on `module` was decided in the plan (Q3), and the usage is found from the module anyway (D7). + +### D6. publicationDate + +- New module: `publicationDate` = the import's start time (ISO 8601 with offset). This makes it visible to OpenCatalogi, given a catalogue that covers the stackiq `module` schema. +- Existing module: `publicationDate` and `depublicationDate` are never written, also when they are empty. An admin who depublished an imported module keeps it depublished. + +### D7. Related objects and their order + +Per row, in this order: + +1. **Municipality** (once per import): `municipalityUuid` must resolve to an `organization` of type `Municipality`. Otherwise 422 `MUNICIPALITY_INVALID`. With `municipalityName`, the service reuses an existing Municipality with the same normalised name, or creates one through the municipality pack. +2. **Manufacturer**: map "Vendor" (the maker of the software) through the manufacturer pack. "Leverancier" (where the municipality buys it) and "Hostingpartij" are not read (follow-up). The normalised name (trim, collapse whitespace, lower case) is looked up in the run cache, then among `organization` objects of type `Supplier`. A new one is created only when neither matches. The first real import (1,137 rows, 479 suppliers) showed no two names that differ only in case, spacing or a legal-form suffix (`B.V.`, `BV`, `Inc.` …), so the normalisation is not widened. +3. **Module** (D5), with `provider` = the manufacturer when there is one. +4. **Owners** (D8). +5. **Usage**: `searchObjects` on `consumer` = municipality and `module` = module uuid. Create or merge the usage pack's fields, plus `consumer`, `module`, `provider` = the manufacturer, and `businessOwner`. `interneAnnotation` ("Beheer geregeld: ja|nee / Cluster / Applicatie Eigenaar (Afdeling)", empty parts left out) is create-only, because it is a free-text note an admin may edit. + +When step 3 succeeds and step 5 fails, the row is `failed` with the step named. The next import completes it, because every step is find-or-create. + +### D8. The owner as contact person + +Stackiq keeps a person's identity in Nextcloud Contacts. A `contactPerson` object holds only `contactsUid`, `role`, `organization` and `roles`. The owner is "Applicatie Eigenaar (Persoon)"; when TOPdesk has no owner the CMDB sheet shows the owner's function there instead, and the import uses that as the display name too. "Applicatie Eigenaar (Functie)" is the role; "Applicatie Eigenaar (Afdeling)" goes into the usage note (D7), because a contact person has no department field. The functional administrator (FB contactpersoon) is not imported, so there is no `technicalOwner`. + +1. The CMDB sheets carry no e-mail address, so the service runs `searchContacts(name)` and accepts only an exact, case-insensitive display-name match; otherwise `StackiqContactSyncService::syncToContacts('contactPerson', ['voornaam' => …, 'achternaam' => …, 'role' => …])` creates the contact. This avoids creating a new contact on every import. +2. Find the `contactPerson` with that `contactsUid` and `organization` = the municipality (run cache, then `searchObjects`). If none exists, create it with `role` = "Applicatie Eigenaar (Functie)" when given. +3. Set `usage.businessOwner` to its uuid. + +**Never public.** `contactPerson` and `usage` have read rules for named groups only, none for `public`, and a published `module` refers to them through `contactPerson` / `usages` relations. An anonymous OpenCatalogi search hit therefore carries at most ids, and OpenRegister's objects API returns no contact person or usage to an anonymous caller. `tests/Unit/Settings/CmdbPersonDataVisibilityTest.php` pins the read rules on the merged register; the e2e test checks the running stack anonymously. + +The import never calls the user-provisioning paths (`ContactpersoonService::processContactpersoon`, `convertToUser`). The scheduled `OrganizationSyncService::performUserSync` provisions users for contact persons. A unit test asserts that a `contactPerson` written by the import does not meet its selection criteria, and the implementation task verifies that before shipping (see Risks). When Contacts is disabled, owners are skipped with a warning and the row is still imported. + +### D9. Progress, cancel and the report + +The import runs inside the upload request, as the SBOM and ArchiMate imports do. The client sends a fresh `operationId`. The service calls `startOperation('cmdb_import', ['total_items' => rowCount])`, then `updateProgress` after each row and `completeOperation($report)` at the end. The UI polls the existing `GET /api/progress/{operationId}`. `POST /api/cmdb-import/{operationId}/cancel` calls `setCancelRequested()`. The service checks `isCancelRequested()` between rows and returns the partial report with `cancelled: true`. + +Report shape (contract.md is authoritative): `summary {rowsRead, created, updated, unchanged, skipped, failed, warnings}`, `importWarnings[]` (for example, a missing optional column), and `rows[] {sheet, row, appId, name, outcome, reasons[], warnings[], moduleUuid, usageUuid}`. Reasons name columns and values. They never name owners, e-mail addresses or other person data, and neither do log lines. + +### D10. Controller and validation order + +`CmdbImportController::import()` has neither `#[NoAdminRequired]` nor `#[NoCSRFRequired]`, so Nextcloud's middleware enforces admin and CSRF before the method runs. The method then checks, in this order: + +1. A file is present: otherwise 400 `NO_FILE_UPLOADED`. +2. Size: otherwise 413 `FILE_TOO_LARGE`. +3. xlsx: otherwise 400 `NOT_XLSX`. +4. `missingRecords` is `keep`: otherwise 422 `MISSING_RECORDS_UNSUPPORTED`. +5. A municipality is given: otherwise 422 `MUNICIPALITY_REQUIRED`. +6. The service runs. It answers 503 `MAPPING_UNAVAILABLE` or `READER_UNAVAILABLE`, 422 `NO_SOURCE_SHEET`, `MISSING_COLUMN`, `TOO_MANY_ROWS` or `MUNICIPALITY_INVALID`, or 200 with the report. + +Every expected service exception is translated to its status code in the controller (hydra gate controller-exception-translation). Only unexpected errors become 500, with a generic message and the detail logged. + +### D11. The settings section + +`src/views/settings/sections/CmdbImport.vue` sits next to `ArchiMateImportExport.vue` in `StackiqSettings.vue`, inside `AlwaysVisibleSection`, and is not added to the vue-router (hydra gate admin-router). + +- Municipality: an `NcSelect` with a label. It lists organisations of type Municipality, read through the OpenRegister objects API, and has an option to type a new name. +- File: an `` with a label. +- Options: an "Update existing records" checkbox. +- Import and Cancel buttons. +- During the import: `NcProgressBar` with a polite live region. +- Afterwards: summary counts and a `CnDataTable` report with an outcome filter and links to the modules. + +Cell values are shown with text interpolation only, never `v-html`. Requests use `@nextcloud/axios`, which sends the CSRF token. + +## Column mapping + +Source columns of "Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB" and where they go, by header name. A column that one sheet lacks is optional there. Columns that are not listed are not read (D3). + +| Column | Sheets | Target | Rule | +|---|---|---|---| +| APPID | both | module.externalNumber; part of module.externalKey | trim, required, numeric to string, match key (D5) | +| Applicatie Naam | both | module.name | trim, required | +| Applicatie Code | both | module.externalId | trim; reference only (the Middel-ID; can change in the source) | +| Roepnaam | both | module.shortDescription | trim; wins over Nickname | +| Nickname | Beheerde | module.shortDescription | trim; used when Roepnaam is empty; listed as absent on Onbeh | +| Functionele Omschrijving | both | module.longDescription | trim | +| Applicatiesoort | both | module.cloudDienstverleningsmodel | lookup: `Saas`/`SaaS` → `["SaaS"]`, `PaaS` → `["PaaS"]`, `IaaS` → `["IaaS"]`, `On-premise(s)` → `["On-premises (self-managed)"]`; another value (such as `Webapplicatie`): warning, field dropped | +| BNN Classificatie | both | module.bbnLevel | `NB` is empty; lookup "BBN1"/"BBN 1"/"BNN1" etc. to `BBN1`/`BBN2`/`BBN3`; unknown value: warning | +| Datum | both | module.externalCreatedAt | Excel serial to date | +| Referentie datum wijziging | both | module.externalModifiedAt | Excel serial to date | +| Vendor | both | organization (Supplier) via module.provider and usage.provider | dedup on normalised name (D7) | +| Applicatie Status | both | usage.status | lookup: In productie → In production, In voorraad → Planned, In ontwikkeling → Acquisition, Uit te faseren → To be phased out, Uitgefaseerd → Phased out; unknown value: warning | +| Classificatie | both | usage.timeClassification | lookup Tolereren/Tolerate, Investeren/Invest, Migreren/Migrate, Elimineren/Eliminate | +| End-of-Life Functioneel | both | usage.startDateOutPhased | `49675` (2036-01-01) is empty; Excel serial to date | +| (sheet constant `Beheer`), Cluster, Applicatie Eigenaar (Afdeling) | both | usage.interneAnnotation | concat with " / ", empty parts dropped, create-only; `Beheer geregeld: nee` (Onbeh) or `ja` (Beheerde) | +| Applicatie Eigenaar (Persoon), Applicatie Eigenaar (Functie) | both | usage.businessOwner (contactPerson + Nextcloud contact; role = Functie) | D8; the person column may hold a function | +| Hostingpartij | both | not mapped | follow-up; "Leverancier" (where the municipality buys the software) is not on the CMDB sheets | +| Software Suite | both | not mapped (suite schema; needs a second pass) | follow-up | +| Applicatiecomponent, Bron, Datum Interface, Referentie element externe ID, Cloud, Rappeldatum, Rappelreden, Locatie BIOToets | both | not mapped | Cloud is derived from Applicatiesoort; Datum Interface is the export date | +| Beschikbaarheid, Integriteit, Vertrouwelijkheid, Applicatienut, Kwaliteit en betrouwbaarheid van leverancier, Flexibiliteit, Gebruikerstevredenheid, Reputatie risico | both | not mapped (no field on module or usage) | schema extension is out of scope | +| Standaard, Behandelgroep, End-of-life Technisch, End-of-support Technisch, Top5, COTS, Applicatie Nummer | Beheerde | not mapped | Applicatie Nummer repeats the APPID | +| every column of the "Invoer" sheets | – | never read | | + +## API Design + +The authoritative interface is in `contract.md`. In short: + +- `POST /api/cmdb-import`: multipart `cmdbFile`, plus `municipalityUuid` or `municipalityName`, `updateExisting` (default `true`), `missingRecords` (default `keep`) and `operationId`. Admin, CSRF. Answers 200 with the report, or one of the errors in D10. +- `POST /api/cmdb-import/{operationId}/cancel`: admin, CSRF. Answers 200 `{cancelRequested: true}`. +- `GET /api/progress/{operationId}`: the existing route, unchanged. + +## Database Changes + +No tables or Nextcloud migrations (ADR-001). The `module` schema gains five optional properties through a register fragment (see Mixed-spec rationale and `migration.md`). + +## Mixed-spec rationale (ADR-032) + +The change is `kind: code`. Its weight is the import service, controller, reader and settings section. It also contains a thin schema delta: `lib/Settings/register.d/topdesk-cmdb-import.json` adds `externalId`, `externalNumber`, `externalKey`, `externalCreatedAt` and `externalModifiedAt` to `module`, all optional strings or dates, and seeds three example modules. This is not the ADR-032 `mixed` anti-pattern: + +1. The delta is additive glue that exists only for this code. Nothing else reads the properties, and the import cannot be idempotent without a stored key (no existing `module` property can hold a TOPdesk id). +2. It follows the app's fragment convention (ADR-037), so it touches no other change's file. +3. It is deployed by the existing register import in the repair step, without a migration class. + +The fragment bumps `module` to `0.3.5`. Fragments are merged in filename order and a scalar `version` is overwritten by the last fragment that sets it. `maintenance-and-roadmap.json` sets `module` to `0.3.4`, so a fragment that sorts before it would have its bump overwritten, and the new properties would never deploy. The file is therefore named `topdesk-cmdb-import.json`, which sorts after it, and a unit test asserts that the merged register declares `module` version `0.3.5` with the five properties. + +## Declarative-vs-imperative decision (ADR-031) + +- **Imperative, because it is an external integration:** reading an uploaded third-party file, splitting a row into four linked objects, resolving contacts in Nextcloud Contacts, progress and cancel. These are not object lifecycle, aggregation, notification or relation rules that an `x-openregister-*` block can express. This is the external-integration exception: the service is imperative glue around the file. +- **Declarative:** what each column becomes (target property, transform, lookup, required) is JSON in OpenRegister's migration-pack format, executed by OpenRegister's `MappingEngine`. Changing the mapping changes no PHP. +- **Matching rule (stated once, enforced in code):** a module matches when its `externalKey` equals `topdesk::`. A usage matches on (`consumer`, `module`). A supplier matches on its normalised name and type `Supplier`. A contact person matches on (`contactsUid`, `organization`). +- **publicationDate rule (stated once, enforced in code):** set to the import's start time on create; never written on update. +- No `x-openregister-*` block is added or changed. The usage name keeps coming from the schema's existing name template. + +## Nextcloud Integration + +- Controllers: `CmdbImportController` (`import`, `cancel`), admin-only with CSRF, no `NoAdminRequired` / `NoCSRFRequired`. +- Services: `CmdbExportImportService` (orchestration), `Cmdb\CmdbWorkbookReader`, `Cmdb\CmdbRowNormaliser`, `Cmdb\CmdbImportProfile` (loads and validates the profile and packs). They reuse `ProgressTracker`, `SettingsService` (register and schema ids) and `StackiqContactSyncService`. +- OCP: `IRequest::getUploadedFile()`, `IUserSession`, `IL10N`, `OCP\Contacts\IManager` (through `StackiqContactSyncService`), `ICacheFactory` (through `ProgressTracker`). +- OpenRegister: `ObjectServiceInterface::searchObjects()` / `saveObject()` (contract), `MigrationPack\MappingEngine` and `PackDefinitionValidator` (container, guarded), PhpSpreadsheet (guarded). +- Mappers/Entities: none (ADR-001, ADR-008: Controller → Service → OpenRegister). +- Events/Hooks: none. Saves go through `saveObject()`, so the existing `ModuleRegistrationSubscriber` and `ModuleComplianceSubscriber` run as they do for any module save. + +## Security Considerations + +- **Auth and CSRF:** both routes are admin-only through Nextcloud's middleware, with CSRF required. This is stricter than `SbomController` and `importArchiMate`, which carry `NoCSRFRequired`. The admin check happens before the body is read. +- **File checks before parsing:** size limit (10 MB, profile), `.xlsx` extension, ZIP signature and `xl/workbook.xml`. `.xlsm` and `.xls` are rejected. The upload is read from PHP's temporary upload file and never written into Nextcloud Files. +- **No evaluation, no fetching:** read-data-only, profile sheets only, cached values for formula cells, no `getCalculatedValue()`, no HTTP client in the reader. External connections, Power Query packages and hyperlinks are inert. +- **Resource bounds:** row cap per sheet, and only allowlisted columns are kept. Memory is bounded by loading only the two CMDB sheets. +- **Injection:** every value is a string that goes through OpenRegister's schema validation on save, and is never used in SQL, file paths or templates. The UI renders values as text only. +- **Isolation:** every row runs in its own try/catch. Errors are reported per row, and the import continues. +- **Privacy:** the column allowlist keeps every person column except the owner out of memory; the "Invoer" sheets, which hold personnel numbers, phones and group mailboxes, are not read at all. Owner identity goes only to Nextcloud Contacts, and `contactPerson` and `usage` are never publicly readable (D8). Reports and logs carry no person data. +- **Fixture hygiene:** the test fixture is the anonymised export with document metadata, the custom properties (sensitivity label), `customXml/` (including the Power Query package) and `xl/connections.xml` removed. One small synthetic connection part is added back for the external-connection test. + +## NL Design System + +Nextcloud and `@conduction/nextcloud-vue` components only (ADR-012): `NcSelect`, `NcButton`, `NcCheckboxRadioSwitch`, `NcProgressBar`, `NcNoteCard` for errors, and `CnDataTable` for the report. The file input follows the label pattern of `ArchiMateImportExport.vue`. Colours and spacing come from Nextcloud CSS variables (ADR-003). Outcome badges reuse the existing status-tag styling. + +## File Structure + +``` +appinfo/ + routes.php (+ cmdbImport#import, cmdbImport#cancel) +lib/ + Controller/ + CmdbImportController.php + Service/ + CmdbExportImportService.php + Cmdb/ + CmdbImportProfile.php + CmdbWorkbookReader.php + CmdbRowNormaliser.php + CmdbImportReport.php + Exception/ + CmdbImportException.php (carries error code + HTTP status) + Settings/ + cmdb-import/ + topdesk-profile.json + topdesk-module.json + topdesk-manufacturer.json + topdesk-municipality.json + topdesk-usage.json + topdesk-business-owner.json + register.d/ + topdesk-cmdb-import.json (module 0.3.5: five properties + seed modules) +src/views/settings/ + StackiqSettings.vue (registers the section) + sections/CmdbImport.vue +tests/ + fixtures/cmdb/ + topdesk-export-anonymised.xlsx (sanitised copy of the test export) + topdesk-missing-appid.xlsx + topdesk-shuffled-columns.xlsx + topdesk-formula-and-connection.xlsx + Unit/Service/CmdbExportImportServiceTest.php + Unit/Service/Cmdb/CmdbWorkbookReaderTest.php + Unit/Service/Cmdb/CmdbRowNormaliserTest.php + Unit/Service/Cmdb/CmdbImportProfileTest.php + Unit/Controller/CmdbImportControllerTest.php + Unit/Settings/TopdeskCmdbFragmentTest.php + Unit/Settings/CmdbPersonDataVisibilityTest.php + e2e/spec-coverage/cmdb-import.spec.ts +docs/features/ + cmdb-import.md +l10n/ + en.json, en.js, nl.json, nl.js +``` + +## Seed Data + +Placeholders: uuids are nil-style (`00000000-0000-0000-0000-00000000000N`). All names are fictional ("Gemeente Voorbeeldstad", "Voorbeeld Software B.V."). No real people, addresses or numbers appear. + +### Schema: `module` (modified; seeded through the fragment's `components.objects`) + +The seeds show the new properties in a fresh install. They carry no `publicationDate`, so they are not published as open data, and no `externalKey`, because the key holds a municipality uuid that only exists at run time. + +| Field | Object 1 | Object 2 | Object 3 | +|---|---|---|---| +| @self | register `stackiq`, schema `module`, slug `voorbeeld-zaaksysteem` | slug `voorbeeld-afsprakenplanner` | slug `voorbeeld-belastingapplicatie` | +| name | Voorbeeld Zaaksysteem | Voorbeeld Afsprakenplanner | Voorbeeld Belastingapplicatie | +| type | Application | Application | Application | +| longDescription | Registreert en volgt zaken van intake tot archivering. | Laat inwoners online een afspraak maken bij de balie. | Berekent en verstuurt gemeentelijke belastingaanslagen. | +| externalId | APP-00001 | APP-00002 | AIA-00003 | +| externalNumber | 101 | 102 | 103 | +| externalCreatedAt | 2023-07-04 | 2022-03-16 | 2024-01-15 | +| externalModifiedAt | 2026-07-29 | 2026-09-01 | 2026-05-20 | +| bbnLevel | BBN2 | BBN1 | BBN2 | + +**Related items per object:** none seeded. Files, notes, tasks and contacts are not used by these modules. Provider and usages come from a real import, not from seeds. + +### Objects an import writes (not seeded; reference shapes for tests and docs) + +`organization` (municipality, created from `municipalityName`): + +| Field | Value | +|---|---| +| uuid | 00000000-0000-0000-0000-000000000001 | +| name | Gemeente Voorbeeldstad | +| type | Municipality | +| status | Active | + +`organization` (manufacturer): + +| Field | Object A | Object B | +|---|---|---| +| uuid | 00000000-0000-0000-0000-000000000002 | 00000000-0000-0000-0000-000000000003 | +| name | Voorbeeld Software B.V. | Fabfrikant | +| type | Supplier | Supplier | +| status | Active | Active | + +`module` (as written by the import): + +| Field | Value | +|---|---| +| uuid | 00000000-0000-0000-0000-000000000004 | +| name | naamtest123 | +| externalId | APP-test123 | +| externalNumber | 2 | +| externalKey | topdesk:00000000-0000-0000-0000-000000000001:2 | +| shortDescription | Naamtest | +| longDescription | Accomodatieplanning. | +| cloudDienstverleningsmodel | ["SaaS"] | +| bbnLevel | BBN2 | +| provider | 00000000-0000-0000-0000-000000000003 | +| publicationDate | 2026-10-01T10:00:00+00:00 | + +`usage`: + +| Field | Value | +|---|---| +| uuid | 00000000-0000-0000-0000-000000000005 | +| consumer | 00000000-0000-0000-0000-000000000001 | +| module | 00000000-0000-0000-0000-000000000004 | +| provider | 00000000-0000-0000-0000-000000000003 | +| status | In production | +| timeClassification | Tolerate | +| startDateOutPhased | 2046-02-01 | +| interneAnnotation | Beheer geregeld: ja / B10 / B10 Maatschappelijke Ontwikkeling | +| businessOwner | 00000000-0000-0000-0000-000000000006 | + +`contactPerson`: + +| Field | Value | +|---|---| +| uuid | 00000000-0000-0000-0000-000000000006 | +| contactsUid | `` | +| organization | 00000000-0000-0000-0000-000000000001 | +| role | Teamleider Applicatiebeheer | + +## Risks / Trade-offs + +- [The user sync might provision accounts for imported contact persons] → The import writes contact persons without e-mail or user fields on the OpenRegister object. A unit test runs `performUserSync`'s selection against an imported `contactPerson`. If the selection would pick it up, the implementation adds an explicit marker that excludes it before shipping, and does not ship otherwise. +- [Owner contacts land in the importing admin's address book] → `StackiqContactSyncService` writes to the first writable address book of the acting user, the same as every other stackiq contact path. The docs say so. A dedicated system address book is a follow-up. +- [Long synchronous request] → Per-row progress, cancel, and "unchanged" rows skip the save. About 1,100 rows is expected to fit. A background job is a follow-up if it does not. +- [OpenRegister internals (`MappingEngine`, `PackDefinitionValidator`, PhpSpreadsheet) change shape] → Guarded resolution with 503, and a contract test that maps the fixture through the real engine in the dev environment. +- [Provisional lookups for Applicatiesoort and BNN Classificatie] → The values of the first real import were not kept (the report lives in the progress cache for an hour). The maps hold the values the anonymised export and the CMDB formulas show (`Saas`, `Webapplicatie`, `NB`) plus the usual spellings. An unknown value is a warning, never a wrong value; once the municipality lists its values, the maps in the JSON are extended, with no code change. +- [CMDB placeholders] → "End-of-Life Functioneel" `2036-01-01` and "BNN Classificatie" `NB` are read as empty. A real end-of-life date of exactly 2036-01-01 would be lost; the municipality confirms. "Classificatie" defaults to `Tolereren` on "Beheerde" when TOPdesk has none; that cannot be told apart from a real `Tolereren` and is imported as Tolerate. +- [An application moves between the sheets] → Same APPID, so the same module and usage; the usage note (create-only) keeps its old `Beheer geregeld` line when it is not empty. +- [An unknown status on create falls back to the usage schema's default "In production"] → Accepted. The warning in the report makes it visible. +- [The fragment version is overwritten by merge order] → Filename ordering plus a unit test on the merged version (Mixed-spec rationale). + +## Migration Plan + +No data migration. The register fragment deploys with the existing repair-step register import (see `migration.md`). Rollback is a revert of the PR. The optional `module` properties may stay deployed without harm. + +## Open Questions + +- Which "Applicatiesoort" and "BNN Classificatie" values occur in the municipality's real export, and which hosting model does each mean (lookup maps)? +- Is 2036-01-01 in "End-of-Life Functioneel" always the placeholder, and should a "Beheerde" row without a TIME class really be Tolerate? +- Should the maintenance status update the usage note on re-import (it is create-only today), or get a field of its own? +- Should OpenRegister promote `MigrationPack\MappingEngine` to its `Contract` namespace? diff --git a/openspec/changes/cmdb-export-import/migration.md b/openspec/changes/cmdb-export-import/migration.md new file mode 100644 index 000000000..b0b8cd43d --- /dev/null +++ b/openspec/changes/cmdb-export-import/migration.md @@ -0,0 +1,52 @@ +# Migration: cmdb-export-import + +## Current State + +The `module` schema in register `stackiq` is at version `0.3.4` after merging `softwarecatalogus_register.json` (0.3.3) with `register.d/maintenance-and-roadmap.json` (0.3.4, `roadmapStatement`). It has no property that holds an identifier from an external source system. No Nextcloud database table is involved (ADR-001). OpenRegister stores modules in its magic table for the `stackiq` register and `module` schema. + +## Target State + +The `module` schema is at version `0.3.5` with five extra optional properties, all `visible`. None is `required`, so every existing module stays valid unchanged. + +| Property | Type | Notes | +|---|---|---| +| `externalId` | string, maxLength 100 | TOPdesk Applicatie Code (the Middel-ID), shown as "Source id"; reference only | +| `externalNumber` | string, maxLength 50 | TOPdesk APPID ("ICT Applicatienummer") | +| `externalKey` | string, maxLength 200, `table.default: false` | `topdesk::`, the import's match key | +| `externalCreatedAt` | string, format date | creation date in the source system | +| `externalModifiedAt` | string, format date | last change in the source system | + +Three seed modules (design.md, Seed Data) are added through the fragment's `components.objects`. + +## Migration Class + +No Nextcloud migration class. The schema change deploys through stackiq's existing register import in the repair step (`SettingsService` loads `softwarecatalogus_register.json`, deep-merges `register.d/*.json` in filename order, and imports the result through OpenRegister's `ConfigurationService`). OpenRegister adds the new columns to the magic table when the schema version increases. + +``` +Version: n/a (register version bump, no lib/Migration class) +File: lib/Settings/register.d/topdesk-cmdb-import.json +Key operations: +- components.schemas.module.version = "0.3.5" +- components.schemas.module.properties += externalId, externalNumber, externalKey, externalCreatedAt, externalModifiedAt +- components.objects += 3 seed modules (no publicationDate, no externalKey) +``` + +## Migration Steps + +1. Add `lib/Settings/register.d/topdesk-cmdb-import.json`. The filename must sort after `maintenance-and-roadmap.json`, so its `module.version` wins the scalar overwrite in the merge. +2. Run the repair step (app upgrade or `occ maintenance:repair`). OpenRegister sees `module` 0.3.5 > deployed 0.3.4, and updates the schema and its magic table. +3. Seed modules are created when absent (matched on slug), as with the other seeds. + +## Data Impact + +Existing modules get five new empty columns. There is no data loss and no transformation. Safe on live data: the change is additive, and the columns are nullable. + +## Rollback Procedure + +Remove the fragment and revert the PR. OpenRegister does not drop columns on a lower version, so the five columns stay, empty, and nothing reads them. To remove imported data, delete the modules with a non-empty `externalKey` and their usages. To remove the seed modules, delete the slugs `voorbeeld-zaaksysteem`, `voorbeeld-afsprakenplanner` and `voorbeeld-belastingapplicatie`. + +## Validation + +- Unit test `tests/Unit/Settings/TopdeskCmdbFragmentTest.php` merges the register exactly as `SettingsService` does, and asserts `module.version === "0.3.5"` with the five properties present and none required. +- On the rig after the repair step: `GET /index.php/apps/openregister/api/schemas/` shows version `0.3.5` and the five properties. +- `GET /index.php/apps/openregister/api/objects/stackiq/module?externalId=APP-00001` returns the seed module `voorbeeld-zaaksysteem`. diff --git a/openspec/changes/cmdb-export-import/proposal.md b/openspec/changes/cmdb-export-import/proposal.md new file mode 100644 index 000000000..5c9b4f93f --- /dev/null +++ b/openspec/changes/cmdb-export-import/proposal.md @@ -0,0 +1,105 @@ +--- +kind: code +depends_on: [] +--- + +# Proposal: cmdb-export-import + +## Summary + +A Nextcloud admin uploads a TOPdesk CMDB export (xlsx) in stackiq's admin settings, picks the municipality the export belongs to, and stackiq turns every application row into OpenRegister objects in the `stackiq` register: a `module` (the application), an `organization` for its vendor, a `usage` that links the application to the municipality and records whether maintenance is arranged, and a `contactPerson` for the application owner, which is never publicly readable. The rows come from the two CMDB sheets, "Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB". Rows are matched on TOPdesk's APPID (ICT Applicatienummer), so a second import of a newer export updates the same records instead of duplicating them. The column-to-field mapping is declarative JSON executed by OpenRegister's migration-pack mapping engine. The admin follows the import live and gets a per-row report: created, updated, unchanged, skipped or failed, with the reason. + +## Motivation + +A municipality wants to search all its applications in one place in OpenCatalogi and see the ones it uses in Portaliq. Its CMDB is the source of that list, but a live API connection is not possible yet (calls must come from the municipality's own IP range), so the municipality delivers a periodic TOPdesk export instead (Jira WOO-586, epic WOO-281). + +The existing paths cannot read this file. A chain baseline on a local rig (2026-10-01) showed: + +- `POST /api/registers/{id}/import` in OpenRegister treats every xlsx sheet as a schema named after the sheet, and stops at the first sheet: `Schema not found (id='Invoer AIA data')`. +- OpenRegister migration packs apply to CSV and JSON imports only, and one pack maps one sheet to one schema. One TOPdesk row has to become a module, a manufacturer organisation, a usage and up to two contact persons, linked to each other. +- OpenCatalogi only lists a stackiq module that has a `publicationDate`. Portaliq only shows an application to a municipality through a `usage` whose `consumer` is that municipality. An import that writes modules alone leaves both apps empty. + +Stackiq already has two upload-and-import flows (SBOM, ArchiMate). This change adds a third one for CMDB exports, built the same way. + +## Capabilities + +### New Capabilities + +- `cmdb-export-import`: an admin uploads a TOPdesk CMDB export (xlsx) and stackiq creates or updates modules, vendor organisations, usages and owner contact persons for one municipality, matched on the TOPdesk APPID, with live progress and a per-row report. + +### Modified Capabilities + +None. The `module` schema gains five optional properties through a register fragment (see design.md, Mixed-spec rationale). No existing requirement changes. + +## Affected Projects + +- [ ] Project: `stackiq`: import service, controller and routes, a "CMDB import" section in admin settings, declarative mapping and import-profile JSON under `lib/Settings/cmdb-import/`, a register fragment that adds external-id properties to `module`, tests, administrator docs and translations. + +## Scope + +### In Scope + +- Upload endpoint for `.xlsx` files only, with a size limit, admin-only and CSRF-protected. +- Reading the two CMDB sheets the municipality uses as its CMDB (decided with the municipality on 2026-10-01): "Onbeh Applicaties CMDB" (from the AIA export: applications without arranged maintenance) and "Beheerde Applicaties CMDB" (from the APP export: with arranged maintenance). The raw "Invoer" sheets are not read. Columns are found by header name per sheet, not position. The CMDB sheets are formulas: the value Excel cached is read; formulas are never evaluated, and a formula without a cached value is an empty cell with a row warning. +- One municipality per import, chosen by the admin from existing stackiq organisations of type Municipality, or created from a name the admin types. +- Per row: upsert the `module` on APPID, find or create the vendor `organization` from the "Vendor" column (one organisation per distinct vendor), upsert the `usage` (consumer = municipality, module = the application, maintenance arranged yes/no in its note), and find or create the `contactPerson` for "Applicatie Eigenaar (Persoon)" (business owner) through Nextcloud Contacts. Contact persons and usages stay out of every public read. +- Declarative mapping: one migration-pack JSON per target (module, manufacturer, municipality, usage, business owner) plus one import profile (sheets, sheet constants, required columns, key column, date columns, placeholder values), executed through OpenRegister's `MappingEngine::mapRow()`. +- Excel serial dates converted to ISO dates before mapping. +- `publicationDate` set to the import time on newly created modules, never changed on update. +- Repeatable import: matched records are updated, records missing from a newer export are left alone (`missingRecords: keep`, the only accepted value for now). +- Per-row error isolation, live progress and cancel through the existing `ProgressTracker`, and a per-row report. +- PHPUnit tests using the anonymised test export as a fixture, a Playwright e2e for the admin flow, an administrator docs page, and Dutch and English strings. + +### Out of Scope + +- The "Invoer" sheets, and the archive sheet "Gearchiveerde Applicaties" (follow-up: mark an APPID that left the CMDB sheets as archived). +- Connections between applications from the "Ouders" / "Kind-middelen" columns (a second pass after all modules exist, as a follow-up change). +- Suites from "Software Suite", hosting parties from "Hostingpartij", "Leverancier" as a second supplier source, and the functional administrator (FB contactpersoon) as technical owner. The mapping can take them later without code once a target is agreed. +- Marking or removing records that disappeared from a newer export (`missingRecords: mark|remove`, reserved values; belongs with operations-record-reconciliation, stackiq#1127). +- A live TOPdesk or ServiceNow connection (stackiq#373, stackiq#1134), a dry-run mode, an `occ` command, and running the import as a background job. +- Configuring OpenCatalogi catalogues or Portaliq account claims. The docs describe both prerequisites; the import does not write to those apps. + +## Approach + +A `CmdbImportController` accepts the upload and options, validates the file before parsing, and hands it to `CmdbExportImportService`. The service reads the two sheets with PhpSpreadsheet's Xlsx reader in read-data-only mode, resolves columns by header name from the import profile, normalises each row (trim, Excel serial to ISO date, numeric ids to strings), and maps it with OpenRegister's migration-pack `MappingEngine` once per target pack. It then resolves the related objects in a fixed order (manufacturer, module, contact persons, usage) and saves each through OpenRegister's `ObjectServiceInterface`. Each row runs in its own try/catch and its outcome goes into the report. Progress and cancel use the existing `ProgressTracker` and `/api/progress/{operationId}` route, as the ArchiMate import does. A new admin-settings section uploads the file, polls progress and shows the report. Details are in design.md. + +## New Dependencies + +None for stackiq's `composer.json` or `package.json`. The xlsx reader (`phpoffice/phpspreadsheet`) and the mapping engine come from OpenRegister, which stackiq already requires. Design.md describes the guard for when either class is not available. + +## Impact + +- **New backend**: `lib/Service/CmdbExportImportService.php` (plus small helpers for sheet reading and row normalisation), `lib/Controller/CmdbImportController.php`, two routes in `appinfo/routes.php`. +- **New configuration**: `lib/Settings/cmdb-import/topdesk-profile.json` and five pack files `lib/Settings/cmdb-import/topdesk-*.json`. +- **Schema**: `lib/Settings/register.d/topdesk-cmdb-import.json` adds `externalId`, `externalNumber`, `externalKey`, `externalCreatedAt` and `externalModifiedAt` to `module` (all optional), with a version bump so the register import deploys them. +- **New frontend**: `src/views/settings/sections/CmdbImport.vue`, registered in `src/views/settings/StackiqSettings.vue`. +- **Data**: imports write `module`, `organization`, `usage` and `contactPerson` objects, and Nextcloud Contacts cards for owners. No existing object is deleted. + +## Cross-Project Dependencies + +- **openregister** (consumed, not changed): `ObjectServiceInterface` (public contract), `MigrationPack\MappingEngine` and `PackDefinitionValidator` (not yet a public contract), and the PhpSpreadsheet library it ships. +- **opencatalogi** and **portaliq** (consumers, not changed): they show the imported data once their own configuration is in place, namely a catalogue that includes the stackiq register's `module` schema, and a portal account with claim `stackiq.organisationId` for the municipality. Portaliq's contribution is stackiq's existing `usage` contribution (hydra ADR-046); this change adds no new portal contribution. + +## Risks + +### Risk 1: Personal data from a third party's export +**Severity:** High — **Mitigation:** only the owner columns the import maps ("Applicatie Eigenaar (Persoon)", "Applicatie Eigenaar (Functie)") are read into stackiq, and they go to Nextcloud Contacts as the existing contact model requires. The "Invoer" sheets, with personnel numbers, phone numbers and group mailboxes, are never read. `contactPerson` and `usage` have no public read rule, so owners never reach OpenCatalogi or Portaliq anonymously; a unit test pins the rule and an e2e test checks it anonymously. The report and the logs name rows by sheet, row number and APPID only. Tests use the anonymised export. The import never creates Nextcloud user accounts, and a test asserts that the contact-person objects it writes do not qualify for the user sync. + +### Risk 2: Hidden coupling to OpenRegister internals +**Severity:** Medium — **Mitigation:** `MappingEngine`, `PackDefinitionValidator` and PhpSpreadsheet are resolved through the container or a `class_exists` check. If any of them is missing, the import endpoint answers 503 with a clear message instead of failing halfway. Promoting `MappingEngine` to an OpenRegister contract is noted as an open question. + +### Risk 3: Long imports over HTTP +**Severity:** Medium — **Mitigation:** an export with about 1,100 rows needs several saves per row. The service reports progress per row, honours cancel between rows, and skips the save when nothing changed. A background-job variant is a follow-up if real exports time out. + +### Risk 4: TOPdesk values that do not match stackiq vocabularies +**Severity:** Low — **Mitigation:** "Applicatie Status", "Classificatie", "Applicatiesoort" and "BNN Classificatie" go through `lookup` maps. An unknown value drops only that field, and the row gets a warning that names the column and the value. Admins can extend the maps in the JSON. + +## Rollback Strategy + +The change is additive. Revert the PR to remove the routes, the settings section, the service and the mapping files. The register fragment only adds optional properties; after a revert they stay in the deployed schema without harm, and imported objects stay as ordinary stackiq objects. To remove imported data, filter modules on a non-empty `externalKey` and delete them together with their usages. + +## Open Questions + +- Answered on 2026-10-01: the CMDB sheets are the source; AIA = without arranged maintenance, APP = with; archived applications are a follow-up; the key is the APPID. +- Which "Applicatiesoort" and "BNN Classificatie" values occur in the real export, and which hosting model does each "Applicatiesoort" mean? The lookup maps are provisional. +- Should OpenRegister expose `MappingEngine` as a public contract (as it does for `ObjectServiceInterface`)? diff --git a/openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md b/openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md new file mode 100644 index 000000000..92ccf2d7c --- /dev/null +++ b/openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md @@ -0,0 +1,409 @@ +# cmdb-export-import Specification + +**Status**: in-progress +**Scope**: stackiq +**OpenSpec changes**: +- [cmdb-export-import](../../changes/cmdb-export-import/) + +## Purpose + +A Nextcloud admin imports a TOPdesk CMDB export (xlsx) into stackiq for one municipality. Every application row of the export's two CMDB sheets ("Onbeh Applicaties CMDB", applications without arranged maintenance, and "Beheerde Applicaties CMDB", with arranged maintenance) becomes, or updates, a `module` (schema:SoftwareApplication) with its vendor `organization` (schema:Organization), a `usage` that links the application to the municipality, and a `contactPerson` (schema:Person) for its owner, which is never publicly readable. All data is stored as OpenRegister objects (ADR-001). The column-to-field mapping is declarative JSON executed by OpenRegister's mapping engine (ADR-011, ADR-031), so the import can be repeated with a newer export without creating duplicates. OpenCatalogi lists the imported applications, and Portaliq shows them to the municipality. + +Nextcloud OCP interfaces used: `OCP\IRequest` (multipart upload), `OCP\IUserSession` and `OCP\IGroupManager` (admin check), `OCP\Contacts\IManager` (owner identity, through `StackiqContactSyncService`), `OCP\ICacheFactory` (progress, through `ProgressTracker`), `OCP\IL10N` (messages). OpenRegister: `OCA\OpenRegister\Contract\ObjectServiceInterface` for every read and write. + +## ADDED Requirements + +### Requirement: REQ-CMDB-001 The import endpoint SHALL accept only a bounded xlsx upload from a Nextcloud admin + +`POST /api/cmdb-import` SHALL be reachable only by Nextcloud admins and SHALL require Nextcloud's CSRF token. The endpoint SHALL NOT carry `#[NoAdminRequired]` or `#[NoCSRFRequired]`. It SHALL reject the upload before any parsing when the file is larger than the configured maximum (default 10 MB), when its name does not end in `.xlsx`, or when its content is not a ZIP package containing `xl/workbook.xml`. Macro-enabled (`.xlsm`), legacy (`.xls`) and CSV files SHALL be rejected. No object SHALL be written in any of these cases. + +#### Scenario: A file that is not xlsx is rejected +@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts + +- **GIVEN** a Nextcloud admin on the CMDB import section +- **WHEN** they upload `applications.csv`, or a file named `export.xlsx` whose content is plain text +- **THEN** the endpoint SHALL answer 400 with error `NOT_XLSX` +- **AND** no `module`, `organization`, `usage` or `contactPerson` object SHALL be created or changed + +#### Scenario: An oversized file is rejected before it is read +@e2e exclude Building a file over 10 MB in the browser adds nothing over the unit test; tests/Unit/Controller/CmdbImportControllerTest.php asserts 413 FILE_TOO_LARGE and that the reader is never called. + +- **GIVEN** an xlsx upload of 10 MB plus one byte +- **WHEN** a Nextcloud admin posts it to `POST /api/cmdb-import` +- **THEN** the endpoint SHALL answer 413 with error `FILE_TOO_LARGE` +- **AND** the workbook reader SHALL NOT be invoked + +#### Scenario: A user who is not a Nextcloud admin cannot import +@e2e exclude Authorisation rule; tests/Unit/Controller/CmdbImportControllerTest.php asserts the method has no NoAdminRequired attribute, and the Newman collection asserts 403 for a non-admin user. + +- **GIVEN** a signed-in user who is not a Nextcloud admin, including a member of `software-catalog-admins` +- **WHEN** they post an export to `POST /api/cmdb-import` +- **THEN** Nextcloud SHALL answer 403 +- **AND** no object SHALL be written + +#### Scenario: A request without a CSRF token is refused +@e2e exclude CSRF is enforced by Nextcloud's middleware; the Newman collection posts without a requesttoken and asserts 412. + +- **GIVEN** a Nextcloud admin session +- **WHEN** a request to `POST /api/cmdb-import` arrives without a valid `requesttoken` header or parameter +- **THEN** Nextcloud SHALL refuse it with 412 +- **AND** no object SHALL be written + +### Requirement: REQ-CMDB-002 The workbook SHALL be read as stored data, without evaluating formulas or following links + +The reader SHALL open the workbook with PhpSpreadsheet's Xlsx reader in read-data-only mode, SHALL load only the sheets named in the import profile, and SHALL read each cell's stored value. For a formula cell it SHALL use the value cached in the file and SHALL NOT evaluate the formula. It SHALL NOT contact external data connections, linked workbooks or URLs found in the file. A formula cell without a cached value SHALL be read as empty and SHALL add a row warning naming the column; it SHALL NOT fail the row or the import. A formula whose cached value is the number 0 (Excel's result for a reference to an empty cell) SHALL be read as empty. It SHALL stop with 422 `TOO_MANY_ROWS` when a source sheet holds more data rows than the profile's limit (default 10,000). + +#### Scenario: A formula cell yields its cached value and is not evaluated +@e2e exclude Reader behaviour; tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php reads a fixture whose source sheet has a formula cell and asserts the cached value is returned and the calculation engine is never invoked. + +- **GIVEN** a CMDB sheet where column "Applicatie Naam" in row 2 holds a formula with a cached value `Rekenmodel` +- **WHEN** the workbook is read +- **THEN** the row SHALL carry `Applicatie Naam = Rekenmodel` +- **AND** the formula SHALL NOT be evaluated + +#### Scenario: A formula without a cached value is read as empty with a warning +@e2e exclude Reader and service behaviour; tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php reads a fixture whose "Roepnaam" formula has no cached value, and tests/Unit/Service/CmdbExportImportServiceTest.php asserts the row warning. + +- **GIVEN** a CMDB sheet where column "Roepnaam" in row 2 holds a formula without a cached value +- **WHEN** the workbook is imported +- **THEN** the row SHALL be imported with an empty "Roepnaam" +- **AND** the row's report entry SHALL carry the warning `Column "Roepnaam": formula without a cached value, read as empty` + +#### Scenario: An external data connection in the workbook is never contacted +@e2e exclude Network isolation; tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php reads a fixture that declares an external connection, with a reader that has no HTTP client, and asserts the read succeeds. + +- **GIVEN** an export that contains `xl/connections.xml` with an external data connection +- **WHEN** the workbook is read +- **THEN** no network request SHALL be made +- **AND** the source sheets SHALL be read normally + +### Requirement: REQ-CMDB-003 Columns SHALL be resolved by header name, and a missing required column SHALL stop the import with 422 + +The source sheets SHALL be the CMDB sheets "Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB"; the "Invoer" sheets SHALL NOT be read. The reader SHALL take the first row of each source sheet as headers and SHALL match them to the profile's column names per sheet, case-insensitively, after trimming whitespace and dropping a trailing `:` or `⚡`. Column order SHALL NOT matter. When a present source sheet lacks a column the profile marks as required (`APPID`, `Applicatie Naam`), the endpoint SHALL answer 422 with error `MISSING_COLUMN`, naming the column and the sheet, before any object is written. When neither source sheet exists, the endpoint SHALL answer 422 with error `NO_SOURCE_SHEET`, naming both expected sheets. A missing optional column SHALL produce one import-level warning and no row error, except for a column the profile lists as absent on that sheet (`Nickname` on "Onbeh Applicaties CMDB"). + +#### Scenario: A missing required column is named in the 422 response +@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts + +- **GIVEN** an export whose sheet "Beheerde Applicaties CMDB" has no column "APPID" +- **WHEN** a Nextcloud admin uploads it +- **THEN** the endpoint SHALL answer 422 with error `MISSING_COLUMN`, column `APPID` and sheet `Beheerde Applicaties CMDB` +- **AND** the section SHALL show that column and sheet name to the admin +- **AND** no object SHALL be written + +#### Scenario: Columns in a different order map the same +@e2e exclude Reader behaviour; tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php reads a fixture with shuffled columns and asserts identical rows. + +- **GIVEN** an export where "Applicatie Naam" comes before "APPID" and the header reads `Vendor⚡` +- **WHEN** the workbook is read +- **THEN** every row SHALL carry the same values under the profile's column names as in the original order + +#### Scenario: A workbook without either source sheet is refused +@e2e exclude Same error path as the missing column; tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php asserts NO_SOURCE_SHEET naming both sheets. + +- **GIVEN** an xlsx that contains only a sheet "Blad1" +- **WHEN** a Nextcloud admin uploads it +- **THEN** the endpoint SHALL answer 422 with error `NO_SOURCE_SHEET` naming "Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB" + +### Requirement: REQ-CMDB-004 Every import SHALL have exactly one consuming municipality, chosen by the admin + +The request SHALL carry either `municipalityUuid`, the uuid of an existing stackiq `organization` of type `Municipality`, or `municipalityName`, a name for a new one. With a name, the service SHALL reuse an existing organisation of type `Municipality` with the same normalised name, or create one through the municipality pack (type `Municipality`, status `Active`). It SHALL answer 422 `MUNICIPALITY_REQUIRED` when neither is given, and 422 `MUNICIPALITY_INVALID` when the uuid does not resolve to an organisation of type `Municipality`. Every `usage` and `contactPerson` the import writes SHALL reference that organisation. + +#### Scenario: The admin picks an existing municipality +@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts + +- **GIVEN** an organisation "Gemeente Voorbeeldstad" of type `Municipality` +- **WHEN** a Nextcloud admin selects it and imports the anonymised export +- **THEN** both imported usages SHALL have `consumer` = the uuid of "Gemeente Voorbeeldstad" +- **AND** no new organisation of type `Municipality` SHALL be created + +#### Scenario: A new municipality is created once from a typed name +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php imports twice with municipalityName "Gemeente Voorbeeldstad" and asserts one organisation of type Municipality. + +- **GIVEN** no organisation named "Gemeente Voorbeeldstad" +- **WHEN** a Nextcloud admin imports with `municipalityName` "Gemeente Voorbeeldstad", and later imports again with the same name +- **THEN** exactly one organisation "Gemeente Voorbeeldstad" of type `Municipality` and status `Active` SHALL exist + +#### Scenario: An import without a municipality is refused +@e2e exclude Validation; tests/Unit/Controller/CmdbImportControllerTest.php asserts 422 MUNICIPALITY_REQUIRED, and 422 MUNICIPALITY_INVALID for the uuid of a Supplier organisation. + +- **GIVEN** a valid export +- **WHEN** a Nextcloud admin posts it with neither `municipalityUuid` nor `municipalityName` +- **THEN** the endpoint SHALL answer 422 with error `MUNICIPALITY_REQUIRED` +- **AND** no object SHALL be written + +### Requirement: REQ-CMDB-005 Field mapping SHALL be declarative and executed by OpenRegister's mapping engine + +The service SHALL map each normalised row with OpenRegister's `MigrationPack\MappingEngine::mapRow()`, once per target pack: module, manufacturer, municipality, usage, business owner. The packs and the import profile SHALL ship as JSON under `lib/Settings/cmdb-import/`. Each pack SHALL pass OpenRegister's `PackDefinitionValidator` when the import starts; an invalid pack, or a missing `MappingEngine`, SHALL stop the import with 503 `MAPPING_UNAVAILABLE` before any row is read. Before mapping, the service SHALL convert the cells of the profile's date columns from Excel serial numbers to `Y-m-d`, SHALL turn numeric id cells into strings without a decimal part, SHALL read a value the profile lists as empty for its column (`NB` in "BNN Classificatie", serial `49675` in "End-of-Life Functioneel") as empty, and SHALL add the constants of the row's sheet (`Beheer` = `Beheer geregeld: nee` or `ja`). A mapping error on a mapping marked `required` in the module pack SHALL skip the row. In the manufacturer and owner packs it SHALL mean the row has no manufacturer or no such owner, without a warning. A mapping error on any other mapping SHALL drop only that field and add a row warning naming the column and the value. The reader SHALL keep only the columns that the profile or a pack references, and SHALL discard every other cell when it reads the row. + +#### Scenario: Excel serial dates are converted before mapping +@e2e exclude Pure transformation; tests/Unit/Service/Cmdb/CmdbRowNormaliserTest.php asserts the conversions below. + +- **GIVEN** the "Onbeh" row of the anonymised export with "Datum" = `45111.380322627316` and "Referentie datum wijziging" = `46232.552113113423`, and the "Beheerde" row with "End-of-Life Functioneel" = `53359` +- **WHEN** the rows are normalised +- **THEN** "Datum" SHALL be `2023-07-04`, "Referentie datum wijziging" SHALL be `2026-07-29`, and "End-of-Life Functioneel" SHALL be `2046-02-01` +- **AND** "APPID" `1234` SHALL be the string `"1234"` +- **AND** "End-of-Life Functioneel" `49675` and "BNN Classificatie" `NB` SHALL be empty + +#### Scenario: Changing a pack changes the mapping without code +@e2e exclude Configuration behaviour; tests/Unit/Service/CmdbExportImportServiceTest.php loads an alternate module pack that maps "Software Suite" to licentietype and asserts the mapped module. + +- **GIVEN** the module pack is edited to add a mapping from "Software Suite" to `licentietype` +- **WHEN** an export is imported whose row has "Software Suite" = `Suite` +- **THEN** the created module SHALL have `licentietype` = `Suite` +- **AND** no PHP code SHALL have changed + +#### Scenario: An unknown status value drops only that field +@e2e exclude Mapping behaviour; tests/Unit/Service/CmdbExportImportServiceTest.php asserts the row outcome and warning. + +- **GIVEN** a row whose "Applicatie Status" is `Onbekende status`, which the usage pack's lookup does not contain +- **WHEN** the row is imported +- **THEN** the module and the usage SHALL be saved without a status from the export +- **AND** the row's report entry SHALL carry a warning naming column "Applicatie Status" and value `Onbekende status` + +#### Scenario: The classifications map to the stackiq fields +@e2e exclude Mapping behaviour; tests/Unit/Service/CmdbExportImportServiceTest.php imports the fixture and asserts the fields. + +- **GIVEN** the "Beheerde" row of the anonymised export with "Applicatiesoort" `Saas`, "BNN Classificatie" `BBN2`, "Classificatie" `Tolereren` and "End-of-Life Functioneel" `53359` +- **WHEN** it is imported +- **THEN** the module SHALL have `cloudDienstverleningsmodel` = `["SaaS"]` and `bbnLevel` = `BBN2` +- **AND** the usage SHALL have `timeClassification` = `Tolerate` and `startDateOutPhased` = `2046-02-01` +- **AND** the "Onbeh" row's "Applicatiesoort" `Webapplicatie`, which is not a hosting model, SHALL be dropped with a warning + +### Requirement: REQ-CMDB-006 A module SHALL be matched on its TOPdesk APPID, so a re-import updates instead of duplicating + +For each row the service SHALL compute `externalKey` = `topdesk::` (the APPID is TOPdesk's ICT Applicatienummer; the Applicatie Code, or Middel-ID, can change in TOPdesk and is stored as `externalId` for reference only) and look up a `module` with that `externalKey`. When none exists it SHALL create one. When one exists it SHALL update only the fields the module pack maps and SHALL leave every other field as it is. When the mapped fields equal the stored values it SHALL NOT save the module and SHALL report the row as `unchanged`. With `updateExisting=false` a matched row SHALL be reported as `skipped` with reason `exists`, without changes. A row without an APPID SHALL be skipped with reason `missing APPID`. When an APPID occurs more than once in one upload, across both sheets, the first occurrence SHALL be imported and every later one SHALL be skipped with reason `duplicate APPID in file`. + +#### Scenario: Re-importing the same export creates no duplicates +@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts + +- **GIVEN** the anonymised export was imported once for "Gemeente Voorbeeldstad", which created the modules with APPID `1234` and `2` +- **WHEN** the same export is imported again for the same municipality +- **THEN** the report SHALL show 0 created and 2 unchanged rows +- **AND** the number of modules, organisations, usages and contact persons in the register SHALL be the same as after the first import + +#### Scenario: A changed field is updated on re-import +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php imports a row, changes "Applicatie Naam" of APPID 2 to `naamtest124` in the row data, imports again, and asserts one module with the new name. + +- **GIVEN** the module with APPID `2` was imported with name `naamtest123`, and an admin has since set its `website` +- **WHEN** a newer export where "Applicatie Naam" for APPID `2` is `naamtest124` is imported +- **THEN** the same module SHALL now have name `naamtest124` +- **AND** its `website` SHALL be unchanged +- **AND** the report SHALL show the row as `updated` + +#### Scenario: An APPID that occurs twice in one file is imported once +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php feeds two rows with the same APPID, also across both sheets. + +- **GIVEN** an upload where APPID `2` appears in row 2 and row 7 of "Beheerde Applicaties CMDB" +- **WHEN** it is imported +- **THEN** row 2 SHALL be imported +- **AND** row 7 SHALL be reported as `skipped` with reason `duplicate APPID in file` + +#### Scenario: A changed Applicatie Code keeps the same module +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php imports APPID 42 with two different codes. + +- **GIVEN** the module with APPID `42` was imported with "Applicatie Code" `APP-Oud` +- **WHEN** a newer export has APPID `42` with "Applicatie Code" `App-Nieuw` +- **THEN** the same module SHALL be updated, with `externalId` = `App-Nieuw` and the same `externalKey` + +### Requirement: REQ-CMDB-007 A newly created module SHALL get a publicationDate, and an existing one SHALL keep its own + +When the service creates a `module` it SHALL set `publicationDate` to the time the import started, as an ISO 8601 date-time, so OpenCatalogi lists the module. When it updates an existing `module` it SHALL NOT change `publicationDate` or `depublicationDate`, also when they are empty. + +#### Scenario: OpenCatalogi can list an imported application +@e2e exclude Crosses into OpenCatalogi, whose catalogue configuration is outside this change; tests/Unit/Service/CmdbExportImportServiceTest.php asserts publicationDate on created modules, and the manual test plan checks the search in OpenCatalogi. + +- **GIVEN** an OpenCatalogi catalogue that includes the stackiq register's `module` schema +- **WHEN** the anonymised export is imported +- **THEN** each created module SHALL have a `publicationDate` that is not later than the moment the import finished +- **AND** a search in OpenCatalogi for `Aangetekend Mailen` SHALL find the module + +#### Scenario: Re-import preserves publicationDate +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php asserts both cases. + +- **GIVEN** the module with APPID `1234` was imported with `publicationDate` 2026-10-01T09:00:00+00:00, and the module with APPID `2` was later depublished by an admin +- **WHEN** a newer export is imported that changes both modules' names +- **THEN** the module with APPID `1234` SHALL keep `publicationDate` 2026-10-01T09:00:00+00:00 +- **AND** the module with APPID `2` SHALL keep its `depublicationDate` and SHALL NOT get a new `publicationDate` + +### Requirement: REQ-CMDB-008 A manufacturer SHALL become one supplier organisation, however many rows name it + +The service SHALL map "Vendor" (the maker of the software) through the manufacturer pack to an `organization` of type `Supplier`. It SHALL match names after trimming, collapsing whitespace and ignoring case, first against the organisations it has already resolved during this import, then against existing organisations of type `Supplier`, and SHALL create one only when neither matches. The imported module's `provider` and the usage's `provider` SHALL reference that organisation. A row with an empty "Vendor" SHALL be imported without a provider. "Leverancier" and "Hostingpartij" SHALL NOT be read. + +#### Scenario: Rows with the same manufacturer share one organisation +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php feeds three rows with "Fabfrikant", "Fabfrikant " and "FABFRIKANT". + +- **GIVEN** three rows whose "Vendor" is `Fabfrikant`, `Fabfrikant ` and `FABFRIKANT` +- **WHEN** they are imported +- **THEN** exactly one organisation `Fabfrikant` of type `Supplier` SHALL exist +- **AND** all three modules SHALL have `provider` = its uuid + +#### Scenario: An existing supplier is reused +@e2e exclude Covered by the service test. + +- **GIVEN** an existing organisation `Aangetekend B.V.` of type `Supplier` +- **WHEN** the "Onbeh" row with "Vendor" `Aangetekend B.V.` is imported +- **THEN** no new organisation SHALL be created +- **AND** the module with APPID `1234` SHALL have `provider` = the existing organisation's uuid + +### Requirement: REQ-CMDB-009 Each imported application SHALL have one usage that links it to the municipality + +For each imported module the service SHALL keep exactly one `usage` with `consumer` = the municipality and `module` = the module, found by those two references and created when missing. The usage pack SHALL map "Applicatie Status" to `status` and "Classificatie" to `timeClassification` through lookups, "End-of-Life Functioneel" to `startDateOutPhased`, and the sheet's `Beheer` constant, "Cluster" and "Applicatie Eigenaar (Afdeling)" to `interneAnnotation`, so the note records whether maintenance is arranged (`Beheer geregeld: nee` for "Onbeh Applicaties CMDB", `ja` for "Beheerde Applicaties CMDB"). Empty parts SHALL be left out of the note. `interneAnnotation` SHALL be written only when the usage is created or the field is empty, so a note an admin wrote is never overwritten. + +#### Scenario: The usage records whether maintenance is arranged +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php imports a row from each sheet. + +- **GIVEN** a row on "Onbeh Applicaties CMDB" with "Cluster" `H10` and "Applicatie Eigenaar (Afdeling)" `H10 Accounting` +- **WHEN** it is imported +- **THEN** its usage SHALL have `interneAnnotation` = `Beheer geregeld: nee / H10 / H10 Accounting` +- **AND** a row on "Beheerde Applicaties CMDB" without a cluster SHALL get `Beheer geregeld: ja / ` + +#### Scenario: Portaliq can show the application to the municipality +@e2e exclude Crosses into Portaliq, whose account claim is outside this change; tests/Unit/Service/CmdbExportImportServiceTest.php asserts the usage references, and the manual test plan checks Portaliq's "Software we use". + +- **GIVEN** a Portaliq account with claim `stackiq.organisationId` = the uuid of "Gemeente Voorbeeldstad" +- **WHEN** the anonymised export is imported for "Gemeente Voorbeeldstad" +- **THEN** a usage SHALL exist for each imported module with `consumer` = that uuid and `module` = the module's uuid +- **AND** that account SHALL see `Aangetekend Mailen` and `naamtest123` under "Software we use" + +#### Scenario: A re-import does not add a second usage +@e2e exclude Covered by the re-import scenario of REQ-CMDB-006 and the service test. + +- **GIVEN** the module with APPID `2` already has a usage for "Gemeente Voorbeeldstad" +- **WHEN** a newer export is imported for the same municipality +- **THEN** the module with APPID `2` SHALL still have exactly one usage for "Gemeente Voorbeeldstad" + +### Requirement: REQ-CMDB-010 The owner SHALL become a contact person of the municipality through Nextcloud Contacts, never a user account, and SHALL never be publicly readable + +The business owner pack SHALL map "Applicatie Eigenaar (Persoon)" (the display name, which may be a function instead of a person's name) and "Applicatie Eigenaar (Functie)" (the role). No technical owner SHALL be imported; the functional administrator columns SHALL NOT be read. For the owner the service SHALL resolve a Nextcloud contact through `StackiqContactSyncService` by an exact match on the display name, and otherwise by creating one. It SHALL then reuse or create one `contactPerson` with that `contactsUid`, `organization` = the municipality and `role` = "Applicatie Eigenaar (Functie)" when given, and SHALL set `usage.businessOwner` to it. The import SHALL NOT create Nextcloud user accounts. When Nextcloud Contacts is unavailable, the row SHALL be imported without an owner and SHALL carry a warning. `contactPerson` and `usage` SHALL have no public read rule, so the owner is never readable by an anonymous visitor; a published module SHALL refer to them by relation only. + +#### Scenario: The owner becomes the business owner +@e2e exclude Needs a Contacts address book; tests/Unit/Service/CmdbExportImportServiceTest.php asserts the calls to a StackiqContactSyncService test double and the saved contactPerson. + +- **GIVEN** the "Onbeh" row with "Applicatie Eigenaar (Persoon)" `Achternaam, Voornaam` and "Applicatie Eigenaar (Functie)" `Afdelingshoofd`, and the "Beheerde" row whose person column holds the function `Teamleider Applicatiebeheer` +- **WHEN** they are imported for "Gemeente Voorbeeldstad" +- **THEN** one `contactPerson` SHALL exist per owner with the resolved `contactsUid`, `organization` = "Gemeente Voorbeeldstad" and the function as `role` +- **AND** each usage SHALL have `businessOwner` = its owner's contact person and no `technicalOwner` +- **AND** no Nextcloud user account SHALL be created + +#### Scenario: Imported owners are never readable anonymously +@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts + +- **GIVEN** the anonymised export was imported, creating contact persons for its owners +- **WHEN** a visitor who is not signed in lists the `contactPerson` and `usage` objects through OpenRegister, or searches OpenCatalogi for an imported application +- **THEN** OpenRegister SHALL return no contact person and no usage +- **AND** the OpenCatalogi search hit SHALL carry no owner name, and its `contactPerson` and `usages` SHALL be empty or ids only + +#### Scenario: The same owner on two rows is one contact person +@e2e exclude Covered by the service test. + +- **GIVEN** two rows with the same "Applicatie Eigenaar (Persoon)" +- **WHEN** they are imported +- **THEN** exactly one `contactPerson` for that contact SHALL exist for the municipality, referenced by both usages + +#### Scenario: Contacts disabled does not block the import +@e2e exclude Environment condition; tests/Unit/Service/CmdbExportImportServiceTest.php sets isAvailable() to false. + +- **GIVEN** the Nextcloud Contacts app is disabled +- **WHEN** the anonymised export is imported +- **THEN** both modules and usages SHALL be saved without owners +- **AND** each row with an owner SHALL carry the warning that owners were skipped because Contacts is unavailable + +### Requirement: REQ-CMDB-011 Each row SHALL be processed in isolation and reported with its outcome + +The service SHALL process every non-empty row in its own error boundary. An exception in one row SHALL mark that row `failed` with the reason and SHALL NOT stop the import or change the outcome of other rows. Rows whose cells are all empty SHALL be ignored and not counted. The response SHALL contain a summary (rows read, created, updated, unchanged, skipped, failed, warnings) and one entry per counted row with sheet, row number, APPID, application name, outcome, reasons, warnings and the uuids of the module and usage. Report entries and log lines SHALL NOT contain owner names, e-mail addresses or other person data. The section SHALL render report values as text, never as HTML. + +#### Scenario: Upload with a per-row report +@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts + +- **GIVEN** a Nextcloud admin, "Gemeente Voorbeeldstad" selected, and the anonymised export +- **WHEN** they start the import and it finishes +- **THEN** the section SHALL show 2 rows read and 2 created +- **AND** the report SHALL list `Onbeh Applicaties CMDB` row 2 APPID `1234` and `Beheerde Applicaties CMDB` row 2 APPID `2`, each with outcome `created` and a link to its module +- **AND** the hundreds of formatted but empty rows in both sheets SHALL NOT appear in the report + +#### Scenario: One bad row does not stop the others +@e2e exclude Fault injection; tests/Unit/Service/CmdbExportImportServiceTest.php makes saveObject() throw for one row of three. + +- **GIVEN** an export with three rows, where saving the module of the second row fails in OpenRegister +- **WHEN** it is imported +- **THEN** rows 1 and 3 SHALL be `created` +- **AND** row 2 SHALL be `failed` with a reason naming the step that failed +- **AND** the response SHALL be 200 with that summary + +#### Scenario: A row without a name is skipped with its reason +@e2e exclude Covered by the service test. + +- **GIVEN** a row on "Beheerde Applicaties CMDB" with an APPID but an empty "Applicatie Naam" +- **WHEN** it is imported +- **THEN** it SHALL be `skipped` with reason `missing Applicatie Naam` + +### Requirement: REQ-CMDB-012 Records missing from a newer export SHALL be left untouched + +The import SHALL accept `missingRecords` with the value `keep`, which is also the default. It SHALL NOT change, depublish or delete a module, usage, organisation or contact person because its APPID is absent from the upload. Any other value, including the reserved `mark` and `remove`, SHALL be refused with 422 `MISSING_RECORDS_UNSUPPORTED`. + +#### Scenario: An application dropped from the export stays +@e2e exclude Covered by the service test; tests/Unit/Service/CmdbExportImportServiceTest.php imports two rows, then one, and asserts the other module and usage are unchanged. + +- **GIVEN** the modules with APPID `1` and `7` were imported for "Gemeente Voorbeeldstad" +- **WHEN** a newer export that only contains APPID `1` is imported +- **THEN** the module with APPID `7` and its usage SHALL be unchanged + +#### Scenario: A reserved value is refused +@e2e exclude Validation; tests/Unit/Controller/CmdbImportControllerTest.php. + +- **GIVEN** a valid export +- **WHEN** a Nextcloud admin posts it with `missingRecords=remove` +- **THEN** the endpoint SHALL answer 422 with error `MISSING_RECORDS_UNSUPPORTED` +- **AND** no object SHALL be written + +### Requirement: REQ-CMDB-013 A running import SHALL report its progress and SHALL stop when cancelled + +The import SHALL run as a `ProgressTracker` operation of type `cmdb_import` under the `operationId` the client sends, and SHALL update the processed row count after every row, readable through the existing `GET /api/progress/{operationId}`. `POST /api/cmdb-import/{operationId}/cancel`, admin-only and CSRF-protected, SHALL request cancellation. The service SHALL check for cancellation between rows, SHALL keep the rows already processed, and SHALL return the report with `cancelled: true`. The final report SHALL also be stored with the operation, so it can be read again within the tracker's lifetime. + +#### Scenario: The admin follows and cancels a running import +@e2e exclude Timing-dependent with a two-row fixture; tests/Unit/Service/CmdbExportImportServiceTest.php requests cancellation after row 1 of three and asserts one processed row and cancelled true. + +- **GIVEN** an import of three rows that is running +- **WHEN** the admin presses Cancel after the first row is done +- **THEN** the service SHALL stop before the second row +- **AND** the report SHALL show 1 processed row and `cancelled: true` +- **AND** the module created for the first row SHALL stay + +### Requirement: REQ-CMDB-014 The admin settings SHALL offer a CMDB import section + +Stackiq's admin settings page SHALL show a section "CMDB import", rendered by the settings page and not registered as an in-app route. The section SHALL let the admin choose an existing municipality or type the name of a new one, choose an `.xlsx` file, and start the import. While the import runs it SHALL show a progress bar and a Cancel button. Afterwards it SHALL show the summary and a report table that can be filtered by outcome. Every control SHALL have a visible label, and every string SHALL be translatable. + +#### Scenario: The admin runs an import from the settings page +@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts + +- **GIVEN** a Nextcloud admin on stackiq's admin settings page +- **WHEN** they choose "Gemeente Voorbeeldstad", choose the anonymised export and press "Import" +- **THEN** a progress bar SHALL appear while the import runs +- **AND** afterwards the summary and the report table SHALL be shown +- **AND** filtering the table on `created` SHALL show the two imported rows + +## Non-Functional Requirements + +- **Performance:** an export of 1,100 rows SHALL import on the local rig without exceeding PHP's default memory limit, by loading only the source sheets in read-data-only mode. A re-import of an unchanged export SHALL make no `saveObject()` call for unchanged modules and usages. Lookups of organisations, modules and contact persons SHALL be cached per import run, so each distinct vendor, APPID and contact is looked up at most once. +- **Security:** an uploaded third-party file is input: xlsx only, bounded size and row count, no formula evaluation (cached values only), no external links, header-name resolution, per-row isolation, admin-only routes with CSRF (REQ-CMDB-001 to 003, 011). No cell value is ever rendered as HTML. +- **Privacy:** only the owner columns named in REQ-CMDB-010 are read into stackiq, and the objects holding them are never publicly readable. The report and the logs contain no person data. Test fixtures are anonymised and carry no document metadata naming real people. +- **Accessibility:** Target WCAG 2.2 AA. The section uses Nextcloud and `@conduction/nextcloud-vue` components: labelled file input and municipality select (SC 1.3.1, 3.3.2; gates `form-label-association`, `nc-input-labels`), a labelled Cancel button (SC 4.1.2; gate `button-name`), a progress bar and summary announced through a polite live region (SC 4.1.3; `axe`), and a report table with header cells (SC 1.3.1; gate `table-headers`). New in 2.2: 2.4.11 Focus Not Obscured applies (the report must not hide focus behind sticky headers); 2.5.7 Dragging Movements does not apply (the file input works without drag and drop); 2.5.8 Target Size applies to the buttons (Nextcloud defaults); 3.2.6 Consistent Help does not apply (no help mechanism added); 3.3.7 Redundant Entry applies (the chosen municipality stays selected after an import); 3.3.8 Accessible Authentication does not apply (no authentication step). +- **Internationalization:** Dutch and English MUST be supported (ADR-005) for the section, the error messages and the report reasons. + +## Acceptance Criteria + +- [ ] A Nextcloud admin imports the anonymised TOPdesk export for a chosen municipality, and the report lists both data rows as created. +- [ ] Importing the same export again creates no object, and reports both rows as unchanged. +- [ ] A changed "Applicatie Naam" in a newer export updates the same module (matched on APPID); `publicationDate` and fields the export does not map stay as they were. +- [ ] Rows with the same "Vendor" share one supplier organisation. +- [ ] Every imported module has one usage whose consumer is the municipality. +- [ ] A missing "APPID" or "Applicatie Naam" column stops the import with 422 naming the column and sheet; a non-xlsx or oversized file is rejected before reading. +- [ ] One failing row is reported as failed while the other rows are imported. +- [ ] The imported owner is not readable without signing in. +- [ ] Imported modules are found by OpenCatalogi's search, and appear in Portaliq's "Software we use" for the municipality's account, once both apps are configured as the docs describe. + +## Notes + +- Mapping decisions per column, including the columns that are not mapped because the target schema has no field, are listed in design.md. +- Connections, suites, hosting parties ("Hostingpartij"), "Leverancier", the archive sheet "Gearchiveerde Applicaties" and `missingRecords: mark|remove` are follow-ups (proposal, Out of Scope). +- Related: stackiq#373 (live TOPdesk connector), stackiq#1127 (record reconciliation), stackiq#1134 (ITSM exchange, the opposite direction), sbom-import and archimate-import (the upload patterns this follows). diff --git a/openspec/changes/cmdb-export-import/tasks.md b/openspec/changes/cmdb-export-import/tasks.md new file mode 100644 index 000000000..a04d31964 --- /dev/null +++ b/openspec/changes/cmdb-export-import/tasks.md @@ -0,0 +1,141 @@ +# Tasks: cmdb-export-import + +Spec: `openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md` (`SPEC` below). Contract: `contract.md` (authoritative for routes, request fields, report shape and error codes). + +## Implementation Tasks + +### Task 1: Sanitised test fixtures +- **spec_ref**: `SPEC#requirement-req-cmdb-002-the-workbook-shall-be-read-as-stored-data-without-evaluating-formulas-or-following-links` (cmdb-export-import#REQ-CMDB-002, also used by every other task) +- **files**: `tests/fixtures/cmdb/topdesk-export-anonymised.xlsx`, `tests/fixtures/cmdb/topdesk-missing-appid.xlsx`, `tests/fixtures/cmdb/topdesk-shuffled-columns.xlsx`, `tests/fixtures/cmdb/topdesk-formula-and-connection.xlsx`, `tests/fixtures/cmdb/README.md`, `tests/fixtures/cmdb/build-fixtures.py` +- **acceptance_criteria**: + - GIVEN the anonymised test export from the WOO-586 plan folder WHEN it is copied to `topdesk-export-anonymised.xlsx` THEN `docProps/core.xml` has no creator or lastModifiedBy, and `docProps/custom.xml`, `customXml/` and `xl/connections.xml` are removed, with their entries in `[Content_Types].xml` and the rels files + - GIVEN the sanitised fixture WHEN every shared string and cell value is scanned THEN no real person name, municipality domain, personnel number or phone number remains, only the placeholder values (`Achternaam, Voornaam`, `letter.achternaam@gemeente.nl`, `123456`) + - GIVEN `build-fixtures.py` WHEN it runs (Python stdlib zipfile only) THEN it writes placeholder cached values into the formula cells of the mapped CMDB columns (idempotent) and derives the variant fixtures: no "APPID" header on "Beheerde Applicaties CMDB"; both CMDB sheets with shuffled columns and header `Vendor⚡`; on "Beheerde" a formula in "Applicatie Naam" with cached value `Rekenmodel`, a "Roepnaam" formula without a cached value, plus a synthetic `xl/connections.xml` + - The original export of the municipality is never used or committed +- [x] Implement +- [x] Test (the scan is a PHPUnit test `tests/Unit/Fixtures/CmdbFixtureHygieneTest.php` that fails on metadata or non-placeholder person data) + +### Task 2: Register fragment with external-id properties and seed modules +- **spec_ref**: `SPEC#requirement-req-cmdb-006-a-module-shall-be-matched-on-its-topdesk-appid-so-a-re-import-updates-instead-of-duplicating` (cmdb-export-import#REQ-CMDB-006) +- **files**: `lib/Settings/register.d/topdesk-cmdb-import.json`, `tests/Unit/Settings/TopdeskCmdbFragmentTest.php` +- **acceptance_criteria**: + - GIVEN all `register.d` fragments WHEN they are merged in filename order the way `SettingsService` does THEN `module.version` is `0.3.5` and `externalId`, `externalNumber`, `externalKey`, `externalCreatedAt`, `externalModifiedAt` exist, none required, with titles (hydra gate schema-property-titles) + - GIVEN the fragment WHEN the register is imported on the rig THEN existing modules load and save unchanged, and the seed modules `voorbeeld-zaaksysteem`, `voorbeeld-afsprakenplanner` and `voorbeeld-belastingapplicatie` exist without `publicationDate` or `externalKey` (design.md, Seed Data) +- [x] Implement +- [x] Test + +### Task 3: Import profile, mapping packs and their loader +- **spec_ref**: `SPEC#requirement-req-cmdb-005-field-mapping-shall-be-declarative-and-executed-by-openregisters-mapping-engine` (cmdb-export-import#REQ-CMDB-005) +- **files**: `lib/Settings/cmdb-import/topdesk-profile.json`, `lib/Settings/cmdb-import/topdesk-module.json`, `lib/Settings/cmdb-import/topdesk-manufacturer.json`, `lib/Settings/cmdb-import/topdesk-municipality.json`, `lib/Settings/cmdb-import/topdesk-usage.json`, `lib/Settings/cmdb-import/topdesk-business-owner.json`, `lib/Service/Cmdb/CmdbImportProfile.php`, `lib/Exception/CmdbImportException.php`, `tests/Unit/Service/Cmdb/CmdbImportProfileTest.php` +- **acceptance_criteria**: + - GIVEN the five packs WHEN each is passed to OpenRegister's `PackDefinitionValidator` THEN all are valid with `sourceFormat: excel` and `idStrategy: generate`, and they implement the column table in design.md + - GIVEN a pack with an unknown transform, or no `MappingEngine` in the container WHEN the profile loads THEN it throws `CmdbImportException` with code `MAPPING_UNAVAILABLE` and status 503 + - GIVEN the profile WHEN its referenced columns are listed THEN no person or group column other than "Applicatie Eigenaar (Persoon)" / "(Functie)" is among them, and neither are the sheet constants +- [x] Implement +- [x] Test + +### Task 4: Workbook reader and row normaliser +- **spec_ref**: `SPEC#requirement-req-cmdb-002-…` and `SPEC#requirement-req-cmdb-003-columns-shall-be-resolved-by-header-name-and-a-missing-required-column-shall-stop-the-import-with-422` (cmdb-export-import#REQ-CMDB-002, #REQ-CMDB-003, #REQ-CMDB-005) +- **files**: `lib/Service/Cmdb/CmdbWorkbookReader.php`, `lib/Service/Cmdb/CmdbRowNormaliser.php`, `tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php`, `tests/Unit/Service/Cmdb/CmdbRowNormaliserTest.php` +- **acceptance_criteria**: + - GIVEN the sanitised fixture WHEN it is read THEN exactly one row per CMDB sheet is returned (empty formatted rows and formula rows that cached `0` dropped), keyed by profile column names, with only allowlisted columns + - GIVEN the formula/connection fixture WHEN it is read THEN "Applicatie Naam" is `Rekenmodel`, "Roepnaam" is empty and listed in the row's `uncached`, `getCalculatedValue()` is never called, and no HTTP client is involved + - GIVEN the shuffled fixture WHEN it is read THEN rows equal those of the original; GIVEN the missing-column fixture THEN `MISSING_COLUMN` names `APPID` and `Beheerde Applicaties CMDB`; GIVEN only "Blad1" THEN `NO_SOURCE_SHEET`; GIVEN more than `maxRowsPerSheet` rows THEN `TOO_MANY_ROWS` + - GIVEN a text file named `.xlsx`, or a `.xlsm` WHEN checked THEN `NOT_XLSX` before PhpSpreadsheet is touched; GIVEN PhpSpreadsheet absent THEN `READER_UNAVAILABLE` + - GIVEN serials `45111.380322627316`, `46232.552113113423`, `53359` and id `1234.0` WHEN normalised THEN `2023-07-04`, `2026-07-29`, `2046-02-01` and `"1234"`; GIVEN "BNN Classificatie" `NB` and "End-of-Life Functioneel" `49675` THEN both are empty +- [x] Implement +- [x] Test + +### Task 5: Import service: municipality, manufacturer, module upsert, usage +- **spec_ref**: `SPEC#requirement-req-cmdb-004-every-import-shall-have-exactly-one-consuming-municipality-chosen-by-the-admin`, `SPEC#requirement-req-cmdb-006-…`, `SPEC#requirement-req-cmdb-007-a-newly-created-module-shall-get-a-publicationdate-and-an-existing-one-shall-keep-its-own`, `SPEC#requirement-req-cmdb-008-a-manufacturer-shall-become-one-supplier-organisation-however-many-rows-name-it`, `SPEC#requirement-req-cmdb-009-each-imported-application-shall-have-one-usage-that-links-it-to-the-municipality`, `SPEC#requirement-req-cmdb-012-records-missing-from-a-newer-export-shall-be-left-untouched` +- **files**: `lib/Service/CmdbExportImportService.php`, `tests/Unit/Service/CmdbExportImportServiceTest.php` +- **acceptance_criteria**: + - GIVEN the sanitised fixture and "Gemeente Voorbeeldstad" WHEN imported THEN two modules with `externalKey` `topdesk::`, `publicationDate` = import start, `provider` set, and two usages with `consumer` = the municipality and `module` = the module + - GIVEN the same import twice WHEN run THEN 0 created / 2 unchanged and no `saveObject()` call for unchanged objects; GIVEN a changed "Applicatie Naam" THEN one module updated; GIVEN a changed "Applicatie Code" for the same APPID THEN the same module updated, `website`, `publicationDate` and `depublicationDate` untouched + - GIVEN `municipalityName` twice THEN one Municipality; GIVEN the uuid of a Supplier THEN `MUNICIPALITY_INVALID` + - GIVEN "Vendor" "Fabfrikant", "Fabfrikant " and "FABFRIKANT" THEN one Supplier; GIVEN an existing Supplier with the same name THEN it is reused + - GIVEN `updateExisting=false` THEN matched rows are `skipped` (`exists`); GIVEN a second export without one APPID THEN that module and usage are unchanged; GIVEN an unknown "Applicatie Status" THEN `status` is dropped with a warning naming column and value + - GIVEN the module pack mapping "Software Suite" to licentietype (test-only pack) THEN the module carries it, with no code change + - GIVEN a row from each sheet THEN the usage note starts with `Beheer geregeld: nee` (Onbeh) or `ja` (Beheerde), followed by the non-empty Cluster and Afdeling + - Every new method carries `@spec openspec/changes/cmdb-export-import/tasks.md#task-5` (hydra gate spec-coverage) +- [x] Implement +- [x] Test + +### Task 6: Owners as contact persons through Nextcloud Contacts +- **spec_ref**: `SPEC#requirement-req-cmdb-010-the-owner-shall-become-a-contact-person-of-the-municipality-through-nextcloud-contacts-never-a-user-account-and-shall-never-be-publicly-readable` (cmdb-export-import#REQ-CMDB-010) +- **files**: `lib/Service/CmdbExportImportService.php`, `tests/Unit/Service/CmdbExportImportServiceTest.php` +- **acceptance_criteria**: + - GIVEN the fixture WHEN imported THEN each row's "Applicatie Eigenaar (Persoon)" (a name, or a function) resolves to a contact by display name, one `contactPerson` per owner exists with that `contactsUid`, `organization` = municipality and `role` = "Applicatie Eigenaar (Functie)", and it is the usage's `businessOwner`; no `technicalOwner` is written + - GIVEN an owner imported twice THEN one contact (exact display-name match) and one `contactPerson` + - GIVEN the merged register THEN `usage` and `contactPerson` have no public read rule and a `module` refers to them by relation only (`tests/Unit/Settings/CmdbPersonDataVisibilityTest.php`); GIVEN the rig THEN an anonymous OpenCatalogi search hit carries no owner and OpenRegister returns no contact person or usage anonymously (e2e) + - GIVEN Contacts disabled WHEN imported THEN modules and usages are saved, owners skipped with a warning + - GIVEN an imported `contactPerson` WHEN `OrganizationSyncService::performUserSync`'s selection is applied THEN it is not selected, and no Nextcloud user is created (if it would be, add an exclusion marker before shipping) + - GIVEN any import WHEN the report and log lines are inspected THEN no owner name or e-mail appears +- [x] Implement +- [x] Test + +### Task 7: Row isolation, report, progress and cancel +- **spec_ref**: `SPEC#requirement-req-cmdb-011-each-row-shall-be-processed-in-isolation-and-reported-with-its-outcome`, `SPEC#requirement-req-cmdb-013-a-running-import-shall-report-its-progress-and-shall-stop-when-cancelled` +- **files**: `lib/Service/CmdbExportImportService.php`, `lib/Service/Cmdb/CmdbImportReport.php`, `tests/Unit/Service/CmdbExportImportServiceTest.php` +- **acceptance_criteria**: + - GIVEN three rows where saving the second module throws WHEN imported THEN rows 1 and 3 are `created`, row 2 is `failed` naming the step, and the summary matches contract.md + - GIVEN duplicate APPID rows (also across both sheets), a missing APPID and a missing "Applicatie Naam" THEN they are `skipped` with the reasons in the spec; GIVEN a formula without a cached value THEN the row is imported with a warning naming the column + - GIVEN an `operationId` WHEN the import runs THEN a `cmdb_import` operation reports per-row progress, and after completion its statistics hold the report + - GIVEN cancel requested after row 1 of three THEN one processed row, `cancelled: true`, row 1's objects kept +- [x] Implement +- [x] Test + +### Task 8: Controller, routes and API tests +- **spec_ref**: `SPEC#requirement-req-cmdb-001-the-import-endpoint-shall-accept-only-a-bounded-xlsx-upload-from-a-nextcloud-admin` (cmdb-export-import#REQ-CMDB-001, #REQ-CMDB-012, #REQ-CMDB-013) +- **files**: `lib/Controller/CmdbImportController.php`, `appinfo/routes.php`, `tests/Unit/Controller/CmdbImportControllerTest.php`, `postman/stackiq-tests.json`, `openapi.json` +- **acceptance_criteria**: + - GIVEN `cmdbImport#import` and `cmdbImport#cancel` WHEN their attributes are inspected THEN neither has `NoAdminRequired` or `NoCSRFRequired` (hydra gates route-auth, csrf-cochange, no-admin-idor) + - GIVEN the validation order in design.md D10 THEN each error code from contract.md is returned with its status, and every service exception is translated (hydra gate controller-exception-translation) + - GIVEN Newman WHEN run against the rig THEN 403 for a non-admin and for a `software-catalog-admins` member, 412 without requesttoken, 413 for an oversized file, 422 `MISSING_RECORDS_UNSUPPORTED`, and 200 with the report for the fixture +- [x] Implement +- [ ] Test + +### Task 9: CMDB import section in admin settings, l10n and Playwright e2e +- **spec_ref**: `SPEC#requirement-req-cmdb-014-the-admin-settings-shall-offer-a-cmdb-import-section` (cmdb-export-import#REQ-CMDB-014, #REQ-CMDB-003, #REQ-CMDB-011) +- **files**: `src/views/settings/sections/CmdbImport.vue`, `src/views/settings/StackiqSettings.vue`, `l10n/en.json`, `l10n/en.js`, `l10n/nl.json`, `l10n/nl.js`, `tests/e2e/spec-coverage/cmdb-import.spec.ts` +- **acceptance_criteria**: + - GIVEN a Nextcloud admin on stackiq's admin settings WHEN they choose "Gemeente Voorbeeldstad" and the sanitised fixture and press Import THEN a progress bar shows, then the summary (2 read, 2 created) and a `CnDataTable` report filterable by outcome with links to the modules + - GIVEN a second import of the same file THEN the report shows 2 unchanged; GIVEN the missing-column fixture THEN the section shows column `APPID` and sheet `Beheerde Applicaties CMDB`; GIVEN a CSV THEN it shows the `NOT_XLSX` message + - GIVEN the section WHEN the hydra gates run THEN admin-router, form-label-association, nc-input-labels, button-name, table-headers and modal-isolation pass, and no `v-html` renders report values + - GIVEN a Dutch and an English locale THEN every new string, error message and report reason is translated + - The e2e file references every `@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts` scenario in the spec (hydra gate e2e-coverage) +- [x] Implement +- [ ] Test + +### Task 10: Administrator documentation with screenshots +- **spec_ref**: `SPEC#purpose` +- **files**: `docs/features/cmdb-import.md`, `docs/images/cmdb-import-*.png`, `docs/features/README.md` +- **acceptance_criteria**: + - GIVEN the docs page WHEN an administrator reads it THEN it covers the steps, the expected file structure (sheets, required and mapped columns, the column table), the error codes and what to do, repeat-import behaviour (match on APPID per municipality, unchanged rows, records missing from the export stay, publicationDate rule), where owner contacts end up, and how to adjust the mapping JSON + - GIVEN the prerequisites section THEN it explains the OpenCatalogi catalogue (registers `stackiq`, schema `module`) and the Portaliq account claim `stackiq.organisationId`, needed to see the data there + - GIVEN Playwright MCP on the rig WHEN screenshots are taken of the empty section, a running import and a finished report (sanitised fixture only) THEN they are committed under `docs/images/` +- [ ] Implement +- [ ] Test (screenshots reviewed: no data other than the sanitised fixture visible) + +### Task 11: Rework to the CMDB sheets (WOO-586 Stap 4b, decisions of 2026-10-01) +- **spec_ref**: `SPEC#requirement-req-cmdb-003-columns-shall-be-resolved-by-header-name-and-a-missing-required-column-shall-stop-the-import-with-422`, `SPEC#requirement-req-cmdb-006-a-module-shall-be-matched-on-its-topdesk-appid-so-a-re-import-updates-instead-of-duplicating`, `SPEC#requirement-req-cmdb-010-the-owner-shall-become-a-contact-person-of-the-municipality-through-nextcloud-contacts-never-a-user-account-and-shall-never-be-publicly-readable` +- **files**: `lib/Settings/cmdb-import/*.json`, `lib/Service/Cmdb/*`, `lib/Service/CmdbExportImportService.php`, `lib/Settings/register.d/topdesk-cmdb-import.json`, `src/views/settings/sections/CmdbImport.vue`, `src/utils/cmdbImport.js`, `l10n/*`, `tests/fixtures/cmdb/*`, `tests/Unit/**/Cmdb*`, `tests/Unit/Settings/CmdbPersonDataVisibilityTest.php`, `tests/e2e/spec-coverage/cmdb-import.spec.ts`, `docs/features/cmdb-import.md`, `openapi.json`, `postman/stackiq-tests.json` +- **acceptance_criteria**: + - The source sheets are "Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB"; the "Invoer" sheets are not read; columns resolve by header name per sheet + - `externalKey` = `topdesk::`; "Applicatie Code" → `externalId`, APPID → `externalNumber`; rows without APPID are skipped (`missing APPID`); the report row field is `appId` + - The column table of design.md is implemented, including `cloudDienstverleningsmodel`, `bbnLevel`, `timeClassification`, `startDateOutPhased` and the maintenance note; the technical-owner pack is removed + - Formula cells give their cached value; no cached value gives an empty cell and a row warning, never a failure + - The rig data of the first import is removed and the fixture re-imported twice (created, then unchanged); the owner is not readable anonymously +- [x] Implement +- [x] Test + +## Quality checklist + +- PHPUnit for all new business logic (`tests/Unit/`), at least 75% coverage of new code (ADR-009), using the sanitised xlsx fixtures (not mocked rows) for reader and service tests +- Newman/Postman for both new endpoints (Task 8); Playwright for the settings flow (Task 9) +- `composer test`, `newman run` and the Playwright spec pass on the local rig +- Test against OpenRegister on the rig: the saved objects pass schema validation (module 0.3.5, organization, usage, contactPerson) +- Hydra gates run locally (`scripts/run-hydra-gates.sh`); read the COVERAGE line and name any SKIPPED gate +- Dutch (`nl_NL`) and English (`en_US`) strings for every new user-facing string (ADR-005) +- Docs in `docs/features/cmdb-import.md` with screenshots (ADR-010) +- `openspec validate cmdb-export-import` passes diff --git a/openspec/changes/cmdb-export-import/test-plan.md b/openspec/changes/cmdb-export-import/test-plan.md new file mode 100644 index 000000000..19378bfe8 --- /dev/null +++ b/openspec/changes/cmdb-export-import/test-plan.md @@ -0,0 +1,147 @@ +# Test Plan: cmdb-export-import + +Spec: `openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md` (abbreviated `spec.md` below). Fixture: `tests/fixtures/cmdb/topdesk-export-anonymised.xlsx` (one fake data row per source sheet, metadata removed). Municipality in every case: "Gemeente Voorbeeldstad". Environment: local rig (Deploy-target n.v.t.). + +## Test Cases + +### TC-1: Admin imports the export and sees a per-row report +- **spec_ref**: `spec.md#requirement-req-cmdb-011-each-row-shall-be-processed-in-isolation-and-reported-with-its-outcome`, `#requirement-req-cmdb-014-the-admin-settings-shall-offer-a-cmdb-import-section`, `#requirement-req-cmdb-004-every-import-shall-have-exactly-one-consuming-municipality-chosen-by-the-admin` +- **type**: functional +- **persona**: Noor Yilmaz (Municipal CISO / Functional Admin) +- **preconditions**: Nextcloud admin; "Gemeente Voorbeeldstad" exists as type Municipality; no imported modules +- **steps**: open stackiq admin settings, section "CMDB import", choose the municipality, choose the fixture, press Import +- **expected result**: progress bar during the run; summary 2 read / 2 created; report rows `Onbeh Applicaties CMDB` row 2 APPID `1234` and `Beheerde Applicaties CMDB` row 2 APPID `2`, each `created` and linking to its module; no empty rows listed +- **test command**: Playwright `tests/e2e/spec-coverage/cmdb-import.spec.ts`, `/test-functional`, `/test-persona-noor` + +### TC-2: Re-import creates no duplicates +- **spec_ref**: `spec.md#requirement-req-cmdb-006-a-module-shall-be-matched-on-its-topdesk-appid-so-a-re-import-updates-instead-of-duplicating`, `#requirement-req-cmdb-009-each-imported-application-shall-have-one-usage-that-links-it-to-the-municipality` +- **type**: functional +- **persona**: Noor Yilmaz +- **preconditions**: TC-1 done; object counts of module, organization, usage, contactPerson recorded +- **steps**: import the same fixture again for the same municipality +- **expected result**: 0 created, 2 unchanged; all four counts equal to before +- **test command**: Playwright `cmdb-import.spec.ts`; PHPUnit `CmdbExportImportServiceTest` + +### TC-3: Changed fields update, publicationDate and unmapped fields are kept +- **spec_ref**: `spec.md#requirement-req-cmdb-006-…`, `#requirement-req-cmdb-007-a-newly-created-module-shall-get-a-publicationdate-and-an-existing-one-shall-keep-its-own` +- **type**: api +- **preconditions**: modules imported; an admin set `website` on APPID `2` and depublished it +- **steps**: import rows where "Applicatie Naam" of APPID `2` is `naamtest124`, and where the "Applicatie Code" of APPID `42` changed +- **expected result**: same uuid, name `naamtest124`, `website` unchanged, `depublicationDate` unchanged, no new `publicationDate`; APPID `1234` keeps its original `publicationDate`; APPID `42` is the same module with the new `externalId` +- **test command**: PHPUnit `tests/Unit/Service/CmdbExportImportServiceTest.php` + +### TC-4: Manufacturer dedup +- **spec_ref**: `spec.md#requirement-req-cmdb-008-a-manufacturer-shall-become-one-supplier-organisation-however-many-rows-name-it` +- **type**: api +- **preconditions**: an existing Supplier `Aangetekend B.V.` +- **steps**: import rows with "Vendor" `Fabfrikant`, `Fabfrikant `, `FABFRIKANT`, and the "Onbeh" row +- **expected result**: one new Supplier `Fabfrikant`; `Aangetekend B.V.` reused; module and usage `provider` set accordingly +- **test command**: PHPUnit `CmdbExportImportServiceTest` + +### TC-5: Upload validation (type, size, columns, sheets, options) +- **spec_ref**: `spec.md#requirement-req-cmdb-001-the-import-endpoint-shall-accept-only-a-bounded-xlsx-upload-from-a-nextcloud-admin`, `#requirement-req-cmdb-003-columns-shall-be-resolved-by-header-name-and-a-missing-required-column-shall-stop-the-import-with-422`, `#requirement-req-cmdb-012-records-missing-from-a-newer-export-shall-be-left-untouched` +- **type**: api +- **preconditions**: admin session +- **steps**: post `applications.csv`; a text file named `.xlsx`; a 10 MB + 1 byte file; `topdesk-missing-appid.xlsx`; a workbook with only "Blad1"; the fixture with `missingRecords=remove`; the fixture without a municipality +- **expected result**: 400 `NOT_XLSX` (twice), 413 `FILE_TOO_LARGE`, 422 `MISSING_COLUMN` naming `APPID` and `Beheerde Applicaties CMDB`, 422 `NO_SOURCE_SHEET`, 422 `MISSING_RECORDS_UNSUPPORTED`, 422 `MUNICIPALITY_REQUIRED`; no object written in any case +- **test command**: PHPUnit `CmdbImportControllerTest`, Newman (Postman collection), `/test-api`; the missing-column UI message also in Playwright + +### TC-6: Authorisation and CSRF +- **spec_ref**: `spec.md#requirement-req-cmdb-001-…`, `#requirement-req-cmdb-013-a-running-import-shall-report-its-progress-and-shall-stop-when-cancelled` +- **type**: security +- **preconditions**: a non-admin user, also one in `software-catalog-admins` +- **steps**: post the fixture and the cancel route as that user; post as admin without `requesttoken` +- **expected result**: 403 for non-admins; 412 without CSRF token; no object written +- **test command**: Newman, `/test-security` + +### TC-7: Safe reading (formulas, external connection, column order) +- **spec_ref**: `spec.md#requirement-req-cmdb-002-the-workbook-shall-be-read-as-stored-data-without-evaluating-formulas-or-following-links`, `#requirement-req-cmdb-003-…` +- **type**: security +- **preconditions**: fixtures `topdesk-formula-and-connection.xlsx`, `topdesk-shuffled-columns.xlsx` +- **steps**: read both through `CmdbWorkbookReader` +- **expected result**: formula cell yields the cached `Rekenmodel`, calculation engine never invoked; no network access; shuffled columns give identical rows; disallowed columns (Personeelsnummer, phones, group mailbox) are absent from the reader output +- **test command**: PHPUnit `tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php` + +### TC-8: Normalisation and declarative mapping +- **spec_ref**: `spec.md#requirement-req-cmdb-005-field-mapping-shall-be-declarative-and-executed-by-openregisters-mapping-engine` +- **type**: api +- **preconditions**: fixture rows; an alternate module pack mapping "Roepnaam" to `shortDescription` +- **steps**: normalise and map the rows through the real `MappingEngine` +- **expected result**: `2023-07-04`, `2026-07-29`, `2046-02-01`, `"1234"`; alternate pack yields `shortDescription`; an unknown "Status" drops only `status` with a warning; an invalid pack or a missing engine gives 503 `MAPPING_UNAVAILABLE` +- **test command**: PHPUnit `CmdbRowNormaliserTest`, `CmdbImportProfileTest`, `CmdbExportImportServiceTest` + +### TC-9: Per-row isolation and cancel +- **spec_ref**: `spec.md#requirement-req-cmdb-011-…`, `#requirement-req-cmdb-013-…` +- **type**: regression +- **preconditions**: three rows; `saveObject()` throws for the second module; separately, cancel requested after row 1 +- **steps**: run the import twice +- **expected result**: run 1: rows 1 and 3 created, row 2 failed naming the step, HTTP 200; run 2: 1 processed row, `cancelled: true`, row 1's module kept +- **test command**: PHPUnit `CmdbExportImportServiceTest` + +### TC-10: Owners as contact persons, no user accounts +- **spec_ref**: `spec.md#requirement-req-cmdb-010-the-owner-shall-become-a-contact-person-of-the-municipality-through-nextcloud-contacts-never-a-user-account-and-shall-never-be-publicly-readable` +- **type**: security +- **preconditions**: Contacts enabled (test double); separately disabled +- **steps**: import a row twice and a second row with the same "Applicatie Eigenaar (Persoon)"; import the fixture, whose "Beheerde" owner is a function; run `performUserSync` selection on the result; then, not signed in, list contact persons and usages through OpenRegister and search OpenCatalogi for `naamtest123` +- **expected result**: one contactPerson per owner with the function as `role` and `organization` = municipality, set as `businessOwner`; no `technicalOwner`; no Nextcloud user created and the contactPerson not selected by the user sync; with Contacts disabled: no owners, a warning, modules and usages saved; report and log contain no owner name; anonymously: no contact person or usage from OpenRegister, and the OpenCatalogi hit holds no owner name and only ids in `contactPerson` / `usages` +- **test command**: PHPUnit `CmdbExportImportServiceTest`, `CmdbPersonDataVisibilityTest`, Playwright `cmdb-import.spec.ts` (anonymous test), `/test-security` + +### TC-11: OpenCatalogi finds an imported application +- **spec_ref**: `spec.md#requirement-req-cmdb-007-…` +- **type**: functional +- **persona**: Sem de Jong (Young Digital Native; anonymous search) +- **preconditions**: OpenCatalogi catalogue with registers `[stackiq]`, schemas `[module]`, listed and published (docs, prerequisites); TC-1 done +- **steps**: anonymous `GET /apps/opencatalogi/api/search?_search=Aangetekend` +- **expected result**: one hit `Aangetekend Mailen` +- **test command**: manual on the rig (USER MANUAL TEST, WOO-586 Stap 6b), `/test-functional` + +### TC-12: Portaliq shows the applications to the municipality +- **spec_ref**: `spec.md#requirement-req-cmdb-009-…` +- **type**: persona +- **persona**: Noor Yilmaz (Municipal CISO / Functional Admin) +- **preconditions**: Portaliq account with claim `stackiq.organisationId` = uuid of "Gemeente Voorbeeldstad", audience participant-org; TC-1 done +- **steps**: sign in to the portal, open "Software we use" +- **expected result**: `Aangetekend Mailen` and `naamtest123` listed; an account for another organisation sees neither +- **test command**: manual on the rig, `/test-persona-noor` + +### TC-13: Accessibility of the section +- **spec_ref**: `spec.md#requirement-req-cmdb-014-…` +- **type**: accessibility +- **preconditions**: section rendered with a finished report +- **steps**: keyboard-only run of TC-1; axe scan; screen-reader check of progress and summary +- **expected result**: every control labelled and reachable; progress and summary announced through a polite live region; report table has header cells; no serious/critical axe violations +- **test command**: `/test-accessibility`, hydra gates `form-label-association`, `nc-input-labels`, `button-name`, `table-headers`, `axe` + +### TC-14: Register fragment deploys the module properties +- **spec_ref**: `spec.md#requirement-req-cmdb-006-…` (stored key), design.md Mixed-spec rationale +- **type**: regression +- **preconditions**: all `register.d` fragments present +- **steps**: merge the register as `SettingsService` does; run the repair step on the rig +- **expected result**: merged `module.version` is `0.3.5` with the five optional properties; existing modules still load and save; seed module `voorbeeld-zaaksysteem` present without `publicationDate` +- **test command**: PHPUnit `tests/Unit/Settings/TopdeskCmdbFragmentTest.php`, `/test-regression` + +## Coverage Summary + +| Requirement | Covered by | +|---|---| +| REQ-CMDB-001 upload bounds, admin, CSRF | TC-5, TC-6 | +| REQ-CMDB-002 safe reading | TC-7 | +| REQ-CMDB-003 header-name columns, 422 | TC-5, TC-7 | +| REQ-CMDB-004 one municipality | TC-1, TC-5, PHPUnit (created once) | +| REQ-CMDB-005 declarative mapping, dates | TC-8 | +| REQ-CMDB-006 upsert on APPID | TC-2, TC-3, TC-14 | +| REQ-CMDB-007 publicationDate rule | TC-3, TC-11 | +| REQ-CMDB-008 manufacturer dedup | TC-4 | +| REQ-CMDB-009 usage per municipality | TC-2, TC-12 | +| REQ-CMDB-010 owner via Contacts, never public | TC-10 | +| REQ-CMDB-011 per-row isolation and report | TC-1, TC-9 | +| REQ-CMDB-012 missing records kept | TC-5, PHPUnit (dropped row stays) | +| REQ-CMDB-013 progress and cancel | TC-6, TC-9 | +| REQ-CMDB-014 settings section | TC-1, TC-13 | + +All requirements are covered. After implementation, TC-1, TC-2 and TC-11/12 are candidates for `/test-scenario-create` (key user flow and cross-app chain). + +## Out of Scope + +- Performance on the real 1,100-row export: the real file never enters a repo or a test run. It is measured once, manually, on the rig, and only the timing is recorded. +- The archive sheet, connections, suites and hosting parties are not built, so they are not tested. diff --git a/openspec/specs/cmdb-export-import/spec.md b/openspec/specs/cmdb-export-import/spec.md new file mode 100644 index 000000000..0ccc1f53e --- /dev/null +++ b/openspec/specs/cmdb-export-import/spec.md @@ -0,0 +1,51 @@ +--- +capability: cmdb-export-import +status: in-progress +built_by: openspec/changes/cmdb-export-import +--- + +# cmdb-export-import Specification + +**Status**: in-progress +**Scope**: stackiq +**OpenSpec changes**: +- [cmdb-export-import](../../changes/cmdb-export-import/) _(active)_ — admin uploads a TOPdesk CMDB export (xlsx); stackiq upserts modules, vendor organisations, usages and owner contact persons for one municipality from the two CMDB sheets, matched on APPID, mapped by OpenRegister migration packs (kind: code) + +## Purpose + +A Nextcloud admin imports a TOPdesk CMDB export (xlsx) into stackiq for one +municipality. Every application row becomes, or updates, a `module` with its +manufacturer `organization`, a `usage` that links it to the municipality, and +`contactPerson` objects for its owners, all stored as OpenRegister objects +(ADR-001). The mapping is declarative JSON executed by OpenRegister's mapping +engine (ADR-031), so a newer export can be imported again without duplicates, +and OpenCatalogi and Portaliq can show the result (Jira WOO-586). + +## Requirements + +Detailed requirements (REQ-CMDB-001 … REQ-CMDB-014) are defined in the active +change's delta spec — +[`openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md`](../../changes/cmdb-export-import/specs/cmdb-export-import/spec.md) +— and are merged here by `openspec sync` when the change is archived. The +umbrella requirement below anchors the capability until then. + +### Requirement: Stackiq imports a TOPdesk CMDB export into OpenRegister objects (REQ-CMDB-000) + +Stackiq MUST offer Nextcloud admins one import path for a TOPdesk CMDB export +(xlsx) that writes only OpenRegister objects in the `stackiq` register +(`module`, `organization`, `usage`, `contactPerson`), with no app-local table, +and that matches rows on the TOPdesk APPID so that a repeated import +creates no duplicates. + +#### Scenario: A repeated import adds no objects + +- GIVEN a TOPdesk export imported once for a municipality +- WHEN the same export is imported again for that municipality +- THEN the number of `module`, `organization`, `usage` and `contactPerson` objects SHALL be unchanged +- @e2e exclude umbrella anchor; the behaviour is covered by REQ-CMDB-006 in the change's delta spec (tests/e2e/spec-coverage/cmdb-import.spec.ts and tests/Unit/Service/CmdbExportImportServiceTest.php) + +## Notes + +- Follows the upload patterns of `sbom-import` and `archimate-import`. +- Related: stackiq#373 (live TOPdesk connector), stackiq#1127 (record + reconciliation), stackiq#1134 (ITSM exchange). diff --git a/postman/stackiq-tests.json b/postman/stackiq-tests.json index 319fbf649..f5544dc6a 100644 --- a/postman/stackiq-tests.json +++ b/postman/stackiq-tests.json @@ -22547,6 +22547,498 @@ ] } ] + }, + { + "name": "12 - CMDB import", + "description": "openspec/changes/cmdb-export-import (contract.md). Run newman from the app root so the fixture path tests/fixtures/cmdb/ resolves.", + "item": [ + { + "name": "CMDB import: 403 for a user who is not a Nextcloud admin", + "request": { + "method": "POST", + "header": [ + { + "key": "OCS-APIRequest", + "value": "true", + "type": "text" + } + ], + "url": { + "raw": "{{stackiq_api}}/cmdb-import", + "host": [ + "{{stackiq_api}}" + ], + "path": [ + "cmdb-import" + ] + }, + "auth": { + "type": "basic", + "basic": [ + { + "key": "username", + "value": "mark.jansen@test.nl" + }, + { + "key": "password", + "value": "{{test_password}}" + } + ] + }, + "body": { + "mode": "formdata", + "formdata": [ + { + "key": "cmdbFile", + "type": "file", + "src": "tests/fixtures/cmdb/topdesk-export-anonymised.xlsx" + }, + { + "key": "municipalityName", + "value": "Gemeente Voorbeeldstad", + "type": "text" + } + ] + } + }, + "response": [], + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "pm.test(\"A non-admin cannot import\", function () {", + " pm.response.to.have.status(403);", + "});" + ], + "type": "text/javascript" + } + } + ] + }, + { + "name": "CMDB import: 403 for a software-catalog-admins member", + "request": { + "method": "POST", + "header": [ + { + "key": "OCS-APIRequest", + "value": "true", + "type": "text" + } + ], + "url": { + "raw": "{{stackiq_api}}/cmdb-import", + "host": [ + "{{stackiq_api}}" + ], + "path": [ + "cmdb-import" + ] + }, + "auth": { + "type": "basic", + "basic": [ + { + "key": "username", + "value": "peter.vandijk@test.nl" + }, + { + "key": "password", + "value": "{{test_password}}" + } + ] + }, + "body": { + "mode": "formdata", + "formdata": [ + { + "key": "cmdbFile", + "type": "file", + "src": "tests/fixtures/cmdb/topdesk-export-anonymised.xlsx" + }, + { + "key": "municipalityName", + "value": "Gemeente Voorbeeldstad", + "type": "text" + } + ] + } + }, + "response": [], + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "pm.test(\"The catalogue admin group is not enough\", function () {", + " pm.response.to.have.status(403);", + "});" + ], + "type": "text/javascript" + } + } + ] + }, + { + "name": "CMDB import: 412 without a CSRF token", + "request": { + "method": "POST", + "header": [ + { + "key": "OCS-APIRequest", + "value": "true", + "type": "text" + } + ], + "url": { + "raw": "{{stackiq_api}}/cmdb-import", + "host": [ + "{{stackiq_api}}" + ], + "path": [ + "cmdb-import" + ] + }, + "auth": { + "type": "basic", + "basic": [ + { + "key": "username", + "value": "{{admin_user}}" + }, + { + "key": "password", + "value": "{{admin_pass}}" + } + ] + }, + "body": { + "mode": "formdata", + "formdata": [ + { + "key": "cmdbFile", + "type": "file", + "src": "tests/fixtures/cmdb/topdesk-export-anonymised.xlsx" + }, + { + "key": "municipalityName", + "value": "Gemeente Voorbeeldstad", + "type": "text" + } + ] + } + }, + "response": [], + "event": [ + { + "listen": "prerequest", + "script": { + "exec": [ + "// The collection adds OCS-APIRequest, which satisfies the CSRF check; this request must go without it.", + "pm.request.headers.remove(\"OCS-APIRequest\");" + ], + "type": "text/javascript" + } + }, + { + "listen": "test", + "script": { + "exec": [ + "pm.test(\"Nextcloud refuses the request without a CSRF token\", function () {", + " pm.response.to.have.status(412);", + "});" + ], + "type": "text/javascript" + } + } + ] + }, + { + "name": "CMDB import: 413 for a file over 10 MB", + "request": { + "method": "POST", + "header": [ + { + "key": "OCS-APIRequest", + "value": "true", + "type": "text" + } + ], + "url": { + "raw": "{{stackiq_api}}/cmdb-import", + "host": [ + "{{stackiq_api}}" + ], + "path": [ + "cmdb-import" + ] + }, + "auth": { + "type": "basic", + "basic": [ + { + "key": "username", + "value": "{{admin_user}}" + }, + { + "key": "password", + "value": "{{admin_pass}}" + } + ] + }, + "body": { + "mode": "formdata", + "formdata": [ + { + "key": "cmdbFile", + "type": "file", + "src": "{{cmdb_oversized_file}}" + }, + { + "key": "municipalityName", + "value": "Gemeente Voorbeeldstad", + "type": "text" + } + ] + }, + "description": "Set cmdb_oversized_file to a file of 10485761 bytes; the request is skipped when it is not set." + }, + "response": [], + "event": [ + { + "listen": "prerequest", + "script": { + "exec": [ + "// Needs a file of 10 MB plus one byte, e.g. `head -c 10485761 /dev/zero > /tmp/cmdb-oversized.xlsx`,", + "// passed as --env-var cmdb_oversized_file=/tmp/cmdb-oversized.xlsx. Without it the request is skipped, not passed.", + "if (!pm.variables.get(\"cmdb_oversized_file\")) {", + " pm.execution.skipRequest();", + "}" + ], + "type": "text/javascript" + } + }, + { + "listen": "test", + "script": { + "exec": [ + "pm.test(\"An oversized upload is refused before it is read\", function () {", + " pm.response.to.have.status(413);", + " pm.expect(pm.response.json().error).to.eql(\"FILE_TOO_LARGE\");", + "});" + ], + "type": "text/javascript" + } + } + ] + }, + { + "name": "CMDB import: 422 MISSING_RECORDS_UNSUPPORTED for missingRecords=remove", + "request": { + "method": "POST", + "header": [ + { + "key": "OCS-APIRequest", + "value": "true", + "type": "text" + } + ], + "url": { + "raw": "{{stackiq_api}}/cmdb-import", + "host": [ + "{{stackiq_api}}" + ], + "path": [ + "cmdb-import" + ] + }, + "auth": { + "type": "basic", + "basic": [ + { + "key": "username", + "value": "{{admin_user}}" + }, + { + "key": "password", + "value": "{{admin_pass}}" + } + ] + }, + "body": { + "mode": "formdata", + "formdata": [ + { + "key": "cmdbFile", + "type": "file", + "src": "tests/fixtures/cmdb/topdesk-export-anonymised.xlsx" + }, + { + "key": "municipalityName", + "value": "Gemeente Voorbeeldstad", + "type": "text" + }, + { + "key": "missingRecords", + "value": "remove", + "type": "text" + } + ] + } + }, + "response": [], + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "pm.test(\"Only missingRecords=keep is accepted\", function () {", + " pm.response.to.have.status(422);", + " pm.expect(pm.response.json().error).to.eql(\"MISSING_RECORDS_UNSUPPORTED\");", + "});" + ], + "type": "text/javascript" + } + } + ] + }, + { + "name": "CMDB import: 200 with the report for the anonymised export", + "request": { + "method": "POST", + "header": [ + { + "key": "OCS-APIRequest", + "value": "true", + "type": "text" + } + ], + "url": { + "raw": "{{stackiq_api}}/cmdb-import", + "host": [ + "{{stackiq_api}}" + ], + "path": [ + "cmdb-import" + ] + }, + "auth": { + "type": "basic", + "basic": [ + { + "key": "username", + "value": "{{admin_user}}" + }, + { + "key": "password", + "value": "{{admin_pass}}" + } + ] + }, + "body": { + "mode": "formdata", + "formdata": [ + { + "key": "cmdbFile", + "type": "file", + "src": "tests/fixtures/cmdb/topdesk-export-anonymised.xlsx" + }, + { + "key": "municipalityName", + "value": "Gemeente Voorbeeldstad", + "type": "text" + }, + { + "key": "updateExisting", + "value": "true", + "type": "text" + }, + { + "key": "missingRecords", + "value": "keep", + "type": "text" + } + ] + } + }, + "response": [], + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "pm.test(\"The export is imported with a per-row report\", function () {", + " pm.response.to.have.status(200);", + " var json = pm.response.json();", + " pm.expect(json.success).to.eql(true);", + " pm.expect(json.municipality.name).to.eql(\"Gemeente Voorbeeldstad\");", + " pm.expect(json.summary.rowsRead).to.eql(2);", + " pm.expect(json.summary.created + json.summary.unchanged + json.summary.updated).to.eql(2);", + " pm.expect(json.summary.failed).to.eql(0);", + " pm.expect(json.rows.map(function (r) { return r.appId; })).to.eql([\"1234\", \"2\"]);", + " pm.expect(json.rows.map(function (r) { return r.sheet; })).to.eql([\"Onbeh Applicaties CMDB\", \"Beheerde Applicaties CMDB\"]);", + " pm.environment.set(\"cmdb_operation_id\", json.operationId);", + "});" + ], + "type": "text/javascript" + } + } + ] + }, + { + "name": "CMDB import: 404 OPERATION_NOT_FOUND when cancelling an unknown import", + "request": { + "method": "POST", + "header": [ + { + "key": "OCS-APIRequest", + "value": "true", + "type": "text" + } + ], + "url": { + "raw": "{{stackiq_api}}/cmdb-import/cmdb-00000000-0000-4000-8000-000000000000/cancel", + "host": [ + "{{stackiq_api}}" + ], + "path": [ + "cmdb-import", + "cmdb-00000000-0000-4000-8000-000000000000", + "cancel" + ] + }, + "auth": { + "type": "basic", + "basic": [ + { + "key": "username", + "value": "{{admin_user}}" + }, + { + "key": "password", + "value": "{{admin_pass}}" + } + ] + } + }, + "response": [], + "event": [ + { + "listen": "test", + "script": { + "exec": [ + "pm.test(\"Cancel needs a running cmdb_import operation\", function () {", + " pm.response.to.have.status(404);", + " pm.expect(pm.response.json().error).to.eql(\"OPERATION_NOT_FOUND\");", + "});" + ], + "type": "text/javascript" + } + } + ] + } + ] } ], "auth": { diff --git a/src/utils/cmdbImport.js b/src/utils/cmdbImport.js new file mode 100644 index 000000000..f53a930f1 --- /dev/null +++ b/src/utils/cmdbImport.js @@ -0,0 +1,494 @@ +// SPDX-License-Identifier: EUPL-1.2 +// SPDX-FileCopyrightText: 2026 Conduction B.V. + +/** + * Client side of the CMDB import: request building, the checks the page can + * make before uploading, and the words for every error code and outcome. + * + * The routes, field names, report shape and error codes are fixed by + * openspec/changes/cmdb-export-import/contract.md. The server stays the + * authority: every check here is repeated there. + * + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-014-the-admin-settings-shall-offer-a-cmdb-import-section + */ + +import { translate as t } from '@nextcloud/l10n' +import { generateUrl } from '@nextcloud/router' + +/** Largest upload the import accepts (contract: profile `maxFileBytes`). */ +export const MAX_FILE_BYTES = 10 * 1024 * 1024 + +/** The two sheets the import reads (contract: NO_SOURCE_SHEET details). */ +export const SOURCE_SHEETS = ['Onbeh Applicaties CMDB', 'Beheerde Applicaties CMDB'] + +/** Every row outcome the report can carry, in display order. */ +export const OUTCOMES = ['created', 'updated', 'unchanged', 'skipped', 'failed'] + +/** + * The words for one row outcome. + * + * @param {string} outcome The outcome key from the report + * @return {string} The label + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-011-each-row-shall-be-processed-in-isolation-and-reported-with-its-outcome + */ +export function outcomeLabel(outcome) { + switch (outcome) { + case 'created': + return t('stackiq', 'Created') + case 'updated': + return t('stackiq', 'Updated') + case 'unchanged': + return t('stackiq', 'Unchanged') + case 'skipped': + return t('stackiq', 'Skipped') + case 'failed': + return t('stackiq', 'Failed') + default: + return String(outcome ?? '') + } +} + +/** + * Make a fresh operation id, in the form the contract's example uses + * (`cmdb-` plus a random version 4 uuid). + * + * `crypto.randomUUID()` exists only in a secure context, and an instance + * served over plain http is not one, so the uuid is built from + * `getRandomValues()`, which is available everywhere. + * + * @return {string} The id + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-013-a-running-import-shall-report-its-progress-and-shall-stop-when-cancelled + */ +export function makeCmdbOperationId() { + const bytes = new Uint8Array(16) + globalThis.crypto.getRandomValues(bytes) + bytes[6] = (bytes[6] & 0x0f) | 0x40 + bytes[8] = (bytes[8] & 0x3f) | 0x80 + const hex = Array.from(bytes, (b) => b.toString(16).padStart(2, '0')).join('') + return ( + 'cmdb-' + + hex.slice(0, 8) + + '-' + + hex.slice(8, 12) + + '-' + + hex.slice(12, 16) + + '-' + + hex.slice(16, 20) + + '-' + + hex.slice(20) + ) +} + +/** + * The check the page makes on a chosen file before it uploads it. + * + * Only the name and size are checked here; the content check (ZIP signature, + * `xl/workbook.xml`) is the server's. + * + * @param {File|null} file The chosen file + * @return {{error: string, details: object}|null} An error in the server's shape, or null when the file may be sent + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-001-the-import-endpoint-shall-accept-only-a-bounded-xlsx-upload-from-a-nextcloud-admin + */ +export function checkFile(file) { + if (!file) { + return { error: 'NO_FILE_UPLOADED', details: {} } + } + if (!/\.xlsx$/i.test(file.name || '')) { + return { error: 'NOT_XLSX', details: {} } + } + if (file.size > MAX_FILE_BYTES) { + return { error: 'FILE_TOO_LARGE', details: {} } + } + return null +} + +/** + * The multipart body for `POST /api/cmdb-import`. + * + * @param {object} options The options + * @param {File} options.file The export + * @param {{uuid: string|null, name: string}} options.municipality The chosen municipality: an existing one has a uuid, a new one only a name + * @param {boolean} options.updateExisting Whether matched rows are updated + * @param {string} options.operationId The progress operation id + * @return {FormData} The body + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-004-every-import-shall-have-exactly-one-consuming-municipality-chosen-by-the-admin + */ +export function buildImportForm({ + file, + municipality, + updateExisting, + operationId, +}) { + const form = new FormData() + form.append('cmdbFile', file) + if (municipality?.uuid) { + form.append('municipalityUuid', municipality.uuid) + } else if (municipality?.name) { + form.append('municipalityName', municipality.name) + } + form.append('updateExisting', updateExisting ? 'true' : 'false') + form.append('missingRecords', 'keep') + form.append('operationId', operationId) + return form +} + +/** + * The URL of the import endpoint. + * + * @return {string} The URL + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-001-the-import-endpoint-shall-accept-only-a-bounded-xlsx-upload-from-a-nextcloud-admin + */ +export function importUrl() { + return generateUrl('/apps/stackiq/api/cmdb-import') +} + +/** + * Ask the server to stop a running import between two rows. + * + * @param {object} options The options + * @param {string} options.operationId The operation to cancel + * @param {object} options.http An axios-like client with post + * @return {Promise} The server's answer + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-013-a-running-import-shall-report-its-progress-and-shall-stop-when-cancelled + */ +export async function cancelCmdbImport({ operationId, http }) { + const response = await http.post( + generateUrl('/apps/stackiq/api/cmdb-import/{operationId}/cancel', { + operationId, + }), + ) + return response.data +} + +/** + * The link to a module's detail page in the app. + * + * The settings page is outside the app's router, so this is a plain URL. + * + * @param {string} uuid The module uuid + * @return {string} The URL + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-011-each-row-shall-be-processed-in-isolation-and-reported-with-its-outcome + */ +export function moduleUrl(uuid) { + return generateUrl('/apps/stackiq/modules/{id}', { id: uuid }) +} + +/** + * Turn a failed request into the server's error shape. + * + * Errors raised by Nextcloud itself (not signed in, not an admin, CSRF) come + * without a CMDB error code, so they get one here from the HTTP status. + * + * @param {object} error The axios error + * @return {{error: string, message: string, details: object, status: number}} The error + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-001-the-import-endpoint-shall-accept-only-a-bounded-xlsx-upload-from-a-nextcloud-admin + */ +export function normaliseError(error) { + const status = error?.response?.status ?? 0 + const body = error?.response?.data + const fromBody = + body && typeof body === 'object' && typeof body.error === 'string' + ? body.error + : '' + let code = fromBody + if (code === '') { + if (status === 401) { + code = 'NOT_SIGNED_IN' + } else if (status === 403) { + code = 'NOT_ADMIN' + } else if (status === 412) { + code = 'CSRF_FAILED' + } else if (status === 413) { + code = 'FILE_TOO_LARGE' + } else if (status === 0) { + code = 'NETWORK_ERROR' + } else { + code = 'IMPORT_FAILED' + } + } + return { + error: code, + message: + body && typeof body === 'object' && typeof body.message === 'string' + ? body.message + : '', + details: + body + && typeof body === 'object' + && body.details + && typeof body.details === 'object' + ? body.details + : {}, + status, + } +} + +/** Every error code the page has its own words for. */ +const KNOWN_ERRORS = new Set([ + 'NO_FILE_UPLOADED', + 'NOT_XLSX', + 'FILE_TOO_LARGE', + 'MISSING_RECORDS_UNSUPPORTED', + 'MUNICIPALITY_REQUIRED', + 'MUNICIPALITY_INVALID', + 'NO_SOURCE_SHEET', + 'MISSING_COLUMN', + 'TOO_MANY_ROWS', + 'MAPPING_UNAVAILABLE', + 'READER_UNAVAILABLE', + 'NOT_CONFIGURED', + 'OPERATION_NOT_FOUND', + 'NOT_SIGNED_IN', + 'NOT_ADMIN', + 'CSRF_FAILED', + 'NETWORK_ERROR', +]) + +/** + * Whether the page has its own words for an error code. For any other code + * (including `IMPORT_FAILED`) the page also shows the server's message. + * + * @param {string} code The error code + * @return {boolean} True for a code with its own text + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-001-the-import-endpoint-shall-accept-only-a-bounded-xlsx-upload-from-a-nextcloud-admin + */ +export function isKnownError(code) { + return KNOWN_ERRORS.has(code) +} + +/** + * What the page says for an error: a title and, where the code has one, a + * hint on what to do. The text is the page's own, so it is translated even + * when the server's message is not. + * + * @param {{error: string, message?: string, details?: object}} error The error in the server's shape + * @return {{title: string, hint: string}} The words + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-003-columns-shall-be-resolved-by-header-name-and-a-missing-required-column-shall-stop-the-import-with-422 + */ +export function errorText(error) { + const details = error?.details || {} + switch (error?.error) { + case 'NO_FILE_UPLOADED': + return { + title: t('stackiq', 'No file was uploaded.'), + hint: t('stackiq', 'Choose the TOPdesk export and try again.'), + } + case 'NOT_XLSX': + return { + title: t('stackiq', 'This file is not an Excel workbook (.xlsx).'), + hint: t( + 'stackiq', + 'Save the TOPdesk export as an Excel workbook (.xlsx). CSV, .xls and macro-enabled .xlsm files are not accepted.', + ), + } + case 'FILE_TOO_LARGE': + return { + title: t('stackiq', 'The file is larger than 10 MB.'), + hint: t( + 'stackiq', + 'Remove sheets the import does not read, or split the export, and try again.', + ), + } + case 'MISSING_RECORDS_UNSUPPORTED': + return { + title: t( + 'stackiq', + 'Records missing from the export can only be kept.', + ), + hint: '', + } + case 'MUNICIPALITY_REQUIRED': + return { + title: t('stackiq', 'Choose a municipality first.'), + hint: t( + 'stackiq', + 'Pick an existing municipality or type the name of a new one.', + ), + } + case 'MUNICIPALITY_INVALID': + return { + title: t( + 'stackiq', + 'The chosen organisation is not a municipality.', + ), + hint: t( + 'stackiq', + 'Pick an organisation of type Municipality, or type the name of a new one.', + ), + } + case 'NO_SOURCE_SHEET': { + const expected = + Array.isArray(details.expected) && details.expected.length > 0 + ? details.expected + : SOURCE_SHEETS + return { + title: t( + 'stackiq', + 'The workbook has none of the sheets the import reads.', + ), + hint: t( + 'stackiq', + 'Expected a sheet named "{first}" or "{second}". Sheet names must match exactly.', + { + first: String(expected[0] ?? SOURCE_SHEETS[0]), + second: String(expected[1] ?? SOURCE_SHEETS[1]), + }, + ), + } + } + case 'MISSING_COLUMN': + return { + title: t( + 'stackiq', + 'The sheet "{sheet}" has no column "{column}".', + { + sheet: String(details.sheet ?? ''), + column: String(details.column ?? ''), + }, + ), + hint: t( + 'stackiq', + 'The columns "APPID" and "Applicatie Naam" are required on every source sheet. Add the column to the export and try again. Nothing was imported.', + ), + } + case 'TOO_MANY_ROWS': + return { + title: details.sheet + ? t( + 'stackiq', + 'The sheet "{sheet}" has more rows than the import can process.', + { sheet: String(details.sheet) }, + ) + : t( + 'stackiq', + 'A sheet has more rows than the import can process.', + ), + hint: t( + 'stackiq', + 'A source sheet may hold at most 10,000 rows. Split the export and import the parts one after the other.', + ), + } + case 'MAPPING_UNAVAILABLE': + return { + title: t('stackiq', 'The import mapping cannot run.'), + hint: t( + 'stackiq', + "OpenRegister's mapping engine is missing or a mapping file is invalid. Update OpenRegister and check the Nextcloud log.", + ), + } + case 'READER_UNAVAILABLE': + return { + title: t('stackiq', 'The Excel reader is not available.'), + hint: t( + 'stackiq', + 'The import reads workbooks with the spreadsheet library that ships with OpenRegister. Make sure OpenRegister is installed and enabled.', + ), + } + case 'NOT_CONFIGURED': + return { + title: t('stackiq', 'Stackiq is not configured for the import.'), + hint: t( + 'stackiq', + 'The stackiq register or its schemas cannot be found. Run Auto Configure at the top of this page, then try again.', + ), + } + case 'OPERATION_NOT_FOUND': + return { + title: t('stackiq', 'This import is no longer running.'), + hint: '', + } + case 'NOT_SIGNED_IN': + return { + title: t('stackiq', 'You are not signed in.'), + hint: t('stackiq', 'Sign in again and retry the import.'), + } + case 'NOT_ADMIN': + return { + title: t( + 'stackiq', + 'Only Nextcloud administrators can import a CMDB export.', + ), + hint: '', + } + case 'CSRF_FAILED': + return { + title: t('stackiq', 'Your session has expired.'), + hint: t('stackiq', 'Reload the page and try again.'), + } + case 'NETWORK_ERROR': + return { + title: t('stackiq', 'The server could not be reached.'), + hint: t('stackiq', 'Check the connection and try again.'), + } + case 'IMPORT_FAILED': + default: + return { + title: t('stackiq', 'The import failed unexpectedly.'), + hint: t( + 'stackiq', + 'Nothing more is known on this page; the Nextcloud log has the details.', + ), + } + } +} + +/** + * What the page shows for a progress snapshot of the running import. + * + * The percentage comes from the processed and total row counts when the + * server has set them, because the tracker's own percentage is weighted by + * the phases of the ArchiMate import. + * + * @param {object|null} progress The snapshot from `GET /api/progress/{operationId}` + * @return {{percentage: number, detail: string}|null} The view, or null before any progress + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-013-a-running-import-shall-report-its-progress-and-shall-stop-when-cancelled + */ +export function cmdbProgressView(progress) { + if (!progress) { + return null + } + const processed = Number(progress.processed_items) || 0 + const total = Number(progress.total_items) || 0 + const percentage = + total > 0 + ? Math.min(100, Math.round((processed / total) * 100)) + : Number(progress.percentage) || 0 + return { + percentage, + detail: + total > 0 + ? t('stackiq', '{processed} of {total} rows processed', { + processed, + total, + }) + : '', + } +} + +/** + * The rows of the report as the table shows them. + * + * @param {Array} rows The report's `rows` + * @return {Array} The table rows + * @spec openspec/changes/cmdb-export-import/specs/cmdb-export-import/spec.md#requirement-req-cmdb-011-each-row-shall-be-processed-in-isolation-and-reported-with-its-outcome + */ +export function reportRows(rows) { + if (!Array.isArray(rows)) { + return [] + } + return rows.map((row, index) => ({ + key: `${row.sheet ?? ''}:${row.row ?? index}:${index}`, + sheet: String(row.sheet ?? ''), + row: row.row ?? '', + appId: String(row.appId ?? ''), + name: String(row.name ?? ''), + outcome: String(row.outcome ?? ''), + notes: [ + ...(Array.isArray(row.reasons) ? row.reasons : []), + ...(Array.isArray(row.warnings) ? row.warnings : []), + ] + .map((note) => String(note)) + .join('; '), + moduleUuid: row.moduleUuid ? String(row.moduleUuid) : '', + })) +} diff --git a/src/views/settings/StackiqSettings.vue b/src/views/settings/StackiqSettings.vue index f9ac89060..05d723232 100644 --- a/src/views/settings/StackiqSettings.vue +++ b/src/views/settings/StackiqSettings.vue @@ -85,6 +85,9 @@ + + + @@ -134,6 +137,7 @@ import { defineComponent } from 'vue' import Web from 'vue-material-design-icons/Web.vue' import AlwaysVisibleSection from '../../components/AlwaysVisibleSection.vue' import ArchiMateImportExport from './sections/ArchiMateImportExport.vue' +import CmdbImport from './sections/CmdbImport.vue' import CronjobConfiguration from './sections/CronjobConfiguration.vue' import EmailConfiguration from './sections/EmailConfiguration.vue' import EolSyncSettings from './sections/EolSyncSettings.vue' @@ -164,6 +168,7 @@ export default defineComponent({ UserGroupsConfiguration, OrganizationSynchronization, ArchiMateImportExport, + CmdbImport, EmailConfiguration, CronjobConfiguration, ModerationQueue, diff --git a/src/views/settings/sections/CmdbImport.vue b/src/views/settings/sections/CmdbImport.vue new file mode 100644 index 000000000..aa8718d96 --- /dev/null +++ b/src/views/settings/sections/CmdbImport.vue @@ -0,0 +1,991 @@ + + + + + + + diff --git a/tests/Unit/Controller/CmdbImportControllerTest.php b/tests/Unit/Controller/CmdbImportControllerTest.php new file mode 100644 index 000000000..665a23171 --- /dev/null +++ b/tests/Unit/Controller/CmdbImportControllerTest.php @@ -0,0 +1,341 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-8 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Controller; + +use OCA\Stackiq\Controller\CmdbImportController; +use OCA\Stackiq\Exception\CmdbImportException; +use OCA\Stackiq\Service\CmdbExportImportService; +use OCP\IL10N; +use OCP\IRequest; +use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\MockObject\MockObject; +use PHPUnit\Framework\TestCase; +use Psr\Log\LoggerInterface; +use ReflectionMethod; +use RuntimeException; + +/** + * The controller in front of CmdbExportImportService. + */ +class CmdbImportControllerTest extends TestCase { + /** + * Nextcloud's annotation regex (ControllerMethodReflector), as AdminAuthPostureTest uses it. + */ + private const ANNOTATION = '/^\h+\*\h+@(?P[A-Z]\w+)((?P.*))?$/m'; + + /** + * Temporary files of the test. + * + * @var array + */ + private array $files = []; + + /** + * Remove temporary files. + * + * @return void + */ + protected function tearDown(): void { + foreach ($this->files as $file) { + if (is_file($file) === true) { + unlink($file); + } + } + }//end tearDown() + + /** + * A temporary upload. + * + * @param string $content The file content. + * + * @return string The path. + */ + private function upload(string $content = "PK\x03\x04"): string { + $path = (string)tempnam(sys_get_temp_dir(), 'cmdb'); + file_put_contents($path, $content); + $this->files[] = $path; + return $path; + }//end upload() + + /** + * The controller with a request carrying the given file and params. + * + * @param array|null $file The uploaded file entry, or null. + * @param array $params Form fields. + * @param CmdbExportImportService|MockObject|null $service The service. + * + * @return CmdbImportController + */ + private function controller(?array $file, array $params, CmdbExportImportService|MockObject|null $service = null): CmdbImportController { + $request = $this->createMock(IRequest::class); + $request->method('getUploadedFile')->willReturnCallback(fn (string $key) => $key === 'cmdbFile' ? $file : null); + $request->method('getParam')->willReturnCallback(fn (string $key, $default = null) => ($params[$key] ?? $default)); + + $l10n = $this->createMock(IL10N::class); + $l10n->method('t')->willReturnCallback(fn (string $text, $parameters = []): string => vsprintf($text, (array)$parameters)); + + if ($service === null) { + $service = $this->createMock(CmdbExportImportService::class); + $service->method('maxFileBytes')->willReturn(10485760); + $service->method('supportsMissingRecords')->willReturnCallback(fn (string $mode): bool => $mode === 'keep'); + } + + return new CmdbImportController(request: $request, importService: $service, l10n: $l10n, logger: $this->createMock(LoggerInterface::class)); + }//end controller() + + /** + * A service double with the defaults the controller reads before import(). + * + * @return CmdbExportImportService|MockObject + */ + private function service(): CmdbExportImportService|MockObject { + $service = $this->createMock(CmdbExportImportService::class); + $service->method('maxFileBytes')->willReturn(10485760); + $service->method('supportsMissingRecords')->willReturnCallback(fn (string $mode): bool => $mode === 'keep'); + return $service; + }//end service() + + /** + * A file entry as PHP puts it in $_FILES. + * + * @param string $path The temporary file. + * @param string $name The original name. + * @param int $size The size. + * + * @return array + */ + private function file(string $path, string $name = 'export.xlsx', int $size = 4): array { + return ['tmp_name' => $path, 'name' => $name, 'size' => $size, 'error' => UPLOAD_ERR_OK]; + }//end file() + + /** + * Neither method declares NoAdminRequired or NoCSRFRequired, as attribute or annotation. + * + * @return void + */ + public function testBothRoutesAreAdminOnlyWithCsrf(): void { + foreach (['import', 'cancel'] as $method) { + $reflection = new ReflectionMethod(CmdbImportController::class, $method); + $this->assertSame([], $reflection->getAttributes(), $method); + + preg_match_all(self::ANNOTATION, (string)$reflection->getDocComment(), $matches); + foreach (['NoAdminRequired', 'NoCSRFRequired', 'PublicPage'] as $annotation) { + $this->assertNotContains($annotation, $matches['annotation'], $method); + } + } + + $routes = require __DIR__ . '/../../../appinfo/routes.php'; + $byName = array_column($routes['routes'], null, 'name'); + $this->assertSame(['name' => 'cmdbImport#import', 'url' => '/api/cmdb-import', 'verb' => 'POST'], $byName['cmdbImport#import']); + $this->assertSame(['name' => 'cmdbImport#cancel', 'url' => '/api/cmdb-import/{operationId}/cancel', 'verb' => 'POST'], $byName['cmdbImport#cancel']); + }//end testBothRoutesAreAdminOnlyWithCsrf() + + /** + * No file is 400 NO_FILE_UPLOADED; a failed upload too. + * + * @return void + */ + public function testNoFileIsRefused(): void { + $response = $this->controller(file: null, params: ['municipalityName' => 'Gemeente Voorbeeldstad'])->import(); + $this->assertSame(400, $response->getStatus()); + $this->assertSame('NO_FILE_UPLOADED', $response->getData()['error']); + $this->assertFalse($response->getData()['success']); + $this->assertNotSame('', $response->getData()['message']); + + $partial = ['tmp_name' => '', 'name' => 'export.xlsx', 'size' => 0, 'error' => UPLOAD_ERR_PARTIAL]; + $this->assertSame('NO_FILE_UPLOADED', $this->controller(file: $partial, params: [])->import()->getData()['error']); + }//end testNoFileIsRefused() + + /** + * A file of 10 MB plus one byte is 413 FILE_TOO_LARGE and the reader is never invoked. + * + * @return void + */ + public function testAnOversizedFileIsRefusedBeforeReading(): void { + $service = $this->service(); + $service->expects($this->never())->method('assertXlsx'); + $service->expects($this->never())->method('import'); + + $response = $this->controller(file: $this->file(path: $this->upload(), size: 10485761), params: ['municipalityName' => 'X'], service: $service)->import(); + $this->assertSame(413, $response->getStatus()); + $this->assertSame('FILE_TOO_LARGE', $response->getData()['error']); + + $tooBig = ['tmp_name' => '', 'name' => 'export.xlsx', 'size' => 0, 'error' => UPLOAD_ERR_INI_SIZE]; + $this->assertSame(413, $this->controller(file: $tooBig, params: [], service: $service)->import()->getStatus()); + }//end testAnOversizedFileIsRefusedBeforeReading() + + /** + * A file that is not xlsx is 400 NOT_XLSX, checked before missingRecords and the municipality. + * + * @return void + */ + public function testANonXlsxFileIsRefused(): void { + $service = $this->service(); + $service->method('assertXlsx')->willThrowException(new CmdbImportException(errorCode: CmdbImportException::NOT_XLSX, message: 'no')); + $service->expects($this->never())->method('import'); + + $response = $this->controller(file: $this->file(path: $this->upload(content: 'Applicatie Naam;APPID'), name: 'applications.csv'), params: ['missingRecords' => 'remove'], service: $service)->import(); + + $this->assertSame(400, $response->getStatus()); + $this->assertSame('NOT_XLSX', $response->getData()['error']); + }//end testANonXlsxFileIsRefused() + + /** + * A reserved missingRecords value is 422 MISSING_RECORDS_UNSUPPORTED, before the municipality check. + * + * @return void + */ + public function testAReservedMissingRecordsValueIsRefused(): void { + $service = $this->service(); + $service->expects($this->never())->method('import'); + + foreach (['remove', 'mark'] as $mode) { + $response = $this->controller(file: $this->file(path: $this->upload()), params: ['missingRecords' => $mode], service: $service)->import(); + $this->assertSame(422, $response->getStatus(), $mode); + $this->assertSame('MISSING_RECORDS_UNSUPPORTED', $response->getData()['error'], $mode); + } + }//end testAReservedMissingRecordsValueIsRefused() + + /** + * Without a municipality the answer is 422 MUNICIPALITY_REQUIRED and nothing is imported. + * + * @return void + */ + public function testAMunicipalityIsRequired(): void { + $service = $this->service(); + $service->expects($this->never())->method('import'); + + $response = $this->controller(file: $this->file(path: $this->upload()), params: ['municipalityName' => ' '], service: $service)->import(); + $this->assertSame(422, $response->getStatus()); + $this->assertSame('MUNICIPALITY_REQUIRED', $response->getData()['error']); + }//end testAMunicipalityIsRequired() + + /** + * Every service exception keeps its contract code, status and details. + * + * @return array}> + */ + public static function serviceErrors(): array { + return [ + 'mapping' => [CmdbImportException::MAPPING_UNAVAILABLE, 503, []], + 'reader' => [CmdbImportException::READER_UNAVAILABLE, 503, []], + 'config' => [CmdbImportException::NOT_CONFIGURED, 503, []], + 'no sheet' => [CmdbImportException::NO_SOURCE_SHEET, 422, ['expected' => ['Onbeh Applicaties CMDB', 'Beheerde Applicaties CMDB']]], + 'column' => [CmdbImportException::MISSING_COLUMN, 422, ['sheet' => 'Beheerde Applicaties CMDB', 'column' => 'APPID']], + 'rows' => [CmdbImportException::TOO_MANY_ROWS, 422, ['sheet' => 'Beheerde Applicaties CMDB', 'limit' => 10000]], + 'municipality' => [CmdbImportException::MUNICIPALITY_INVALID, 422, []], + 'corrupt' => [CmdbImportException::NOT_XLSX, 400, []], + ]; + }//end serviceErrors() + + /** + * A service exception becomes its contract response. + * + * @param string $code The error code. + * @param int $status The HTTP status. + * @param array $details The details. + * + * @return void + */ + #[DataProvider('serviceErrors')] + public function testServiceErrorsAreTranslated(string $code, int $status, array $details): void { + $service = $this->service(); + $service->method('import')->willThrowException(new CmdbImportException(errorCode: $code, message: 'internal', details: $details)); + + $response = $this->controller(file: $this->file(path: $this->upload()), params: ['municipalityUuid' => '00000000-0000-0000-0000-000000000001'], service: $service)->import(); + + $this->assertSame($status, $response->getStatus()); + $this->assertSame($code, $response->getData()['error']); + $this->assertEquals((object)$details, $response->getData()['details']); + $this->assertStringNotContainsString('internal', $response->getData()['message']); + if ($code === CmdbImportException::MISSING_COLUMN) { + $this->assertStringContainsString('Beheerde Applicaties CMDB', $response->getData()['message']); + $this->assertStringContainsString('APPID', $response->getData()['message']); + } + }//end testServiceErrorsAreTranslated() + + /** + * An unexpected error is 500 IMPORT_FAILED with a generic message. + * + * @return void + */ + public function testAnUnexpectedErrorIsAGeneric500(): void { + $service = $this->service(); + $service->method('import')->willThrowException(new RuntimeException('SQLSTATE secret detail')); + + $response = $this->controller(file: $this->file(path: $this->upload()), params: ['municipalityName' => 'Gemeente Voorbeeldstad'], service: $service)->import(); + + $this->assertSame(500, $response->getStatus()); + $this->assertSame('IMPORT_FAILED', $response->getData()['error']); + $this->assertStringNotContainsString('SQLSTATE', $response->getData()['message']); + }//end testAnUnexpectedErrorIsAGeneric500() + + /** + * A valid upload passes the options through and answers 200 with the report. + * + * @return void + */ + public function testAValidUploadReturnsTheReport(): void { + $path = $this->upload(); + $service = $this->service(); + $service->expects($this->once())->method('assertXlsx')->with($path, 'export.xlsx'); + $service->expects($this->once())->method('import') + ->with( + $path, + [ + 'municipalityUuid' => '', + 'municipalityName' => 'Gemeente Voorbeeldstad', + 'updateExisting' => false, + 'operationId' => 'cmdb-00000000-0000-0000-0000-000000000000', + ] + ) + ->willReturn(['success' => true, 'summary' => ['created' => 2]]); + + $response = $this->controller( + file: $this->file(path: $path), + params: ['municipalityName' => 'Gemeente Voorbeeldstad', 'updateExisting' => 'false', 'missingRecords' => 'keep', 'operationId' => 'cmdb-00000000-0000-0000-0000-000000000000'], + service: $service + )->import(); + + $this->assertSame(200, $response->getStatus()); + $this->assertSame(['success' => true, 'summary' => ['created' => 2]], $response->getData()); + }//end testAValidUploadReturnsTheReport() + + /** + * Cancel answers 200 for a running import and 404 OPERATION_NOT_FOUND otherwise. + * + * @return void + */ + public function testCancel(): void { + $service = $this->service(); + $service->method('requestCancel')->willReturnCallback(fn (string $id): bool => $id === 'cmdb-running-1'); + $controller = $this->controller(file: null, params: [], service: $service); + + $ok = $controller->cancel(operationId: 'cmdb-running-1'); + $this->assertSame(200, $ok->getStatus()); + $this->assertSame(['success' => true, 'cancelRequested' => true], $ok->getData()); + + $missing = $controller->cancel(operationId: 'cmdb-unknown-1'); + $this->assertSame(404, $missing->getStatus()); + $this->assertSame('OPERATION_NOT_FOUND', $missing->getData()['error']); + }//end testCancel() +}//end class diff --git a/tests/Unit/Fixtures/CmdbFixtureHygieneTest.php b/tests/Unit/Fixtures/CmdbFixtureHygieneTest.php new file mode 100644 index 000000000..b7d1c8988 --- /dev/null +++ b/tests/Unit/Fixtures/CmdbFixtureHygieneTest.php @@ -0,0 +1,305 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-1 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Fixtures; + +use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\TestCase; +use ZipArchive; + +/** + * Scans every fixture package. + */ +class CmdbFixtureHygieneTest extends TestCase { + /** + * The only e-mail addresses a fixture may hold. + * + * @var array + */ + private const PLACEHOLDER_EMAILS = ['letter.achternaam@gemeente.nl', 'groepsmail.test@gemeente.nl']; + + /** + * The only hosts a fixture's content may link to. + * + * @var array + */ + private const PLACEHOLDER_HOSTS = ['wiki.gemeente.nl', 'example.invalid']; + + /** + * The only runs of six or more digits (personnel numbers, phone numbers, ids) a fixture may hold. + * + * @var array + */ + private const PLACEHOLDER_NUMBERS = ['123456', '123457', '612345678', '0612345678', '143211234', '10000000001']; + + /** + * The only values a person-name column may hold, besides a placeholder e-mail address + * (TOPdesk puts the address of the configuration coordinator in that column), and + * the placeholder function a CMDB sheet shows as owner when no person is set. + * + * @var array + */ + private const PLACEHOLDER_NAMES = ['', 'Achternaam, Voornaam', 'Achternaam, voornaam', 'Teamleider Applicatiebeheer']; + + /** + * Columns that hold a person's name. + * + * @var array + */ + private const NAME_COLUMNS = [ + 'Eigenaar', + 'FB contactpersoon 1', + 'FB contactpersoon 2', + 'Groepseigenaar naam⚡', + 'Configuratie coördinator⚡', + 'Applicatie Eigenaar (Persoon)', + '|Asset eigenaar', + ]; + + /** + * Hosts of XML namespaces and schemas, which are not content. + * + * @var array + */ + private const SCHEMA_HOSTS = ['schemas.openxmlformats.org', 'schemas.microsoft.com', 'purl.org', 'www.w3.org']; + + /** + * Every fixture. + * + * @return array + */ + public static function fixtures(): array { + $cases = []; + foreach (glob(__DIR__ . '/../../fixtures/cmdb/*.xlsx') as $path) { + $cases[basename($path)] = [$path]; + } + + return $cases; + }//end fixtures() + + /** + * Every part of a package, by name. + * + * @param string $path The package. + * + * @return array + */ + private function parts(string $path): array { + $zip = new ZipArchive(); + $this->assertTrue($zip->open($path, ZipArchive::RDONLY), basename($path)); + $parts = []; + for ($index = 0; $index < $zip->numFiles; $index++) { + $name = (string)$zip->getNameIndex($index); + $parts[$name] = (string)$zip->getFromIndex($index); + } + + $zip->close(); + return $parts; + }//end parts() + + /** + * There are fixtures to scan, including the four the reader tests use. + * + * @return void + */ + public function testTheFixturesExist(): void { + $names = array_keys(self::fixtures()); + foreach (['topdesk-export-anonymised.xlsx', 'topdesk-missing-appid.xlsx', 'topdesk-shuffled-columns.xlsx', 'topdesk-formula-and-connection.xlsx'] as $expected) { + $this->assertContains($expected, $names); + } + }//end testTheFixturesExist() + + /** + * No author, no custom properties, no customXml, no absolute save path; connections only the synthetic one. + * + * @param string $path The fixture. + * + * @return void + */ + #[DataProvider('fixtures')] + public function testNoDocumentMetadata(string $path): void { + $parts = $this->parts(path: $path); + $name = basename($path); + + $this->assertArrayNotHasKey('docProps/custom.xml', $parts, $name); + foreach (array_keys($parts) as $part) { + $this->assertStringStartsNotWith('customXml/', $part, $name); + } + + if (isset($parts['docProps/core.xml']) === true) { + $core = $parts['docProps/core.xml']; + $this->assertDoesNotMatchRegularExpression('#[^<]+#', $core, $name); + $this->assertDoesNotMatchRegularExpression('#[^<]+#', $core, $name); + } + + $this->assertStringNotContainsString('absPath', ($parts['xl/workbook.xml'] ?? ''), $name); + foreach (['[Content_Types].xml', '_rels/.rels', 'xl/_rels/workbook.xml.rels'] as $index) { + $this->assertStringNotContainsString('custom.xml', ($parts[$index] ?? ''), $name . ' ' . $index); + $this->assertStringNotContainsString('customXml', ($parts[$index] ?? ''), $name . ' ' . $index); + } + + if ($name === 'topdesk-formula-and-connection.xlsx') { + $this->assertStringContainsString('https://example.invalid/', $parts['xl/connections.xml'], 'the synthetic connection'); + return; + } + + $this->assertArrayNotHasKey('xl/connections.xml', $parts, $name); + $this->assertStringNotContainsString('connections.xml', $parts['[Content_Types].xml'], $name); + $this->assertStringNotContainsString('connections.xml', ($parts['xl/_rels/workbook.xml.rels'] ?? ''), $name); + }//end testNoDocumentMetadata() + + /** + * Every e-mail address, linked host and long digit run is a known placeholder. + * + * @param string $path The fixture. + * + * @return void + */ + #[DataProvider('fixtures')] + public function testOnlyPlaceholderContactData(string $path): void { + $name = basename($path); + foreach ($this->parts(path: $path) as $part => $content) { + if (preg_match('/\.(xml|rels)$/', $part) !== 1) { + continue; + } + + preg_match_all('/[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)+/', $content, $emails); + foreach (array_unique($emails[0]) as $email) { + $this->assertContains(strtolower($email), self::PLACEHOLDER_EMAILS, $name . ' ' . $part); + } + + preg_match_all('#https?://([^/"<\s]+)#', $content, $urls); + foreach (array_unique($urls[1]) as $host) { + if (in_array($host, self::SCHEMA_HOSTS, true) === false) { + $this->assertContains($host, self::PLACEHOLDER_HOSTS, $name . ' ' . $part); + } + } + + // Digit runs in cell values and shared strings; attributes such as + // widths, ids and dates are not content. + preg_match_all('#>(\+?\d[\d\s-]{5,}\d)<#', $content, $numbers); + foreach (array_unique($numbers[1]) as $number) { + $digits = (string)preg_replace('/\D/', '', $number); + if (strlen($digits) >= 6) { + $this->assertContains($digits, self::PLACEHOLDER_NUMBERS, $name . ' ' . $part); + } + } + }//end foreach + }//end testOnlyPlaceholderContactData() + + /** + * Every person-name cell of the raw TOPdesk sheets and the CMDB sheets holds a placeholder. + * + * @param string $path The fixture. + * + * @return void + */ + #[DataProvider('fixtures')] + public function testPersonNameColumnsHoldPlaceholders(string $path): void { + $parts = $this->parts(path: $path); + $strings = $this->sharedStrings(xml: ($parts['xl/sharedStrings.xml'] ?? '')); + + foreach ($parts as $part => $content) { + if (preg_match('#^xl/worksheets/sheet\d+\.xml$#', $part) !== 1) { + continue; + } + + $sheet = simplexml_load_string($content); + $this->assertNotFalse($sheet, $part); + $headers = []; + foreach ($sheet->sheetData->row as $row) { + foreach ($row->c as $cell) { + preg_match('/^([A-Z]+)(\d+)$/', (string)$cell['r'], $ref); + $value = $this->cellText(cell: $cell, strings: $strings); + if ($ref[2] === '1') { + $headers[$ref[1]] = $value; + continue; + } + + // The raw TOPdesk sheets (Middel-ID) and the CMDB sheets (APPID) + // have their headers in row 1; the other sheets are covered by + // the e-mail and number scan. + if (in_array('Middel-ID', $headers, true) === false && in_array('APPID', $headers, true) === false) { + break 2; + } + + $header = ($headers[$ref[1]] ?? ''); + if (in_array($header, self::NAME_COLUMNS, true) === true) { + $this->assertContains(trim($value), array_merge(self::PLACEHOLDER_NAMES, self::PLACEHOLDER_EMAILS), basename($path) . ' ' . $part . ' ' . $cell['r']); + } + } + } + }//end foreach + }//end testPersonNameColumnsHoldPlaceholders() + + /** + * The shared strings table as a list. + * + * @param string $xml The sharedStrings part. + * + * @return array + */ + private function sharedStrings(string $xml): array { + if ($xml === '') { + return []; + } + + $table = simplexml_load_string($xml); + $strings = []; + foreach ($table->si as $item) { + $text = (string)$item->t; + foreach ($item->r as $run) { + $text .= (string)$run->t; + } + + $strings[] = $text; + } + + return $strings; + }//end sharedStrings() + + /** + * The text of a cell. + * + * @param \SimpleXMLElement $cell The cell. + * @param array $strings The shared strings. + * + * @return string + */ + private function cellText(\SimpleXMLElement $cell, array $strings): string { + $type = (string)$cell['t']; + if ($type === 's') { + return ($strings[(int)$cell->v] ?? ''); + } + + if ($type === 'inlineStr') { + return (string)$cell->is->t; + } + + return (string)$cell->v; + }//end cellText() +}//end class diff --git a/tests/Unit/Service/Cmdb/CmdbImportProfileTest.php b/tests/Unit/Service/Cmdb/CmdbImportProfileTest.php new file mode 100644 index 000000000..fe36e789e --- /dev/null +++ b/tests/Unit/Service/Cmdb/CmdbImportProfileTest.php @@ -0,0 +1,305 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-3 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Service\Cmdb; + +require_once __DIR__ . '/../../Support/CmdbTestSupport.php'; + +use OCA\OpenRegister\Service\MigrationPack\MappingEngine; +use OCA\OpenRegister\Service\MigrationPack\PackDefinitionValidator; +use OCA\Stackiq\Exception\CmdbImportException; +use OCA\Stackiq\Service\Cmdb\CmdbImportProfile; +use OCA\Stackiq\Tests\Unit\Support\CmdbTestSupport; +use PHPUnit\Framework\TestCase; +use Psr\Container\ContainerInterface; + +/** + * The packs are valid OpenRegister packs that implement design.md's column table. + */ +class CmdbImportProfileTest extends TestCase { + /** + * A container that knows nothing, so the validator comes from class_exists. + * + * @return ContainerInterface + */ + private function emptyContainer(): ContainerInterface { + $container = $this->createMock(ContainerInterface::class); + $container->method('has')->willReturn(false); + return $container; + }//end emptyContainer() + + /** + * A copy of the shipped profile directory, to break on purpose. + * + * @return string The directory. + */ + private function copyOfShippedDirectory(): string { + $directory = sys_get_temp_dir() . '/stackiq-cmdb-profile-' . bin2hex(random_bytes(4)); + mkdir($directory); + foreach (glob(CmdbTestSupport::appRoot() . '/lib/Settings/cmdb-import/*.json') as $file) { + copy($file, $directory . '/' . basename($file)); + } + + return $directory; + }//end copyOfShippedDirectory() + + /** + * Remove a directory made by copyOfShippedDirectory(). + * + * @param string $directory The directory. + * + * @return void + */ + private function remove(string $directory): void { + array_map('unlink', glob($directory . '/*.json')); + rmdir($directory); + }//end remove() + + /** + * Every pack passes OpenRegister's validator as an excel pack with a generated id. + * + * @return void + */ + public function testEveryPackIsAValidOpenRegisterPack(): void { + CmdbTestSupport::loadMigrationPack(); + $profile = new CmdbImportProfile(container: $this->emptyContainer()); + $profile->load(); + + $validator = new PackDefinitionValidator(); + foreach (CmdbImportProfile::TARGETS as $target) { + $pack = $profile->pack(target: $target); + $this->assertSame([], $validator->validate($pack), $target); + $this->assertSame('excel', $pack['sourceFormat'], $target); + $this->assertSame(['type' => 'generate'], $pack['idStrategy'], $target); + } + }//end testEveryPackIsAValidOpenRegisterPack() + + /** + * The packs implement the column table of design.md. + * + * @return void + */ + public function testThePacksImplementTheColumnTable(): void { + CmdbTestSupport::loadMigrationPack(); + $profile = new CmdbImportProfile(container: $this->emptyContainer()); + $profile->load(); + + $targets = function (string $pack) use ($profile): array { + $map = []; + foreach ($profile->pack(target: $pack)['fieldMappings'] as $mapping) { + $map[$mapping['source']] = $mapping['target']; + } + + return $map; + }; + + $this->assertSame( + [ + 'Applicatie Naam' => 'name', + 'APPID' => 'externalNumber', + 'Applicatie Code' => 'externalId', + 'Nickname' => 'shortDescription', + 'Roepnaam' => 'shortDescription', + 'Functionele Omschrijving' => 'longDescription', + 'Applicatiesoort' => 'cloudDienstverleningsmodel', + 'BNN Classificatie' => 'bbnLevel', + 'Datum' => 'externalCreatedAt', + 'Referentie datum wijziging' => 'externalModifiedAt', + ], + $targets('module') + ); + $this->assertSame(['Vendor' => 'name'], $targets('manufacturer')); + $this->assertSame(['municipalityName' => 'name'], $targets('municipality')); + $this->assertSame( + ['Applicatie Status' => 'status', 'Classificatie' => 'timeClassification', 'End-of-Life Functioneel' => 'startDateOutPhased', 'Beheer' => 'interneAnnotation'], + $targets('usage') + ); + $this->assertSame(['Applicatie Eigenaar (Persoon)' => 'name', 'Applicatie Eigenaar (Functie)' => 'role'], $targets('businessOwner')); + $this->assertSame(['module', 'manufacturer', 'municipality', 'usage', 'businessOwner'], CmdbImportProfile::TARGETS, 'no technical owner'); + + $this->assertSame(['type' => 'Supplier', 'status' => 'Active'], $profile->pack(target: 'manufacturer')['defaults']); + $this->assertSame(['type' => 'Municipality', 'status' => 'Active'], $profile->pack(target: 'municipality')['defaults']); + $this->assertSame(['type' => 'Application'], $profile->createOnlyDefaults(target: 'module')); + $this->assertSame(['interneAnnotation'], $profile->createOnlyFields(target: 'usage')); + $this->assertSame(['publicationDate', 'depublicationDate'], $profile->neverWrittenOnUpdate(target: 'module')); + $this->assertSame(['APPID', 'Applicatie Naam'], $profile->requiredColumns()); + $this->assertSame('APPID', $profile->keyColumn()); + $this->assertSame(['Onbeh Applicaties CMDB', 'Beheerde Applicaties CMDB'], $profile->sheetNames()); + $this->assertSame(['Beheer' => 'Beheer geregeld: nee'], $profile->sheetConstants(sheetName: 'Onbeh Applicaties CMDB')); + $this->assertSame(['Beheer' => 'Beheer geregeld: ja'], $profile->sheetConstants(sheetName: 'Beheerde Applicaties CMDB')); + $this->assertSame(['Nickname'], $profile->absentColumns(sheetName: 'Onbeh Applicaties CMDB')); + $this->assertSame([], $profile->absentColumns(sheetName: 'Beheerde Applicaties CMDB')); + $this->assertSame(['BNN Classificatie' => ['NB'], 'End-of-Life Functioneel' => ['49675']], $profile->emptyValues()); + $this->assertSame(10485760, $profile->maxFileBytes()); + $this->assertSame(10000, $profile->maxRowsPerSheet()); + }//end testThePacksImplementTheColumnTable() + + /** + * The lookups map the TOPdesk values through the real engine, and an unknown value errors instead of passing through. + * + * @return void + */ + public function testTheLookupsMapThroughTheEngine(): void { + CmdbTestSupport::loadMigrationPack(); + $profile = new CmdbImportProfile(container: $this->emptyContainer()); + $profile->load(); + $engine = new MappingEngine(); + + $usage = $engine->mapRow( + $profile->pack(target: 'usage'), + [ + 'Applicatie Status' => 'In voorraad', + 'Classificatie' => 'Tolereren', + 'End-of-Life Functioneel' => '2046-02-01', + 'Beheer' => 'Beheer geregeld: ja', + 'Cluster' => 'H10', + 'Applicatie Eigenaar (Afdeling)' => 'H10 Accounting', + ], + 2 + ); + $this->assertSame([], $usage['errors']); + $this->assertSame( + ['status' => 'Planned', 'timeClassification' => 'Tolerate', 'startDateOutPhased' => '2046-02-01', 'interneAnnotation' => 'Beheer geregeld: ja / H10 / H10 Accounting'], + $usage['data'] + ); + + $module = $engine->mapRow( + $profile->pack(target: 'module'), + ['Applicatie Naam' => 'X', 'APPID' => '1', 'BNN Classificatie' => 'BBN 2', 'Applicatiesoort' => 'Saas', 'Nickname' => 'Bijnaam', 'Roepnaam' => 'Roep'], + 2 + ); + $this->assertSame([], $module['errors']); + $this->assertSame('BBN2', $module['data']['bbnLevel']); + $this->assertSame(['SaaS'], $module['data']['cloudDienstverleningsmodel']); + $this->assertSame('Roep', $module['data']['shortDescription'], 'Roepnaam wins over Nickname'); + $nickname = $engine->mapRow($profile->pack(target: 'module'), ['Applicatie Naam' => 'X', 'APPID' => '1', 'Nickname' => 'Bijnaam', 'Roepnaam' => ''], 2); + $this->assertSame('Bijnaam', $nickname['data']['shortDescription'], 'Nickname when Roepnaam is empty'); + + $unknown = $engine->mapRow($profile->pack(target: 'usage'), ['Applicatie Status' => 'Onbekende status'], 3); + $this->assertArrayNotHasKey('status', $unknown['data']); + $this->assertSame('Applicatie Status', $unknown['errors'][0]['source']); + $this->assertStringContainsString('Onbekende status', $unknown['errors'][0]['message']); + + $soort = $engine->mapRow($profile->pack(target: 'module'), ['Applicatie Naam' => 'X', 'APPID' => '1', 'Applicatiesoort' => 'Webapplicatie'], 3); + $this->assertArrayNotHasKey('cloudDienstverleningsmodel', $soort['data'], 'an application kind is not a hosting model'); + $this->assertSame('Applicatiesoort', $soort['errors'][0]['source']); + }//end testTheLookupsMapThroughTheEngine() + + /** + * The read allowlist holds the owner columns but no other person or group column, nor + * the unmapped columns of the CMDB sheets. + * + * @return void + */ + public function testPersonColumnsAreNeverReferenced(): void { + CmdbTestSupport::loadMigrationPack(); + $profile = new CmdbImportProfile(container: $this->emptyContainer()); + $columns = $profile->referencedColumns(); + + foreach ([ + 'Personeelsnummer', + 'Eigenaar', + 'Eigenaar e-mail', + 'FB contactpersoon 1', + 'FB contactpersoon 2', + 'Groepseigenaar mail⚡', + 'Behandelgroep', + 'Hostingpartij', + 'Leverancier', + 'Beschikbaarheid', + 'Rappelreden', + 'Opmerkingen', + 'municipalityName', + 'Beheer', + ] as $never) { + $this->assertNotContains($never, $columns); + } + + $this->assertContains('Applicatie Eigenaar (Persoon)', $columns); + $this->assertContains('Applicatie Eigenaar (Functie)', $columns); + $this->assertContains('Applicatie Eigenaar (Afdeling)', $columns, 'the concat field is read too'); + $this->assertContains('Cluster', $columns); + }//end testPersonColumnsAreNeverReferenced() + + /** + * A pack with an unknown transform stops the import with MAPPING_UNAVAILABLE (503). + * + * @return void + */ + public function testAnInvalidPackIsMappingUnavailable(): void { + CmdbTestSupport::loadMigrationPack(); + $directory = $this->copyOfShippedDirectory(); + $pack = json_decode((string)file_get_contents($directory . '/topdesk-usage.json'), true); + $pack['fieldMappings'][0]['transform'] = ['type' => 'uppercase']; + file_put_contents($directory . '/topdesk-usage.json', json_encode($pack)); + + try { + (new CmdbImportProfile(container: $this->emptyContainer(), directory: $directory))->load(); + $this->fail('MAPPING_UNAVAILABLE expected'); + } catch (CmdbImportException $e) { + $this->assertSame('MAPPING_UNAVAILABLE', $e->getErrorCode()); + $this->assertSame(503, $e->getHttpStatus()); + $this->assertStringContainsString('topdesk-usage.json', $e->getMessage()); + } finally { + $this->remove(directory: $directory); + } + }//end testAnInvalidPackIsMappingUnavailable() + + /** + * A missing pack file, or a validator the container cannot give and that does not exist, is MAPPING_UNAVAILABLE. + * + * @return void + */ + public function testAMissingPackOrValidatorIsMappingUnavailable(): void { + CmdbTestSupport::loadMigrationPack(); + $directory = $this->copyOfShippedDirectory(); + unlink($directory . '/topdesk-module.json'); + + try { + (new CmdbImportProfile(container: $this->emptyContainer(), directory: $directory))->load(); + $this->fail('MAPPING_UNAVAILABLE expected'); + } catch (CmdbImportException $e) { + $this->assertSame('MAPPING_UNAVAILABLE', $e->getErrorCode()); + } finally { + $this->remove(directory: $directory); + } + + $profile = new class(container: $this->emptyContainer()) extends CmdbImportProfile { + public const VALIDATOR_CLASS = 'OCA\OpenRegister\Service\MigrationPack\NoSuchValidator'; + }; + try { + $profile->load(); + $this->fail('MAPPING_UNAVAILABLE expected without a validator'); + } catch (CmdbImportException $e) { + $this->assertSame('MAPPING_UNAVAILABLE', $e->getErrorCode()); + $this->assertSame(503, $e->getHttpStatus()); + } + }//end testAMissingPackOrValidatorIsMappingUnavailable() + + /** + * The upload limit is readable without OpenRegister. + * + * @return void + */ + public function testTheUploadLimitNeedsNoOpenRegister(): void { + $profile = new CmdbImportProfile(container: $this->emptyContainer(), directory: '/nonexistent'); + $this->assertSame(CmdbImportProfile::DEFAULT_MAX_FILE_BYTES, $profile->maxFileBytes()); + }//end testTheUploadLimitNeedsNoOpenRegister() +}//end class diff --git a/tests/Unit/Service/Cmdb/CmdbRowNormaliserTest.php b/tests/Unit/Service/Cmdb/CmdbRowNormaliserTest.php new file mode 100644 index 000000000..390ffc4ee --- /dev/null +++ b/tests/Unit/Service/Cmdb/CmdbRowNormaliserTest.php @@ -0,0 +1,134 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Service\Cmdb; + +use OCA\Stackiq\Service\Cmdb\CmdbRowNormaliser; +use PHPUnit\Framework\TestCase; + +/** + * Excel serial dates, ids and trimming. + */ +class CmdbRowNormaliserTest extends TestCase { + /** + * The serials of the anonymised export become the dates design.md names; ids lose their decimal part. + * + * @return void + */ + public function testSerialDatesAndIdsAreNormalised(): void { + $row = (new CmdbRowNormaliser())->normalise( + cells: [ + 'Datum' => 45111.380322627316, + 'Referentie datum wijziging' => 46232.552113113423, + 'End-of-Life Functioneel' => 53359, + 'APPID' => 1234.0, + 'Applicatie Code' => ' APP-test123 ', + 'Applicatie Naam' => ' naamtest123 ', + 'Vendor' => null, + ], + dateColumns: ['Datum', 'Referentie datum wijziging', 'End-of-Life Functioneel'], + idColumns: ['APPID'] + ); + + $this->assertSame( + [ + 'Datum' => '2023-07-04', + 'Referentie datum wijziging' => '2026-07-29', + 'End-of-Life Functioneel' => '2046-02-01', + 'APPID' => '1234', + 'Applicatie Code' => 'APP-test123', + 'Applicatie Naam' => 'naamtest123', + 'Vendor' => '', + ], + $row + ); + }//end testSerialDatesAndIdsAreNormalised() + + /** + * A value the profile lists as empty for its column becomes '', case-insensitively and before + * the date conversion; the same value in another column stays. + * + * @return void + */ + public function testEmptyValuesBecomeEmpty(): void { + $row = (new CmdbRowNormaliser())->normalise( + cells: ['BNN Classificatie' => 'nb', 'End-of-Life Functioneel' => 49675, 'Roepnaam' => 'NB', 'Datum' => 49675], + dateColumns: ['End-of-Life Functioneel', 'Datum'], + idColumns: [], + emptyValues: ['BNN Classificatie' => ['NB'], 'End-of-Life Functioneel' => ['49675']] + ); + + $this->assertSame(['BNN Classificatie' => '', 'End-of-Life Functioneel' => '', 'Roepnaam' => 'NB', 'Datum' => '2036-01-01'], $row); + }//end testEmptyValuesBecomeEmpty() + + /** + * The string forms of serials and ids convert the same way. + * + * @return void + */ + public function testStringSerialsAndIdsConvertToo(): void { + $normaliser = new CmdbRowNormaliser(); + + $this->assertSame('2023-07-04', $normaliser->normaliseDate(value: '45111.380322627316')); + $this->assertSame('1234', $normaliser->normaliseId(value: '1234.0')); + $this->assertSame('1234', $normaliser->normaliseId(value: 1234)); + $this->assertSame('12.5', $normaliser->normaliseId(value: 12.5)); + $this->assertSame('APP-1.0', $normaliser->normaliseId(value: 'APP-1.0')); + }//end testStringSerialsAndIdsConvertToo() + + /** + * A non-numeric date stays as it is, for the pack's date transform to judge; out-of-range serials too. + * + * @return void + */ + public function testNonSerialDatesStayAsTheyAre(): void { + $normaliser = new CmdbRowNormaliser(); + + $this->assertSame('2026-10-01', $normaliser->normaliseDate(value: ' 2026-10-01 ')); + $this->assertSame('onbekend', $normaliser->normaliseDate(value: 'onbekend')); + $this->assertSame('0', $normaliser->normaliseDate(value: 0)); + $this->assertSame('', $normaliser->normaliseDate(value: null)); + }//end testNonSerialDatesStayAsTheyAre() + + /** + * Excel's 1900 leap-year bug and the 1904 date system. + * + * @return void + */ + public function testDateSystemsAndTheLeapYearBug(): void { + $normaliser = new CmdbRowNormaliser(); + + $this->assertSame('1900-01-01', $normaliser->normaliseDate(value: 1)); + $this->assertSame('1900-02-28', $normaliser->normaliseDate(value: 59)); + $this->assertSame('1900-03-01', $normaliser->normaliseDate(value: 61)); + $this->assertSame('2023-07-04', $normaliser->normaliseDate(value: 43649, date1904: true)); + }//end testDateSystemsAndTheLeapYearBug() + + /** + * Booleans become TRUE/FALSE text; whole floats lose their decimal part. + * + * @return void + */ + public function testOtherScalarsBecomeText(): void { + $row = (new CmdbRowNormaliser())->normalise(cells: ['a' => true, 'b' => false, 'c' => 2.0, 'd' => 2.25], dateColumns: [], idColumns: []); + + $this->assertSame(['a' => 'TRUE', 'b' => 'FALSE', 'c' => '2', 'd' => '2.25'], $row); + }//end testOtherScalarsBecomeText() +}//end class diff --git a/tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php b/tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php new file mode 100644 index 000000000..56d13386f --- /dev/null +++ b/tests/Unit/Service/Cmdb/CmdbWorkbookReaderTest.php @@ -0,0 +1,323 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Service\Cmdb; + +require_once __DIR__ . '/../../Support/CmdbTestSupport.php'; + +use OCA\Stackiq\Exception\CmdbImportException; +use OCA\Stackiq\Service\Cmdb\CmdbImportProfile; +use OCA\Stackiq\Service\Cmdb\CmdbWorkbookReader; +use OCA\Stackiq\Tests\Unit\Support\CmdbTestSupport; +use PHPUnit\Framework\TestCase; +use Psr\Container\ContainerInterface; + +/** + * Reads the fixtures through PhpSpreadsheet as OpenRegister ships it. + */ +class CmdbWorkbookReaderTest extends TestCase { + /** + * The shipped profile, validated with OpenRegister's validator. + * + * @param string|null $directory A profile directory other than the shipped one. + * + * @return CmdbImportProfile + */ + private function profile(?string $directory = null): CmdbImportProfile { + CmdbTestSupport::loadMigrationPack(); + $container = $this->createMock(ContainerInterface::class); + $container->method('has')->willReturn(false); + + $profile = new CmdbImportProfile(container: $container, directory: $directory); + $profile->load(); + return $profile; + }//end profile() + + /** + * Skip unless PhpSpreadsheet can be loaded from an OpenRegister checkout. + * + * @return void + */ + private function requireSpreadsheet(): void { + if (CmdbTestSupport::loadPhpSpreadsheet() === false) { + $this->markTestSkipped('PhpSpreadsheet not found: set OPENREGISTER_DIR to an OpenRegister app with its vendor/ installed.'); + } + }//end requireSpreadsheet() + + /** + * Read a fixture. + * + * @param string $name The fixture file. + * + * @return array + */ + private function read(string $name): array { + $this->requireSpreadsheet(); + $path = CmdbTestSupport::fixtures() . '/' . $name; + $reader = new CmdbWorkbookReader(); + $reader->assertXlsx(path: $path, fileName: $name); + return $reader->read(path: $path, profile: $this->profile()); + }//end read() + + /** + * One data row per CMDB sheet, read from the cached formula values; the formatted + * empty rows and the rows whose formulas cached 0 are dropped; only allowlisted columns. + * + * @return void + */ + public function testTheSanitisedExportYieldsOneRowPerSheet(): void { + $result = $this->read(name: 'topdesk-export-anonymised.xlsx'); + $rows = $result['rows']; + + $this->assertCount(2, $rows); + $this->assertSame(['Onbeh Applicaties CMDB', 2], [$rows[0]['sheet'], $rows[0]['row']]); + $this->assertSame(['Beheerde Applicaties CMDB', 2], [$rows[1]['sheet'], $rows[1]['row']]); + $this->assertSame(1234, (int)$rows[0]['cells']['APPID']); + $this->assertSame('AIA-AangetekendMailen', $rows[0]['cells']['Applicatie Code']); + $this->assertSame('Aangetekend Mailen', $rows[0]['cells']['Applicatie Naam']); + $this->assertSame('Mailen', $rows[0]['cells']['Roepnaam']); + $this->assertSame('Webapplicatie', $rows[0]['cells']['Applicatiesoort']); + $this->assertSame('Achternaam, Voornaam', $rows[0]['cells']['Applicatie Eigenaar (Persoon)']); + $this->assertArrayNotHasKey('Nickname', $rows[0]['cells'], 'Onbeh has no Nickname column'); + $this->assertSame('naamtest123', $rows[1]['cells']['Applicatie Naam']); + $this->assertSame(2, (int)$rows[1]['cells']['APPID']); + $this->assertSame(53359, (int)$rows[1]['cells']['End-of-Life Functioneel']); + $this->assertSame('Saas', $rows[1]['cells']['Applicatiesoort']); + $this->assertSame('BBN2', $rows[1]['cells']['BNN Classificatie']); + $this->assertSame('Tolereren', $rows[1]['cells']['Classificatie']); + $this->assertSame('NT123', $rows[1]['cells']['Nickname']); + $this->assertSame('Teamleider Applicatiebeheer', $rows[1]['cells']['Applicatie Eigenaar (Persoon)']); + $this->assertSame([], $rows[0]['uncached']); + $this->assertSame([], $rows[1]['uncached']); + + $allowed = $this->profile()->referencedColumns(); + foreach ($rows as $row) { + foreach (array_keys($row['cells']) as $column) { + $this->assertContains($column, $allowed); + } + + foreach (['Beschikbaarheid', 'Behandelgroep', 'Hostingpartij', 'Rappelreden', 'Locatie BIOToets', 'Beheer'] as $never) { + $this->assertArrayNotHasKey($never, $row['cells']); + } + } + + $this->assertFalse($result['date1904']); + $this->assertSame([], $result['importWarnings'], 'Nickname is listed as absent on Onbeh, so its absence is no warning'); + }//end testTheSanitisedExportYieldsOneRowPerSheet() + + /** + * A formula cell yields the value Excel cached, not its result; a formula without + * a cached value yields an empty cell and is listed; the connection is never contacted. + * + * @return void + */ + public function testAFormulaYieldsItsCachedValue(): void { + $rows = $this->read(name: 'topdesk-formula-and-connection.xlsx')['rows']; + + // The formula evaluates to "Evaluated"; the cached value is "Rekenmodel". + $this->assertSame('Rekenmodel', $rows[1]['cells']['Applicatie Naam']); + $this->assertSame('APP-test123', $rows[1]['cells']['Applicatie Code']); + $this->assertNull($rows[1]['cells']['Roepnaam'], 'no cached value: empty, never evaluated'); + $this->assertSame(['Roepnaam'], $rows[1]['uncached']); + $this->assertSame([], $rows[0]['uncached']); + }//end testAFormulaYieldsItsCachedValue() + + /** + * The reader source never calls the calculation engine nor an HTTP client. + * + * @return void + */ + public function testTheReaderNeverEvaluatesOrFetches(): void { + $source = (string)file_get_contents(CmdbTestSupport::appRoot() . '/lib/Service/Cmdb/CmdbWorkbookReader.php'); + $code = (string)preg_replace('#/\*.*?\*/|//[^\n]*#s', '', $source); + + $this->assertStringNotContainsString('getCalculatedValue', $code); + $this->assertStringNotContainsString('toArray', $code); + $this->assertStringNotContainsString('Calculation', $code); + $this->assertDoesNotMatchRegularExpression('/Http|Guzzle|curl_|file_get_contents\(\s*\$url/i', $code); + $this->assertStringContainsString('getOldCalculatedValue', $code); + $this->assertStringContainsString('setReadDataOnly(true)', $code); + }//end testTheReaderNeverEvaluatesOrFetches() + + /** + * Shuffled columns and decorated headers map to the same rows. + * + * @return void + */ + public function testShuffledColumnsMapTheSame(): void { + $original = $this->read(name: 'topdesk-export-anonymised.xlsx')['rows']; + $shuffled = $this->read(name: 'topdesk-shuffled-columns.xlsx')['rows']; + + $this->assertCount(count($original), $shuffled); + foreach ($original as $index => $row) { + $expected = $row['cells']; + $actual = $shuffled[$index]['cells']; + ksort($expected); + ksort($actual); + $this->assertSame($expected, $actual); + } + }//end testShuffledColumnsMapTheSame() + + /** + * A CMDB sheet without APPID stops the import, naming column and sheet. + * + * @return void + */ + public function testAMissingRequiredColumnIsNamed(): void { + try { + $this->read(name: 'topdesk-missing-appid.xlsx'); + $this->fail('MISSING_COLUMN expected'); + } catch (CmdbImportException $e) { + $this->assertSame('MISSING_COLUMN', $e->getErrorCode()); + $this->assertSame(422, $e->getHttpStatus()); + $this->assertSame(['sheet' => 'Beheerde Applicaties CMDB', 'column' => 'APPID'], $e->getDetails()); + } + }//end testAMissingRequiredColumnIsNamed() + + /** + * A workbook with only "Blad1" names both expected sheets. + * + * @return void + */ + public function testAWorkbookWithoutSourceSheetsIsRefused(): void { + try { + $this->read(name: 'topdesk-no-source-sheet.xlsx'); + $this->fail('NO_SOURCE_SHEET expected'); + } catch (CmdbImportException $e) { + $this->assertSame('NO_SOURCE_SHEET', $e->getErrorCode()); + $this->assertSame(['expected' => ['Onbeh Applicaties CMDB', 'Beheerde Applicaties CMDB']], $e->getDetails()); + } + }//end testAWorkbookWithoutSourceSheetsIsRefused() + + /** + * More non-empty rows than the profile allows stops the import. + * + * @return void + */ + public function testTooManyRowsIsRefused(): void { + $this->requireSpreadsheet(); + $directory = sys_get_temp_dir() . '/stackiq-cmdb-profile-' . bin2hex(random_bytes(4)); + mkdir($directory); + $shipped = CmdbTestSupport::appRoot() . '/lib/Settings/cmdb-import'; + foreach (glob($shipped . '/*.json') as $file) { + copy($file, $directory . '/' . basename($file)); + } + + $profile = json_decode((string)file_get_contents($directory . '/topdesk-profile.json'), true); + $profile['maxRowsPerSheet'] = 0; + file_put_contents($directory . '/topdesk-profile.json', json_encode($profile)); + + try { + (new CmdbWorkbookReader())->read(path: CmdbTestSupport::fixtures() . '/topdesk-export-anonymised.xlsx', profile: $this->profile(directory: $directory)); + $this->fail('TOO_MANY_ROWS expected'); + } catch (CmdbImportException $e) { + $this->assertSame('TOO_MANY_ROWS', $e->getErrorCode()); + $this->assertSame(422, $e->getHttpStatus()); + } finally { + array_map('unlink', glob($directory . '/*.json')); + rmdir($directory); + } + }//end testTooManyRowsIsRefused() + + /** + * A text file named .xlsx, a .xlsm and a CSV are refused before PhpSpreadsheet is touched. + * + * @return void + */ + public function testNonXlsxIsRefusedBeforeParsing(): void { + $reader = new CmdbWorkbookReader(); + $text = tempnam(sys_get_temp_dir(), 'cmdb'); + file_put_contents($text, "Applicatie Naam;APPID\nVoorbeeld;1\n"); + $cases = [ + [$text, 'export.xlsx'], + [CmdbTestSupport::fixtures() . '/topdesk-export-anonymised.xlsx', 'export.xlsm'], + [$text, 'applications.csv'], + [CmdbTestSupport::fixtures() . '/topdesk-export-anonymised.xlsx', 'export.xls'], + ]; + + try { + foreach ($cases as [$path, $name]) { + try { + $reader->assertXlsx(path: $path, fileName: $name); + $this->fail('NOT_XLSX expected for ' . $name); + } catch (CmdbImportException $e) { + $this->assertSame('NOT_XLSX', $e->getErrorCode(), $name); + $this->assertSame(400, $e->getHttpStatus(), $name); + } + } + + // A ZIP without xl/workbook.xml. + $zipPath = tempnam(sys_get_temp_dir(), 'cmdb') . '.xlsx'; + $zip = new \ZipArchive(); + $zip->open($zipPath, \ZipArchive::CREATE); + $zip->addFromString('word/document.xml', ''); + $zip->close(); + try { + $reader->assertXlsx(path: $zipPath, fileName: 'export.xlsx'); + $this->fail('NOT_XLSX expected for a zip without a workbook'); + } catch (CmdbImportException $e) { + $this->assertSame('NOT_XLSX', $e->getErrorCode()); + } finally { + unlink($zipPath); + } + } finally { + unlink($text); + } + + $this->assertTrue(true); + }//end testNonXlsxIsRefusedBeforeParsing() + + /** + * Without PhpSpreadsheet the reader answers READER_UNAVAILABLE. + * + * @return void + */ + public function testAMissingReaderIsReported(): void { + $reader = new class extends CmdbWorkbookReader { + /** + * PhpSpreadsheet is absent. + * + * @return bool + */ + public function isAvailable(): bool { + return false; + }//end isAvailable() + }; + + try { + $reader->read(path: CmdbTestSupport::fixtures() . '/topdesk-export-anonymised.xlsx', profile: $this->profile()); + $this->fail('READER_UNAVAILABLE expected'); + } catch (CmdbImportException $e) { + $this->assertSame('READER_UNAVAILABLE', $e->getErrorCode()); + $this->assertSame(503, $e->getHttpStatus()); + } + }//end testAMissingReaderIsReported() + + /** + * Headers match after trimming, collapsing whitespace, dropping a trailing ":" or "⚡" and lower-casing. + * + * @return void + */ + public function testHeadersAreNormalised(): void { + $this->assertSame('vendor', CmdbWorkbookReader::normaliseHeader(header: 'Vendor⚡')); + $this->assertSame('applicatie eigenaar (persoon)', CmdbWorkbookReader::normaliseHeader(header: ' Applicatie Eigenaar (Persoon): ')); + $this->assertSame('appid', CmdbWorkbookReader::normaliseHeader(header: 'APPID')); + }//end testHeadersAreNormalised() +}//end class diff --git a/tests/Unit/Service/CmdbExportImportServiceTest.php b/tests/Unit/Service/CmdbExportImportServiceTest.php new file mode 100644 index 000000000..cde666c23 --- /dev/null +++ b/tests/Unit/Service/CmdbExportImportServiceTest.php @@ -0,0 +1,1209 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-5 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Service; + +require_once __DIR__ . '/../Support/CmdbTestSupport.php'; + +use OCA\OpenRegister\Contract\ObjectEntityInterface; +use OCA\OpenRegister\Contract\ObjectServiceInterface; +use OCA\Stackiq\Exception\CmdbImportException; +use OCA\Stackiq\Service\Cmdb\CmdbImportProfile; +use OCA\Stackiq\Service\Cmdb\CmdbRowNormaliser; +use OCA\Stackiq\Service\Cmdb\CmdbWorkbookReader; +use OCA\Stackiq\Service\CmdbExportImportService; +use OCA\Stackiq\Service\ProgressTracker; +use OCA\Stackiq\Service\SettingsService; +use OCA\Stackiq\Service\StackiqContactSyncService; +use OCA\Stackiq\Tests\Unit\Support\CmdbTestSupport; +use OCP\ICache; +use OCP\ICacheFactory; +use OCP\IL10N; +use OCP\IUserSession; +use PHPUnit\Framework\TestCase; +use Psr\Container\ContainerInterface; +use Psr\Log\AbstractLogger; +use RuntimeException; + +/** + * The import, row by row, against an in-memory OpenRegister. + * + * @SuppressWarnings(PHPMD.ExcessiveClassLength) + * @SuppressWarnings(PHPMD.TooManyPublicMethods) + */ +class CmdbExportImportServiceTest extends TestCase { + private const REGISTER = 20; + private const MODULE = 43; + private const ORGANIZATION = 33; + private const USAGE = 34; + private const CONTACT_PERSON = 32; + + /** + * Objects per schema id, by uuid. + * + * @var array>> + */ + private array $store = []; + + /** + * Every saveObject() call: schema, uuid, data, create. + * + * @var array, create: bool}> + */ + private array $saves = []; + + /** + * Called before every save; may throw. + * + * @var callable|null + */ + private $beforeSave = null; + + /** + * Contacts in the fake address book: uid => name, email. + * + * @var array + */ + private array $contacts = []; + + /** + * Whether Contacts is enabled. + * + * @var bool + */ + private bool $contactsEnabled = true; + + /** + * The in-memory distributed cache behind the ProgressTracker. + * + * @var array + */ + private array $cache = []; + + /** + * Every log line, message plus encoded context. + * + * @var array + */ + private array $logLines = []; + + /** + * The ProgressTracker of the current service. + * + * @var ProgressTracker|null + */ + private ?ProgressTracker $tracker = null; + + /** + * Reset the doubles. + * + * @return void + */ + protected function setUp(): void { + CmdbTestSupport::loadMigrationPack(); + $this->store = [self::MODULE => [], self::ORGANIZATION => [], self::USAGE => [], self::CONTACT_PERSON => []]; + $this->saves = []; + $this->beforeSave = null; + $this->contacts = []; + $this->contactsEnabled = true; + $this->cache = []; + $this->logLines = []; + }//end setUp() + + // ------------------------------------------------------------------ + // Doubles + // ------------------------------------------------------------------ + + /** + * An entity as OpenRegister returns it. + * + * @param string $uuid The uuid. + * @param array $data The object data. + * + * @return ObjectEntityInterface + */ + private function entity(string $uuid, array $data): ObjectEntityInterface { + return new class($uuid, $data) implements ObjectEntityInterface { + /** + * Constructor. + * + * @param string $uuid The uuid. + * @param array $data The data. + */ + public function __construct( + private string $uuid, + private array $data, + ) { + } + + public function getUuid(): ?string { + return $this->uuid; + } + + public function getObject(): array { + return $this->data; + } + + public function getRegister(): ?string { + return '20'; + } + + public function getSchema(): ?string { + return null; + } + + public function getOrganisation(): ?string { + return null; + } + + public function getOwner(): ?string { + return null; + } + + public function jsonSerialize(): array { + return $this->data; + } + }; + }//end entity() + + /** + * The in-memory OpenRegister. + * + * @return ObjectServiceInterface + */ + private function objectService(): ObjectServiceInterface { + $service = $this->createMock(ObjectServiceInterface::class); + $service->method('saveObject')->willReturnCallback( + function (array $object, ?array $extend = [], $register = null, $schema = null, ?string $uuid = null): ObjectEntityInterface { + $schema = (int)$schema; + if ($this->beforeSave !== null) { + ($this->beforeSave)($schema, $object); + } + + $create = ($uuid === null); + if ($create === true) { + $uuid = sprintf('00000000-0000-4000-8000-%012d', count($this->saves) + 1); + } + + $object['id'] = $uuid; + $this->store[$schema][$uuid] = $object; + $this->saves[] = ['schema' => $schema, 'uuid' => $uuid, 'data' => $object, 'create' => $create]; + return $this->entity(uuid: $uuid, data: $object); + } + ); + $service->method('searchObjects')->willReturnCallback( + function (array $query = []): array { + $schema = (int)($query['@self']['schema'] ?? 0); + $limit = (int)($query['_limit'] ?? 30); + $offset = (int)($query['_offset'] ?? 0); + $filters = array_filter($query, fn ($key): bool => $key !== '@self' && str_starts_with((string)$key, '_') === false, ARRAY_FILTER_USE_KEY); + $found = []; + foreach (($this->store[$schema] ?? []) as $uuid => $data) { + foreach ($filters as $field => $value) { + if ((string)($data[$field] ?? '') !== (string)$value) { + continue 2; + } + } + + $found[] = $this->entity(uuid: $uuid, data: $data); + } + + return array_slice($found, $offset, $limit); + } + ); + $service->method('find')->willReturnCallback( + function ($id, ?array $_extend = [], bool $files = false, $register = null, $schema = null): ?ObjectEntityInterface { + $data = ($this->store[(int)$schema][(string)$id] ?? null); + if ($data === null) { + return null; + } + + return $this->entity(uuid: (string)$id, data: $data); + } + ); + + return $service; + }//end objectService() + + /** + * The Contacts bridge over a fake address book. + * + * @return StackiqContactSyncService + */ + private function contactSync(): StackiqContactSyncService { + $sync = $this->createMock(StackiqContactSyncService::class); + $sync->method('isAvailable')->willReturnCallback(fn (): bool => $this->contactsEnabled); + $sync->method('searchContacts')->willReturnCallback( + function (string $query): array { + $found = []; + foreach ($this->contacts as $uid => $contact) { + if (str_contains(mb_strtolower($contact['name']), mb_strtolower($query)) === true) { + $found[] = ['uid' => $uid, 'name' => $contact['name'], 'email' => $contact['email']]; + } + } + + return $found; + } + ); + $sync->method('syncToContacts')->willReturnCallback( + function (string $objectType, array $record): ?string { + $email = (string)($record['email'] ?? ''); + foreach ($this->contacts as $uid => $contact) { + if ($email !== '' && strcasecmp($contact['email'], $email) === 0) { + return $uid; + } + } + + $uid = 'contact-' . (count($this->contacts) + 1); + $this->contacts[$uid] = ['name' => trim(($record['voornaam'] ?? '') . ' ' . ($record['achternaam'] ?? '')), 'email' => $email]; + return $uid; + } + ); + + return $sync; + }//end contactSync() + + /** + * A ProgressTracker on an in-memory distributed cache. + * + * @return ProgressTracker + */ + private function progressTracker(): ProgressTracker { + $cache = $this->createMock(ICache::class); + $cache->method('get')->willReturnCallback(fn ($key) => ($this->cache[$key] ?? null)); + $cache->method('set')->willReturnCallback( + function ($key, $value): bool { + $this->cache[$key] = $value; + return true; + } + ); + $cache->method('remove')->willReturnCallback( + function ($key): bool { + unset($this->cache[$key]); + return true; + } + ); + $factory = $this->createMock(ICacheFactory::class); + $factory->method('createDistributed')->willReturn($cache); + + return new ProgressTracker(cacheFactory: $factory, userSession: $this->createMock(IUserSession::class), logger: $this->logger()); + }//end progressTracker() + + /** + * An IL10N that returns the English source with its parameters filled in. + * + * @return IL10N + */ + private function l10n(): IL10N { + $l10n = $this->createMock(IL10N::class); + $l10n->method('t')->willReturnCallback(fn (string $text, $parameters = []): string => vsprintf($text, (array)$parameters)); + return $l10n; + }//end l10n() + + /** + * A logger that keeps every line. + * + * @return AbstractLogger + */ + private function logger(): AbstractLogger { + $lines = &$this->logLines; + return new class($lines) extends AbstractLogger { + /** + * Constructor. + * + * @param array $lines The collected lines. + */ + public function __construct( + private array &$lines, + ) { + } + + /** + * Keep a line. + * + * @param mixed $level The level. + * @param string|\Stringable $message The message. + * @param array $context The context. + * + * @return void + */ + public function log($level, string|\Stringable $message, array $context = []): void { + array_walk_recursive( + $context, + function (&$value): void { + if (is_object($value) === true) { + $value = get_class($value) . ($value instanceof \Throwable ? ': ' . $value->getMessage() : ''); + } + } + ); + $this->lines[] = $message . ' ' . json_encode($context, JSON_UNESCAPED_UNICODE); + } + }; + }//end logger() + + /** + * A reader that hands out given rows, for tests that do not need the fixture. + * + * @param array}> $rows The rows. + * + * @return CmdbWorkbookReader + */ + private function rowsReader(array $rows): CmdbWorkbookReader { + return new class($rows) extends CmdbWorkbookReader { + /** + * Constructor. + * + * @param array> $rows The rows. + */ + public function __construct( + private array $rows, + ) { + } + + /** + * The given rows. + * + * @param string $path Ignored. + * @param CmdbImportProfile $profile Ignored. + * + * @return array + */ + public function read(string $path, CmdbImportProfile $profile): array { + return ['rows' => $this->rows, 'importWarnings' => [], 'date1904' => false]; + } + }; + }//end rowsReader() + + /** + * The service under test. + * + * @param CmdbWorkbookReader|null $reader The reader; null is the real one. + * @param string|null $profileDir A profile directory other than the shipped one. + * @param array $config The voorzieningen config. + * + * @return CmdbExportImportService + */ + private function service(?CmdbWorkbookReader $reader = null, ?string $profileDir = null, array $config = ['register' => '20']): CmdbExportImportService { + $objectService = $this->objectService(); + $container = $this->createMock(ContainerInterface::class); + $container->method('has')->willReturn(false); + $container->method('get')->willReturnCallback( + function (string $id) use ($objectService) { + if ($id === ObjectServiceInterface::class) { + return $objectService; + } + + throw new RuntimeException('not in this container: ' . $id); + } + ); + + $settings = $this->createMock(SettingsService::class); + $settings->method('getVoorzieningenConfig')->willReturn($config); + $settings->method('getSchemaIdForObjectType')->willReturnCallback( + fn (string $type): ?int => ['module' => self::MODULE, 'organization' => self::ORGANIZATION, 'usage' => self::USAGE, 'contactPerson' => self::CONTACT_PERSON][$type] ?? null + ); + + $this->tracker = $this->progressTracker(); + + return new CmdbExportImportService( + container: $container, + settingsService: $settings, + contactSync: $this->contactSync(), + progressTracker: $this->tracker, + profile: new CmdbImportProfile(container: $container, directory: $profileDir), + reader: ($reader ?? new CmdbWorkbookReader()), + normaliser: new CmdbRowNormaliser(), + l10n: $this->l10n(), + logger: $this->logger() + ); + }//end service() + + /** + * Skip unless the fixture can be read. + * + * @return string The fixture path. + */ + private function fixture(): string { + if (CmdbTestSupport::loadPhpSpreadsheet() === false) { + $this->markTestSkipped('PhpSpreadsheet not found: set OPENREGISTER_DIR to an OpenRegister app with its vendor/ installed.'); + } + + return CmdbTestSupport::fixtures() . '/topdesk-export-anonymised.xlsx'; + }//end fixture() + + /** + * A synthetic application row of a CMDB sheet. + * + * @param string $appId The APPID. + * @param array $cells Overrides. + * @param int $row The row number. + * @param string $sheet The sheet. + * @param array $uncached Columns whose formula has no cached value. + * + * @return array{sheet: string, row: int, cells: array, uncached: array} + */ + private function row(string $appId, array $cells = [], int $row = 2, string $sheet = 'Beheerde Applicaties CMDB', array $uncached = []): array { + return [ + 'sheet' => $sheet, + 'row' => $row, + 'cells' => array_merge( + ['APPID' => $appId, 'Applicatie Code' => 'APP-' . $appId, 'Applicatie Naam' => 'Applicatie ' . $appId, 'Vendor' => 'Fabfrikant', 'Applicatie Status' => 'In productie'], + $cells + ), + 'uncached' => $uncached, + ]; + }//end row() + + /** + * The stored objects of a schema. + * + * @param int $schema The schema id. + * + * @return array> + */ + private function objects(int $schema): array { + return array_values($this->store[$schema]); + }//end objects() + + /** + * Seed an organisation. + * + * @param string $uuid The uuid. + * @param string $name The name. + * @param string $type The type. + * + * @return void + */ + private function seedOrganisation(string $uuid, string $name, string $type): void { + $this->store[self::ORGANIZATION][$uuid] = ['id' => $uuid, 'name' => $name, 'type' => $type, 'status' => 'Active']; + }//end seedOrganisation() + + // ------------------------------------------------------------------ + // Task 5: municipality, manufacturer, module upsert, usage + // ------------------------------------------------------------------ + + /** + * The sanitised export creates two modules, two suppliers, two usages and the municipality. + * + * @return void + */ + public function testTheFixtureCreatesModulesUsagesAndSuppliers(): void { + $path = $this->fixture(); + $before = (new \DateTimeImmutable('now', new \DateTimeZone('UTC')))->modify('-1 second'); + $report = $this->service()->import(path: $path, options: ['municipalityName' => 'Gemeente Voorbeeldstad', 'operationId' => 'cmdb-test-0001']); + + $this->assertTrue($report['success']); + $this->assertFalse($report['cancelled']); + $this->assertSame('cmdb-test-0001', $report['operationId']); + $this->assertSame(['rowsRead' => 2, 'processed' => 2, 'created' => 2, 'updated' => 0, 'unchanged' => 0, 'skipped' => 0, 'failed' => 0, 'warnings' => 1], $report['summary']); + $this->assertSame('Gemeente Voorbeeldstad', $report['municipality']['name']); + $this->assertTrue($report['municipality']['created']); + $this->assertSame([], $report['importWarnings']); + // "Webapplicatie" is an application kind, not a hosting model: the field is dropped with a warning. + $this->assertSame(['Column "Applicatiesoort": Value "Webapplicatie" has no mapping and no default is configured'], $report['rows'][0]['warnings']); + + $municipality = $report['municipality']['uuid']; + $this->assertSame('Municipality', $this->store[self::ORGANIZATION][$municipality]['type']); + $this->assertSame('Active', $this->store[self::ORGANIZATION][$municipality]['status']); + + $suppliers = array_filter($this->objects(self::ORGANIZATION), fn (array $o): bool => $o['type'] === 'Supplier'); + $this->assertEqualsCanonicalizing(['Aangetekend B.V.', 'Fabfrikant'], array_column($suppliers, 'name')); + + $modules = []; + foreach ($this->objects(self::MODULE) as $module) { + $modules[$module['externalNumber']] = $module; + } + + $this->assertSame(['1234', '2'], array_map('strval', array_keys($modules))); + $onbeh = $modules[1234]; + $this->assertSame('topdesk:' . $municipality . ':1234', $onbeh['externalKey']); + $this->assertSame('AIA-AangetekendMailen', $onbeh['externalId']); + $this->assertSame('Aangetekend Mailen', $onbeh['name']); + $this->assertSame('Mailen', $onbeh['shortDescription']); + $this->assertSame('Application', $onbeh['type']); + $this->assertSame('2023-07-04', $onbeh['externalCreatedAt']); + $this->assertSame('2026-07-29', $onbeh['externalModifiedAt']); + $this->assertSame('Functionele omschrijving test123', $onbeh['longDescription']); + $this->assertArrayNotHasKey('bbnLevel', $onbeh, '"NB" means unknown'); + $this->assertArrayNotHasKey('cloudDienstverleningsmodel', $onbeh); + $publication = new \DateTimeImmutable($onbeh['publicationDate']); + $this->assertGreaterThanOrEqual($before, $publication); + $this->assertLessThanOrEqual(new \DateTimeImmutable('now'), $publication); + + $beheerd = $modules[2]; + $this->assertSame($onbeh['publicationDate'], $beheerd['publicationDate'], 'one start time for the whole import'); + $this->assertSame('topdesk:' . $municipality . ':2', $beheerd['externalKey']); + $this->assertSame('APP-test123', $beheerd['externalId']); + $this->assertSame('naamtest123', $beheerd['name']); + $this->assertSame('Naamtest', $beheerd['shortDescription'], 'Roepnaam wins over Nickname'); + $this->assertSame('Accomodatieplanning.', $beheerd['longDescription']); + $this->assertSame(['SaaS'], $beheerd['cloudDienstverleningsmodel']); + $this->assertSame('BBN2', $beheerd['bbnLevel']); + + $supplierByName = array_column($suppliers, 'id', 'name'); + $this->assertSame($supplierByName['Aangetekend B.V.'], $onbeh['provider']); + $this->assertSame($supplierByName['Fabfrikant'], $beheerd['provider']); + + $usages = $this->objects(self::USAGE); + $this->assertCount(2, $usages); + $usageByModule = array_column($usages, null, 'module'); + $aia = $usageByModule[$onbeh['id']]; + $this->assertSame($municipality, $aia['consumer']); + $this->assertSame('Planned', $aia['status']); + $this->assertSame('Beheer geregeld: nee / H10 / H10 Accounting', $aia['interneAnnotation']); + $this->assertArrayNotHasKey('startDateOutPhased', $aia, 'the CMDB placeholder 2036-01-01 means no date'); + $this->assertArrayNotHasKey('timeClassification', $aia); + $this->assertSame($supplierByName['Aangetekend B.V.'], $aia['provider']); + $app = $usageByModule[$beheerd['id']]; + $this->assertSame('In production', $app['status']); + $this->assertSame('Tolerate', $app['timeClassification']); + $this->assertSame('2046-02-01', $app['startDateOutPhased']); + $this->assertSame('Beheer geregeld: ja / B10 / B10 Maatschappelijke Ontwikkeling', $app['interneAnnotation']); + $this->assertArrayNotHasKey('technicalOwner', $app); + + $this->assertSame($onbeh['id'], $report['rows'][0]['moduleUuid']); + $this->assertSame($aia['id'], $report['rows'][0]['usageUuid']); + $this->assertSame( + ['Onbeh Applicaties CMDB', 2, '1234', 'Aangetekend Mailen', 'created'], + [$report['rows'][0]['sheet'], $report['rows'][0]['row'], $report['rows'][0]['appId'], $report['rows'][0]['name'], $report['rows'][0]['outcome']] + ); + $this->assertSame('Beheerde Applicaties CMDB', $report['rows'][1]['sheet']); + }//end testTheFixtureCreatesModulesUsagesAndSuppliers() + + /** + * The same export again: 0 created, 2 unchanged, no save at all, one municipality. + * + * @return void + */ + public function testReimportingTheSameExportChangesNothing(): void { + $path = $this->fixture(); + $service = $this->service(); + $service->import(path: $path, options: ['municipalityName' => 'Gemeente Voorbeeldstad']); + $counts = array_map('count', $this->store); + $savesAfterFirst = count($this->saves); + + $report = $service->import(path: $path, options: ['municipalityName' => ' gemeente VOORBEELDSTAD ']); + + $this->assertSame(0, $report['summary']['created']); + $this->assertSame(2, $report['summary']['unchanged']); + $this->assertFalse($report['municipality']['created']); + $this->assertSame($savesAfterFirst, count($this->saves), 'no saveObject() call for unchanged objects'); + $this->assertSame($counts, array_map('count', $this->store)); + $municipalities = array_filter($this->objects(self::ORGANIZATION), fn (array $o): bool => $o['type'] === 'Municipality'); + $this->assertCount(1, $municipalities); + }//end testReimportingTheSameExportChangesNothing() + + /** + * The match key is the APPID: a changed Applicatie Code updates the same module. + * + * @return void + */ + public function testTheKeyIsTheAppIdNotTheCode(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '42', cells: ['Applicatie Code' => 'APP-Oud'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + $uuid = array_key_first($this->store[self::MODULE]); + + $report = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '42', cells: ['Applicatie Code' => 'App-Nieuw'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame('updated', $report['rows'][0]['outcome']); + $this->assertCount(1, $this->store[self::MODULE]); + $this->assertSame('App-Nieuw', $this->store[self::MODULE][$uuid]['externalId']); + $this->assertSame('topdesk:muni-1:42', $this->store[self::MODULE][$uuid]['externalKey']); + $this->assertSame('42', $this->store[self::MODULE][$uuid]['externalNumber']); + }//end testTheKeyIsTheAppIdNotTheCode() + + /** + * A changed Applicatie Naam updates the module; website, publicationDate and depublicationDate stay. + * + * @return void + */ + public function testAChangedNameUpdatesOnlyTheMappedFields(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $service = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '2', cells: ['Applicatie Naam' => 'naamtest123'])])); + $service->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $uuid = array_key_first($this->store[self::MODULE]); + $this->store[self::MODULE][$uuid]['website'] = 'https://voorbeeld.example'; + $this->store[self::MODULE][$uuid]['depublicationDate'] = '2026-10-02T00:00:00+00:00'; + $published = $this->store[self::MODULE][$uuid]['publicationDate']; + + $service = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '2', cells: ['Applicatie Naam' => 'naamtest124'])])); + $report = $service->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame('updated', $report['rows'][0]['outcome']); + $this->assertCount(1, $this->store[self::MODULE]); + $module = $this->store[self::MODULE][$uuid]; + $this->assertSame('naamtest124', $module['name']); + $this->assertSame('https://voorbeeld.example', $module['website']); + $this->assertSame($published, $module['publicationDate']); + $this->assertSame('2026-10-02T00:00:00+00:00', $module['depublicationDate']); + $this->assertCount(1, $this->store[self::USAGE], 'still one usage'); + }//end testAChangedNameUpdatesOnlyTheMappedFields() + + /** + * An existing module without publicationDate does not get one on update. + * + * @return void + */ + public function testAnUpdateNeverWritesPublicationDate(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->store[self::MODULE]['mod-1'] = ['id' => 'mod-1', 'name' => 'Oud', 'externalKey' => 'topdesk:muni-1:1']; + + $report = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1')]))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame('updated', $report['rows'][0]['outcome']); + $this->assertArrayNotHasKey('publicationDate', $this->store[self::MODULE]['mod-1']); + $this->assertArrayNotHasKey('type', $this->store[self::MODULE]['mod-1'], 'type is create-only'); + $this->assertSame('Applicatie 1', $this->store[self::MODULE]['mod-1']['name']); + }//end testAnUpdateNeverWritesPublicationDate() + + /** + * A municipality uuid must be an organisation of type Municipality. + * + * @return void + */ + public function testTheMunicipalityMustBeAMunicipality(): void { + $this->seedOrganisation(uuid: 'supplier-1', name: 'Voorbeeld Software B.V.', type: 'Supplier'); + foreach ([['municipalityUuid' => 'supplier-1'], ['municipalityUuid' => 'unknown-uuid']] as $options) { + try { + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1')]))->import(path: '', options: $options); + $this->fail('MUNICIPALITY_INVALID expected'); + } catch (CmdbImportException $e) { + $this->assertSame('MUNICIPALITY_INVALID', $e->getErrorCode()); + $this->assertSame(422, $e->getHttpStatus()); + } + } + + try { + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1')]))->import(path: '', options: ['municipalityName' => ' ']); + $this->fail('MUNICIPALITY_REQUIRED expected'); + } catch (CmdbImportException $e) { + $this->assertSame('MUNICIPALITY_REQUIRED', $e->getErrorCode()); + } + + $this->assertSame([], $this->saves, 'nothing is written'); + }//end testTheMunicipalityMustBeAMunicipality() + + /** + * "Fabfrikant", "Fabfrikant " and "FABFRIKANT" are one supplier; an existing supplier is reused. + * + * @return void + */ + public function testAVendorIsOneSupplier(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->seedOrganisation(uuid: 'aangetekend', name: 'Aangetekend B.V.', type: 'Supplier'); + $rows = [ + $this->row(appId: '1', cells: ['Vendor' => 'Fabfrikant'], row: 2), + $this->row(appId: '2', cells: ['Vendor' => 'Fabfrikant '], row: 3), + $this->row(appId: '3', cells: ['Vendor' => 'FABFRIKANT'], row: 4), + $this->row(appId: '4', cells: ['Vendor' => 'aangetekend b.v.'], row: 5), + $this->row(appId: '5', cells: ['Vendor' => ''], row: 6), + ]; + + $report = $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame(5, $report['summary']['created']); + $suppliers = array_filter($this->objects(self::ORGANIZATION), fn (array $o): bool => $o['type'] === 'Supplier'); + $this->assertCount(2, $suppliers); + $fabfrikant = array_values(array_filter($suppliers, fn (array $o): bool => $o['name'] === 'Fabfrikant'))[0]['id']; + $providers = array_column($this->objects(self::MODULE), 'provider', 'externalNumber'); + $this->assertSame([1 => $fabfrikant, 2 => $fabfrikant, 3 => $fabfrikant, 4 => 'aangetekend'], $providers); + }//end testAVendorIsOneSupplier() + + /** + * updateExisting=false reports a match as skipped "exists" and writes nothing. + * + * @return void + */ + public function testUpdateExistingFalseSkipsMatches(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1')]))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + $saves = count($this->saves); + + $report = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: ['Applicatie Naam' => 'Anders'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1', 'updateExisting' => false]); + + $this->assertSame('skipped', $report['rows'][0]['outcome']); + $this->assertSame(['exists'], $report['rows'][0]['reasons']); + $this->assertSame($saves, count($this->saves)); + }//end testUpdateExistingFalseSkipsMatches() + + /** + * A module missing from a newer export, and its usage, are left as they are. + * + * @return void + */ + public function testRecordsMissingFromTheExportStay(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', row: 2), $this->row(appId: '7', row: 3, sheet: 'Onbeh Applicaties CMDB')])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + $modules = $this->store[self::MODULE]; + $usages = $this->store[self::USAGE]; + + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: ['Applicatie Naam' => 'Nieuw'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + foreach ($modules as $uuid => $module) { + if ($module['externalNumber'] === '7') { + $this->assertSame($module, $this->store[self::MODULE][$uuid]); + } + } + + $this->assertSame($usages, $this->store[self::USAGE]); + $this->assertCount(2, $this->store[self::MODULE]); + }//end testRecordsMissingFromTheExportStay() + + /** + * An unknown Applicatie Status drops only that field and warns with column and value. + * + * @return void + */ + public function testAnUnknownStatusDropsOnlyThatField(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $report = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: ['Applicatie Status' => 'Onbekende status'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame('created', $report['rows'][0]['outcome']); + $this->assertCount(1, $report['rows'][0]['warnings']); + $this->assertStringContainsString('"Applicatie Status"', $report['rows'][0]['warnings'][0]); + $this->assertStringContainsString('Onbekende status', $report['rows'][0]['warnings'][0]); + $this->assertSame(1, $report['summary']['warnings']); + $usage = $this->objects(self::USAGE)[0]; + $this->assertArrayNotHasKey('status', $usage); + $this->assertCount(1, $this->store[self::MODULE]); + }//end testAnUnknownStatusDropsOnlyThatField() + + /** + * The sheet a row comes from records whether maintenance is arranged, in the usage's internal note; + * empty Cluster or Afdeling leave no empty part behind. + * + * @return void + */ + public function testTheSheetRecordsWhetherMaintenanceIsArranged(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $rows = [ + $this->row(appId: '1', cells: ['Cluster' => 'H10', 'Applicatie Eigenaar (Afdeling)' => 'H10 Accounting'], sheet: 'Onbeh Applicaties CMDB'), + $this->row(appId: '2', cells: ['Cluster' => '', 'Applicatie Eigenaar (Afdeling)' => 'B10 Ontwikkeling'], row: 3), + $this->row(appId: '3', cells: ['Cluster' => '', 'Applicatie Eigenaar (Afdeling)' => ''], row: 4), + ]; + + $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $notes = array_column($this->objects(self::USAGE), 'interneAnnotation'); + $this->assertSame(['Beheer geregeld: nee / H10 / H10 Accounting', 'Beheer geregeld: ja / B10 Ontwikkeling', 'Beheer geregeld: ja'], $notes); + }//end testTheSheetRecordsWhetherMaintenanceIsArranged() + + /** + * "NB" in BNN Classificatie and the CMDB end-of-life placeholder (serial 49675) mean empty: no field, no warning. + * + * @return void + */ + public function testTheCmdbPlaceholdersMeanEmpty(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $rows = [ + $this->row(appId: '1', cells: ['BNN Classificatie' => 'NB', 'End-of-Life Functioneel' => 49675]), + $this->row(appId: '2', cells: ['BNN Classificatie' => 'BBN 3', 'End-of-Life Functioneel' => 53359, 'Classificatie' => 'Migreren'], row: 3), + ]; + + $report = $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame(0, $report['summary']['warnings']); + $modules = array_column($this->objects(self::MODULE), null, 'externalNumber'); + $this->assertArrayNotHasKey('bbnLevel', $modules[1]); + $this->assertSame('BBN3', $modules[2]['bbnLevel']); + $usages = array_column($this->objects(self::USAGE), null, 'module'); + $this->assertArrayNotHasKey('startDateOutPhased', $usages[$modules[1]['id']]); + $this->assertSame('2046-02-01', $usages[$modules[2]['id']]['startDateOutPhased']); + $this->assertSame('Migrate', $usages[$modules[2]['id']]['timeClassification']); + }//end testTheCmdbPlaceholdersMeanEmpty() + + /** + * A formula without a cached value reads as empty and warns on its row; the row is still imported. + * + * @return void + */ + public function testAFormulaWithoutACachedValueWarns(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $report = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: ['Roepnaam' => null], uncached: ['Roepnaam'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame('created', $report['rows'][0]['outcome']); + $this->assertSame(['Column "Roepnaam": formula without a cached value, read as empty'], $report['rows'][0]['warnings']); + $this->assertArrayNotHasKey('shortDescription', $this->objects(self::MODULE)[0]); + }//end testAFormulaWithoutACachedValueWarns() + + /** + * A test-only module pack that maps one more column changes the import without code. + * + * @return void + */ + public function testAPackChangeChangesTheMapping(): void { + $directory = sys_get_temp_dir() . '/stackiq-cmdb-pack-' . bin2hex(random_bytes(4)); + mkdir($directory); + foreach (glob(CmdbTestSupport::appRoot() . '/lib/Settings/cmdb-import/*.json') as $file) { + copy($file, $directory . '/' . basename($file)); + } + + $pack = json_decode((string)file_get_contents($directory . '/topdesk-module.json'), true); + $pack['fieldMappings'][] = ['source' => 'Software Suite', 'target' => 'licentietype', 'transform' => ['type' => 'trim']]; + file_put_contents($directory . '/topdesk-module.json', json_encode($pack)); + + try { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: ['Software Suite' => 'Suite'])]), profileDir: $directory) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + } finally { + array_map('unlink', glob($directory . '/*.json')); + rmdir($directory); + } + + $this->assertSame('Suite', $this->objects(self::MODULE)[0]['licentietype']); + }//end testAPackChangeChangesTheMapping() + + // ------------------------------------------------------------------ + // Task 6: the owner as contact person + // ------------------------------------------------------------------ + + /** + * Each row's Applicatie Eigenaar (Persoon) becomes the usage's business owner, by display name; a + * function in that column is used as the display name too; the function becomes the role. + * + * @return void + */ + public function testTheOwnerBecomesTheBusinessOwner(): void { + $path = $this->fixture(); + $report = $this->service()->import(path: $path, options: ['municipalityName' => 'Gemeente Voorbeeldstad']); + $municipality = $report['municipality']['uuid']; + + $this->assertEqualsCanonicalizing(['Voornaam Achternaam', 'Teamleider Applicatiebeheer'], array_column($this->contacts, 'name')); + $this->assertSame(['', ''], array_column($this->contacts, 'email'), 'the CMDB sheets carry no e-mail address'); + + $people = $this->objects(self::CONTACT_PERSON); + $this->assertCount(2, $people); + $uidByName = array_flip(array_map(fn (array $c): string => $c['name'], $this->contacts)); + $byUid = array_column($people, null, 'contactsUid'); + $this->assertSame( + ['contactsUid' => $uidByName['Voornaam Achternaam'], 'organization' => $municipality, 'role' => 'Afdelingshoofd'], + array_diff_key($byUid[$uidByName['Voornaam Achternaam']], ['id' => true]) + ); + $this->assertSame('Teamleider Applicatiebeheer', $byUid[$uidByName['Teamleider Applicatiebeheer']]['role']); + + $usages = array_column($this->objects(self::USAGE), null, 'module'); + $this->assertSame($byUid[$uidByName['Voornaam Achternaam']]['id'], $usages[$report['rows'][0]['moduleUuid']]['businessOwner']); + $this->assertSame($byUid[$uidByName['Teamleider Applicatiebeheer']]['id'], $usages[$report['rows'][1]['moduleUuid']]['businessOwner']); + }//end testTheOwnerBecomesTheBusinessOwner() + + /** + * The same owner on two rows is one contact person, referenced by both usages. + * + * @return void + */ + public function testTheSameOwnerOnTwoRowsIsOneContactPerson(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $owner = ['Applicatie Eigenaar (Persoon)' => 'Achternaam, Voornaam', 'Applicatie Eigenaar (Functie)' => 'Afdelingshoofd']; + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: $owner, row: 2), $this->row(appId: '2', cells: $owner, row: 3)])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertCount(1, $this->objects(self::CONTACT_PERSON)); + $owners = array_unique(array_column($this->objects(self::USAGE), 'businessOwner')); + $this->assertSame([$this->objects(self::CONTACT_PERSON)[0]['id']], array_values($owners)); + }//end testTheSameOwnerOnTwoRowsIsOneContactPerson() + + /** + * An owner imported twice is one contact and one contact person; a near-namesake is not reused. + * + * @return void + */ + public function testAnOwnerByNameIsMatchedExactly(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + // A contact whose name merely contains the owner's name must not match. + $this->contacts['contact-other'] = ['name' => 'Voornaam Achternaam-Anders', 'email' => '']; + $rows = [$this->row(appId: '1', cells: ['Applicatie Eigenaar (Persoon)' => 'Achternaam, Voornaam'])]; + + $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertCount(2, $this->contacts, 'one new contact next to the near-namesake'); + $people = $this->objects(self::CONTACT_PERSON); + $this->assertCount(1, $people); + $this->assertNotSame('contact-other', $people[0]['contactsUid']); + $this->assertArrayNotHasKey('role', $people[0]); + $this->assertSame($people[0]['id'], $this->objects(self::USAGE)[0]['businessOwner']); + }//end testAnOwnerByNameIsMatchedExactly() + + /** + * No technical owner is written, whatever the row holds. + * + * @return void + */ + public function testNoTechnicalOwnerIsWritten(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: ['FB contactpersoon 1' => 'Achternaam, Voornaam'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertArrayNotHasKey('technicalOwner', $this->objects(self::USAGE)[0]); + $this->assertArrayNotHasKey('businessOwner', $this->objects(self::USAGE)[0]); + $this->assertSame([], $this->objects(self::CONTACT_PERSON)); + $this->assertSame([], $this->contacts); + }//end testNoTechnicalOwnerIsWritten() + + /** + * With Contacts disabled the modules and usages are saved without owners, with a warning on each row that has an owner. + * + * @return void + */ + public function testContactsDisabledDoesNotBlockTheImport(): void { + $path = $this->fixture(); + $this->contactsEnabled = false; + $report = $this->service()->import(path: $path, options: ['municipalityName' => 'Gemeente Voorbeeldstad']); + + $this->assertSame(2, $report['summary']['created']); + $this->assertCount(2, $this->objects(self::USAGE)); + $this->assertSame([], $this->objects(self::CONTACT_PERSON)); + $this->assertContains('Owners skipped: Nextcloud Contacts is unavailable', $report['rows'][0]['warnings']); + $this->assertSame(['Owners skipped: Nextcloud Contacts is unavailable'], $report['rows'][1]['warnings']); + }//end testContactsDisabledDoesNotBlockTheImport() + + /** + * An imported contact person has no e-mail and no username, so neither user-provisioning path picks it up. + * + * @return void + */ + public function testAnImportedContactPersonIsNeverAUser(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: ['Applicatie Eigenaar (Persoon)' => 'Achternaam, Voornaam', 'Applicatie Eigenaar (Functie)' => 'Afdelingshoofd'])])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $people = $this->objects(self::CONTACT_PERSON); + $this->assertNotEmpty($people); + foreach ($people as $person) { + $this->assertSame([], array_diff(array_keys($person), ['id', 'contactsUid', 'organization', 'role'])); + } + + // OrganizationSyncService::performUserSync() selects contact persons with a username. + $sync = (string)file_get_contents(CmdbTestSupport::appRoot() . '/lib/Service/OrganizationSyncService.php'); + $this->assertStringContainsString('o.username IS NOT NULL', $sync, 'the selection changed: re-check that imported contact persons stay out of it'); + // ContactpersoonService::processContactpersoon() provisions only from an e-mail on the object. + $listener = (string)file_get_contents(CmdbTestSupport::appRoot() . '/lib/Service/ContactpersoonService.php'); + $this->assertStringContainsString("\$email = (\$contactData['email'] ?? \$contactData['e-mailadres'] ?? '');", $listener); + }//end testAnImportedContactPersonIsNeverAUser() + + /** + * Neither the report nor any log line names an owner. + * + * @return void + */ + public function testNoPersonDataInReportOrLog(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $owner = ['Applicatie Eigenaar (Persoon)' => 'Achternaam, Voornaam', 'Applicatie Eigenaar (Functie)' => 'Afdelingshoofd']; + $this->beforeSave = function (int $schema, array $data): void { + if ($schema === self::USAGE && ($data['module'] ?? '') !== '' && count($this->objects(self::USAGE)) === 1) { + throw new RuntimeException('usage refused'); + } + }; + $report = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1', cells: $owner, row: 2), $this->row(appId: '2', cells: $owner, row: 3)])) + ->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $text = json_encode($report, JSON_UNESCAPED_UNICODE) . "\n" . implode("\n", $this->logLines); + foreach (['Achternaam', 'Voornaam'] as $personData) { + $this->assertStringNotContainsString($personData, $text); + } + + $this->assertSame('failed', $report['rows'][1]['outcome'], 'the injected failure ran'); + }//end testNoPersonDataInReportOrLog() + + // ------------------------------------------------------------------ + // Task 7: row isolation, report, progress and cancel + // ------------------------------------------------------------------ + + /** + * A failing module save fails only its row, naming the step. + * + * @return void + */ + public function testOneBadRowDoesNotStopTheOthers(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $this->beforeSave = function (int $schema, array $data): void { + if ($schema === self::MODULE && ($data['externalNumber'] ?? '') === '2') { + throw new RuntimeException('Validation failed for name'); + } + }; + $rows = [$this->row(appId: '1', row: 2), $this->row(appId: '2', row: 3), $this->row(appId: '3', row: 4)]; + + $report = $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame(['created', 'failed', 'created'], array_column($report['rows'], 'outcome')); + $this->assertStringStartsWith('step "module" failed', $report['rows'][1]['reasons'][0]); + $this->assertSame(['rowsRead' => 3, 'processed' => 3, 'created' => 2, 'updated' => 0, 'unchanged' => 0, 'skipped' => 0, 'failed' => 1, 'warnings' => 0], $report['summary']); + $this->assertCount(2, $this->store[self::MODULE]); + }//end testOneBadRowDoesNotStopTheOthers() + + /** + * A duplicate APPID (also across the two sheets), a missing APPID and a missing Applicatie Naam are skipped with their reasons. + * + * @return void + */ + public function testRowsAreSkippedWithTheirReasons(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $rows = [ + $this->row(appId: '2', row: 2), + $this->row(appId: '2', row: 7), + $this->row(appId: '', row: 8), + $this->row(appId: '9', cells: ['Applicatie Naam' => ' '], row: 10), + $this->row(appId: '2', row: 2, sheet: 'Onbeh Applicaties CMDB'), + ]; + + $report = $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1']); + + $this->assertSame( + [ + ['created', []], + ['skipped', ['duplicate APPID in file']], + ['skipped', ['missing APPID']], + ['skipped', ['missing Applicatie Naam']], + ['skipped', ['duplicate APPID in file']], + ], + array_map(fn (array $row): array => [$row['outcome'], $row['reasons']], $report['rows']) + ); + $this->assertCount(1, $this->store[self::MODULE]); + }//end testRowsAreSkippedWithTheirReasons() + + /** + * The import runs as a cmdb_import operation with per-row progress; afterwards its statistics hold the report. + * + * @return void + */ + public function testProgressIsRecordedAndHoldsTheReport(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $seen = []; + $this->beforeSave = function (int $schema) use (&$seen): void { + if ($schema === self::MODULE) { + $seen[] = $this->tracker->getProgress(operationId: 'cmdb-progress-1')['processed_items']; + } + }; + $rows = [$this->row(appId: '1', row: 2), $this->row(appId: '2', row: 3)]; + + $report = $this->service(reader: $this->rowsReader(rows: $rows))->import(path: '', options: ['municipalityUuid' => 'muni-1', 'operationId' => 'cmdb-progress-1']); + + $this->assertSame([0, 1], $seen, 'progress advances after every row'); + $stored = $this->cache['progress_cmdb-progress-1']; + $this->assertSame('cmdb_import', $stored['operation_type']); + $this->assertSame('completed', $stored['status']); + $this->assertSame($report, $stored['statistics']['report']); + }//end testProgressIsRecordedAndHoldsTheReport() + + /** + * A cancel after row 1 of 3 keeps row 1 and reports cancelled with one processed row. + * + * @return void + */ + public function testACancelStopsBetweenRows(): void { + $this->seedOrganisation(uuid: 'muni-1', name: 'Gemeente Voorbeeldstad', type: 'Municipality'); + $service = null; + $this->beforeSave = function (int $schema) use (&$service): void { + if ($schema === self::USAGE) { + $this->assertTrue($service->requestCancel(operationId: 'cmdb-cancel-01')); + } + }; + $rows = [$this->row(appId: '1', row: 2), $this->row(appId: '2', row: 3), $this->row(appId: '3', row: 4)]; + $service = $this->service(reader: $this->rowsReader(rows: $rows)); + + $report = $service->import(path: '', options: ['municipalityUuid' => 'muni-1', 'operationId' => 'cmdb-cancel-01']); + + $this->assertTrue($report['cancelled']); + $this->assertSame(1, $report['summary']['processed']); + $this->assertSame(3, $report['summary']['rowsRead']); + $this->assertCount(1, $report['rows']); + $this->assertCount(1, $this->store[self::MODULE], 'row 1 stays'); + $this->assertSame('cancelled', $this->cache['progress_cmdb-cancel-01']['status']); + $this->assertSame($report, $this->cache['progress_cmdb-cancel-01']['statistics']['report']); + }//end testACancelStopsBetweenRows() + + /** + * Cancel answers false for an unknown id, a malformed id or another operation type. + * + * @return void + */ + public function testCancelNeedsACmdbOperation(): void { + $service = $this->service(reader: $this->rowsReader(rows: [])); + $this->tracker->startOperation(operationType: 'archimate_import', operationId: 'cmdb-not-mine-1'); + + $this->assertFalse($service->requestCancel(operationId: 'cmdb-unknown-1')); + $this->assertFalse($service->requestCancel(operationId: 'archimate_import_abcdefgh')); + $this->assertFalse($service->requestCancel(operationId: 'cmdb-not-mine-1')); + }//end testCancelNeedsACmdbOperation() + + /** + * Without a mapping engine, or without configuration, nothing is read or written. + * + * @return void + */ + public function testMissingEngineOrConfigurationStopsBeforeReading(): void { + try { + $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1')]), config: [])->import(path: '', options: ['municipalityName' => 'Gemeente Voorbeeldstad']); + $this->fail('NOT_CONFIGURED expected'); + } catch (CmdbImportException $e) { + $this->assertSame('NOT_CONFIGURED', $e->getErrorCode()); + $this->assertSame(503, $e->getHttpStatus()); + } + + $base = $this->service(reader: $this->rowsReader(rows: [$this->row(appId: '1')])); + $reflection = new \ReflectionClass($base); + $args = []; + foreach ($reflection->getConstructor()->getParameters() as $parameter) { + $property = $reflection->getProperty($parameter->getName()); + $args[$parameter->getName()] = $property->getValue($base); + } + + $withoutEngine = new class(...$args) extends CmdbExportImportService { + public const ENGINE_CLASS = 'OCA\OpenRegister\Service\MigrationPack\NoSuchEngine'; + }; + + try { + $withoutEngine->import(path: '', options: ['municipalityName' => 'Gemeente Voorbeeldstad']); + $this->fail('MAPPING_UNAVAILABLE expected'); + } catch (CmdbImportException $e) { + $this->assertSame('MAPPING_UNAVAILABLE', $e->getErrorCode()); + $this->assertSame(503, $e->getHttpStatus()); + } + + $this->assertSame([], $this->saves); + }//end testMissingEngineOrConfigurationStopsBeforeReading() + + /** + * Person names split as TOPdesk writes them ("Achternaam, Voornaam"). + * + * @return void + */ + public function testPersonNamesSplit(): void { + $this->assertSame(['voornaam' => 'Voornaam', 'achternaam' => 'Achternaam'], CmdbExportImportService::splitPersonName(name: 'Achternaam, Voornaam ')); + $this->assertSame(['voornaam' => '', 'achternaam' => 'Functioneel Beheer'], CmdbExportImportService::splitPersonName(name: 'Functioneel Beheer')); + }//end testPersonNamesSplit() +}//end class diff --git a/tests/Unit/Settings/CmdbPersonDataVisibilityTest.php b/tests/Unit/Settings/CmdbPersonDataVisibilityTest.php new file mode 100644 index 000000000..ccb8d6f25 --- /dev/null +++ b/tests/Unit/Settings/CmdbPersonDataVisibilityTest.php @@ -0,0 +1,134 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-6 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Settings; + +use OCA\Stackiq\Service\SettingsService; +use PHPUnit\Framework\TestCase; +use ReflectionMethod; + +/** + * Pins the register rules that keep imported owners out of public reads. + */ +class CmdbPersonDataVisibilityTest extends TestCase { + /** + * The register after merging every fragment in sorted filename order. + * + * @return array + */ + private function mergedRegister(): array { + $dir = __DIR__ . '/../../../lib/Settings'; + $register = json_decode((string)file_get_contents($dir . '/softwarecatalogus_register.json'), true); + $merge = new ReflectionMethod(SettingsService::class, 'deepMergeConfig'); + + $files = glob($dir . '/register.d/*.json'); + sort($files); + foreach ($files as $file) { + $register = $merge->invoke(null, $register, json_decode((string)file_get_contents($file), true)); + } + + return $register; + }//end mergedRegister() + + /** + * Whether a read rule lets the public group in. + * + * @param mixed $rule A string group or a {group, match} rule. + * + * @return bool + */ + private static function isPublic(mixed $rule): bool { + if (is_string($rule) === true) { + return $rule === 'public'; + } + + return is_array($rule) === true && ($rule['group'] ?? null) === 'public'; + }//end isPublic() + + /** + * Neither usage nor contactPerson can be read anonymously. + * + * @return void + */ + public function testUsageAndContactPersonHaveNoPublicReadRule(): void { + $schemas = $this->mergedRegister()['components']['schemas']; + + foreach (['usage', 'contactPerson'] as $schema) { + $read = ($schemas[$schema]['authorization']['read'] ?? null); + $this->assertIsArray($read, $schema . ' must have an explicit read rule; without one OpenRegister does not restrict reads'); + $this->assertNotEmpty($read, $schema); + foreach ($read as $rule) { + $this->assertFalse(self::isPublic(rule: $rule), $schema . ' has a public read rule: imported owners would be readable anonymously'); + } + } + }//end testUsageAndContactPersonHaveNoPublicReadRule() + + /** + * A published module refers to its contact person and usages by relation only, and holds no person field. + * + * @return void + */ + public function testAModuleOnlyRefersToPeopleByRelation(): void { + $module = $this->mergedRegister()['components']['schemas']['module']; + + $this->assertTrue( + array_filter($module['authorization']['read'], fn (mixed $rule): bool => self::isPublic(rule: $rule)) !== [], + 'modules are public once published; that is why the person data must stay on other schemas' + ); + $this->assertSame('#/components/schemas/contactPerson', $module['properties']['contactPerson']['$ref']); + $this->assertSame('#/components/schemas/usage', $module['properties']['usages']['$ref']); + foreach (['businessOwner', 'technicalOwner', 'email', 'owner', 'eigenaar'] as $field) { + $this->assertArrayNotHasKey($field, $module['properties'], 'module.' . $field . ' would be public'); + } + }//end testAModuleOnlyRefersToPeopleByRelation() + + /** + * The import writes no person data onto a module; only the owner pack reads a person column. + * + * @return void + */ + public function testTheImportWritesNoPersonDataOntoAModule(): void { + $dir = __DIR__ . '/../../../lib/Settings/cmdb-import'; + $person = ['Applicatie Eigenaar (Persoon)', 'Applicatie Eigenaar (Functie)']; + + foreach (glob($dir . '/topdesk-*.json') as $file) { + $pack = json_decode((string)file_get_contents($file), true); + foreach (($pack['fieldMappings'] ?? []) as $mapping) { + if (basename($file) === 'topdesk-module.json') { + $this->assertNotContains($mapping['target'], ['contactPerson', 'usages'], 'the module pack writes ' . $mapping['target']); + } + + if (in_array($mapping['source'], $person, true) === true) { + $this->assertSame('topdesk-business-owner.json', basename($file), $mapping['source'] . ' is read outside the owner pack'); + } + } + } + }//end testTheImportWritesNoPersonDataOntoAModule() +}//end class diff --git a/tests/Unit/Settings/TopdeskCmdbFragmentTest.php b/tests/Unit/Settings/TopdeskCmdbFragmentTest.php new file mode 100644 index 000000000..71badecff --- /dev/null +++ b/tests/Unit/Settings/TopdeskCmdbFragmentTest.php @@ -0,0 +1,132 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-2 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Settings; + +use OCA\Stackiq\Service\SettingsService; +use PHPUnit\Framework\TestCase; +use ReflectionMethod; + +/** + * Merges every fragment in filename order, exactly as SettingsService::loadSettings() does. + */ +class TopdeskCmdbFragmentTest extends TestCase { + /** + * The external-id properties the fragment adds. + * + * @var array + */ + private const PROPERTIES = ['externalId', 'externalNumber', 'externalKey', 'externalCreatedAt', 'externalModifiedAt']; + + /** + * The register after merging every fragment in sorted filename order. + * + * @return array + */ + private function mergedRegister(): array { + $dir = __DIR__ . '/../../../lib/Settings'; + $register = json_decode((string)file_get_contents($dir . '/softwarecatalogus_register.json'), true); + $merge = new ReflectionMethod(SettingsService::class, 'deepMergeConfig'); + + $files = glob($dir . '/register.d/*.json'); + sort($files); + foreach ($files as $file) { + $register = $merge->invoke(null, $register, json_decode((string)file_get_contents($file), true)); + } + + return $register; + }//end mergedRegister() + + /** + * The merged module is 0.3.5 and carries the five optional, titled properties. + * + * @return void + */ + public function testTheMergedModuleIsVersion035WithTheExternalIds(): void { + $module = $this->mergedRegister()['components']['schemas']['module']; + + $this->assertSame('0.3.5', $module['version'], 'a fragment sorting after topdesk-cmdb-import.json overwrote the bump'); + foreach (self::PROPERTIES as $property) { + $this->assertArrayHasKey($property, $module['properties']); + $this->assertNotEmpty($module['properties'][$property]['title'] ?? '', $property); + $this->assertNotEmpty($module['properties'][$property]['description'] ?? '', $property); + $this->assertSame('string', $module['properties'][$property]['type'], $property); + $this->assertNotContains($property, $module['required'] ?? [], $property); + $this->assertNotTrue($module['properties'][$property]['required'] ?? false, $property); + } + + $this->assertSame(100, $module['properties']['externalId']['maxLength']); + $this->assertSame(50, $module['properties']['externalNumber']['maxLength']); + $this->assertSame(200, $module['properties']['externalKey']['maxLength']); + $this->assertSame(['default' => false], $module['properties']['externalKey']['table']); + $this->assertSame('date', $module['properties']['externalCreatedAt']['format']); + $this->assertSame('date', $module['properties']['externalModifiedAt']['format']); + $this->assertArrayHasKey('roadmapStatement', $module['properties'], 'the 0.3.4 fragment still applies'); + $this->assertSame(['name'], $module['required']); + }//end testTheMergedModuleIsVersion035WithTheExternalIds() + + /** + * The fragment sorts after the fragment that set module 0.3.4. + * + * @return void + */ + public function testTheFragmentSortsAfterTheRoadmapFragment(): void { + $names = ['maintenance-and-roadmap.json', 'topdesk-cmdb-import.json']; + $sorted = $names; + sort($sorted); + $this->assertSame($names, $sorted); + }//end testTheFragmentSortsAfterTheRoadmapFragment() + + /** + * The three seed modules exist without publicationDate or externalKey, and every seed field is a schema property. + * + * @return void + */ + public function testTheSeedModulesShowTheNewProperties(): void { + $register = $this->mergedRegister(); + $properties = $register['components']['schemas']['module']['properties']; + $seeds = []; + foreach ($register['components']['objects'] as $object) { + if (($object['@self']['schema'] ?? null) === 'module') { + $seeds[$object['@self']['slug']] = $object; + } + } + + $this->assertSame(['voorbeeld-zaaksysteem', 'voorbeeld-afsprakenplanner', 'voorbeeld-belastingapplicatie'], array_keys($seeds)); + $this->assertSame(['APP-00001', 'APP-00002', 'AIA-00003'], array_column(array_values($seeds), 'externalId')); + foreach ($seeds as $slug => $seed) { + $this->assertArrayNotHasKey('publicationDate', $seed, $slug); + $this->assertArrayNotHasKey('externalKey', $seed, $slug); + $this->assertSame('stackiq', $seed['@self']['register'], $slug); + foreach (array_keys($seed) as $field) { + if ($field !== '@self') { + $this->assertArrayHasKey($field, $properties, $slug . '.' . $field); + } + } + + $this->assertContains($seed['bbnLevel'], $properties['bbnLevel']['enum'], $slug); + $this->assertContains($seed['type'], $properties['type']['enum'], $slug); + } + + // The base seeds are still there: the fragment appends, it does not replace. + $this->assertGreaterThan(3, count($register['components']['objects'])); + }//end testTheSeedModulesShowTheNewProperties() +}//end class diff --git a/tests/Unit/Support/CmdbTestSupport.php b/tests/Unit/Support/CmdbTestSupport.php new file mode 100644 index 000000000..90f51c0e0 --- /dev/null +++ b/tests/Unit/Support/CmdbTestSupport.php @@ -0,0 +1,159 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * @link https://github.com/ConductionNL/stackiq + * + * @spec openspec/changes/cmdb-export-import/tasks.md#task-4 + * + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * SPDX-License-Identifier: EUPL-1.2 + */ + +declare(strict_types=1); + +namespace OCA\Stackiq\Tests\Unit\Support; + +/** + * Locates and loads the OpenRegister pieces the CMDB import uses. + */ +final class CmdbTestSupport { + /** + * Which migration-pack classes were loaded: "real" or "copy". + * + * @var string|null + */ + private static ?string $packSource = null; + + /** + * The app root. + * + * @return string + */ + public static function appRoot(): string { + return dirname(__DIR__, 3); + }//end appRoot() + + /** + * The fixture directory. + * + * @return string + */ + public static function fixtures(): string { + return self::appRoot() . '/tests/fixtures/cmdb'; + }//end fixtures() + + /** + * The OpenRegister app directory, when one is available. + * + * @return string|null + */ + public static function openRegisterDir(): ?string { + $candidates = []; + $env = getenv('OPENREGISTER_DIR'); + if (is_string($env) === true && $env !== '') { + $candidates[] = $env; + } + + // An app next to openregister, or a worktree two levels below the apps directory. + $candidates[] = dirname(self::appRoot()) . '/openregister'; + $candidates[] = dirname(self::appRoot(), 2) . '/openregister'; + + foreach ($candidates as $candidate) { + if (is_file($candidate . '/lib/Service/MigrationPack/MappingEngine.php') === true) { + return $candidate; + } + } + + return null; + }//end openRegisterDir() + + /** + * Load MappingEngine and PackDefinitionValidator. + * + * @return string "real" or "copy". + */ + public static function loadMigrationPack(): string { + if (self::$packSource !== null) { + return self::$packSource; + } + + $dir = self::openRegisterDir(); + $source = 'copy'; + $base = __DIR__ . '/OpenRegister'; + if ($dir !== null) { + $source = 'real'; + $base = $dir . '/lib/Service/MigrationPack'; + } + + foreach (['PackDefinitionValidator', 'MappingEngine'] as $class) { + if (class_exists('OCA\\OpenRegister\\Service\\MigrationPack\\' . $class, false) === false) { + require_once $base . '/' . $class . '.php'; + } + } + + self::$packSource = $source; + return $source; + }//end loadMigrationPack() + + /** + * Make PhpSpreadsheet loadable from OpenRegister's vendor directory. + * + * @return bool Whether the Xlsx reader can be loaded. + */ + public static function loadPhpSpreadsheet(): bool { + if (class_exists('PhpOffice\\PhpSpreadsheet\\Reader\\Xlsx') === true) { + return true; + } + + $dir = self::openRegisterDir(); + if ($dir === null || is_dir($dir . '/vendor/phpoffice/phpspreadsheet') === false) { + return false; + } + + $vendor = $dir . '/vendor'; + $prefixes = [ + 'PhpOffice\\PhpSpreadsheet\\' => $vendor . '/phpoffice/phpspreadsheet/src/PhpSpreadsheet/', + 'Psr\\SimpleCache\\' => $vendor . '/psr/simple-cache/src/', + 'Composer\\Pcre\\' => $vendor . '/composer/pcre/src/', + 'Matrix\\' => $vendor . '/markbaker/matrix/classes/src/', + 'Complex\\' => $vendor . '/markbaker/complex/classes/src/', + ]; + + // Appended, so a library this app already ships keeps winning. + spl_autoload_register( + static function (string $class) use ($prefixes): void { + foreach ($prefixes as $prefix => $path) { + if (str_starts_with($class, $prefix) === false) { + continue; + } + + $file = $path . str_replace('\\', '/', substr($class, strlen($prefix))) . '.php'; + if (is_file($file) === true) { + require_once $file; + } + + return; + } + } + ); + + return class_exists('PhpOffice\\PhpSpreadsheet\\Reader\\Xlsx') === true; + }//end loadPhpSpreadsheet() +}//end class diff --git a/tests/Unit/Support/OpenRegister/MappingEngine.php b/tests/Unit/Support/OpenRegister/MappingEngine.php new file mode 100644 index 000000000..9e06f56c3 --- /dev/null +++ b/tests/Unit/Support/OpenRegister/MappingEngine.php @@ -0,0 +1,353 @@ + + * value, with `id` recognised by the existing update-by-id convention), so + * the single write path (`ObjectService::saveObjects()`/`saveObject()`) is + * unchanged. + * + * Literal-leak guard (fleet lesson — a transform/template reference that + * doesn't resolve must ERROR the row, never pass the literal through): the + * `lookup` transform errors the row when the source value is present but + * has no entry in the map and no `default` is configured, rather than + * silently passing the raw, unmapped source value through to the target + * schema property. + * + * SPDX-License-Identifier: EUPL-1.2 + * SPDX-FileCopyrightText: 2026 Conduction B.V. + * + * @category Service + * @package OCA\OpenRegister\Service\MigrationPack + * + * @author Conduction Development Team + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * @link https://OpenRegister.app + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + */ + +declare(strict_types=1); + +/* + * TEST COPY, not loaded in production. Verbatim copy of OpenRegister + * lib/Service/MigrationPack/MappingEngine.php at version 2.1.34, so the + * CMDB import tests run where OpenRegister is not checked out (CI). Where it is, + * tests/Unit/Support/CmdbTestSupport.php loads the real class instead (set + * OPENREGISTER_DIR). Refresh this copy when OpenRegister changes the pack format. + */ + + +namespace OCA\OpenRegister\Service\MigrationPack; + +use DateTime; + +/** + * Maps one source row (CSV row / Excel row / decoded JSON object) onto a set + * of target schema-property values, per a migration-pack definition. + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + */ +class MappingEngine { + /** + * Apply a pack definition to one source row. + * + * @param array $pack The validated pack definition (decoded JSON). + * @param array $sourceRow The parsed source row (flat for CSV/Excel, possibly nested for JSON). + * @param int $rowNumber 1-based row number, used only to label errors. + * + * @return array{data: array, errors: list} + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + */ + public function mapRow(array $pack, array $sourceRow, int $rowNumber): array { + $data = $pack['defaults'] ?? []; + $errors = []; + + foreach (($pack['fieldMappings'] ?? []) as $mapping) { + $source = (string)($mapping['source'] ?? ''); + $target = (string)($mapping['target'] ?? ''); + $required = ($mapping['required'] ?? false) === true; + $transform = $mapping['transform'] ?? null; + $transformId = null; + if (is_array($transform) === true) { + $transformId = ($transform['type'] ?? null); + } + + $rawValue = $this->resolveSource(row: $sourceRow, pointer: $source); + $isEmpty = ($rawValue === null || $rawValue === ''); + + if ($required === true && $isEmpty === true) { + $errors[] = [ + 'row' => $rowNumber, + 'source' => $source, + 'target' => $target, + 'transform' => $transformId, + 'message' => sprintf('Required source field "%s" is missing or empty', $source), + ]; + continue; + } + + // A `const` transform always applies, regardless of the source value. + // Every other transform is skipped (leaving any seeded default in + // place) when the source is empty and the mapping is optional — + // there is nothing to map, and nothing to error. + if ($isEmpty === true && $transformId !== 'const') { + continue; + } + + $result = $this->applyTransform( + value: $rawValue, + transform: $transform, + sourceRow: $sourceRow + ); + + if ($result['error'] !== null) { + $errors[] = [ + 'row' => $rowNumber, + 'source' => $source, + 'target' => $target, + 'transform' => $transformId, + 'message' => $result['error'], + ]; + continue; + } + + $data[$target] = $result['value']; + }//end foreach + + $data = $this->applyIdStrategy(pack: $pack, sourceRow: $sourceRow, data: $data); + + return [ + 'data' => $data, + 'errors' => $errors, + ]; + }//end mapRow() + + /** + * Whether a given (1-based) row number is listed in the pack's `skipRows`. + * + * @param array $pack The pack definition. + * @param int $rowNumber 1-based row number. + * + * @return bool + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + */ + public function isRowSkipped(array $pack, int $rowNumber): bool { + $skipRows = $pack['skipRows'] ?? []; + return in_array($rowNumber, $skipRows, true); + }//end isRowSkipped() + + /** + * Resolve the id/uuid target from `idStrategy`, mutating `data['id']` + * when the strategy is `sourceField` and a value is present. The + * `generate` strategy is a no-op here — leaving `data['id']` unset lets + * the existing import pipeline treat the row as a create, exactly as it + * already does for CSV/JSON rows with no id column. + * + * @param array $pack The pack definition. + * @param array $sourceRow The source row. + * @param array $data The mapped target data so far. + * + * @return array The (possibly id-augmented) target data. + */ + private function applyIdStrategy(array $pack, array $sourceRow, array $data): array { + $idStrategy = $pack['idStrategy'] ?? ['type' => 'generate']; + if (($idStrategy['type'] ?? 'generate') !== 'sourceField') { + return $data; + } + + $idValue = $this->resolveSource(row: $sourceRow, pointer: (string)($idStrategy['field'] ?? '')); + if ($idValue !== null && $idValue !== '') { + $data['id'] = (string)$idValue; + } + + return $data; + }//end applyIdStrategy() + + /** + * Resolve a source value from a row, given either a flat key or a + * JSON-Pointer-style `/a/b/c` path. + * + * @param array $row The source row. + * @param string $pointer A flat key or a leading-`/` pointer path. + * + * @return mixed The resolved value, or null when not found. + */ + private function resolveSource(array $row, string $pointer) { + if ($pointer === '') { + return null; + } + + if ($pointer[0] !== '/') { + return $row[$pointer] ?? null; + } + + $segments = explode('/', ltrim($pointer, '/')); + $cursor = $row; + foreach ($segments as $segment) { + $segment = str_replace(['~1', '~0'], ['/', '~'], $segment); + if (is_array($cursor) === false || array_key_exists($segment, $cursor) === false) { + return null; + } + + $cursor = $cursor[$segment]; + } + + return $cursor; + }//end resolveSource() + + /** + * Apply one transform to a resolved source value. + * + * @param mixed $value The resolved source value (never null/'' — callers filter + * that). + * @param array|null $transform The transform block, or null for identity passthrough. + * @param array $sourceRow The full source row (needed by `concat` to resolve extra fields). + * + * @return array{value: mixed, error: ?string} + * + * @SuppressWarnings(PHPMD.CyclomaticComplexity) One branch per transform type. + */ + private function applyTransform($value, ?array $transform, array $sourceRow): array { + if ($transform === null) { + return ['value' => $value, 'error' => null]; + } + + switch ($transform['type'] ?? null) { + case 'trim': + $stringValue = (string)$value; + if (is_string($value) === true) { + $stringValue = $value; + } + return ['value' => trim($stringValue), 'error' => null]; + case 'date': + return $this->applyDateTransform(value: $value, transform: $transform); + case 'bool-map': + return $this->applyMapTransform(value: $value, transform: $transform, coerceBool: true); + case 'lookup': + return $this->applyMapTransform(value: $value, transform: $transform, coerceBool: false); + case 'concat': + return $this->applyConcatTransform(value: $value, transform: $transform, sourceRow: $sourceRow); + case 'const': + return ['value' => ($transform['value'] ?? null), 'error' => null]; + default: + return ['value' => null, 'error' => 'Unknown transform type "' . (string)($transform['type'] ?? '') . '"']; + }//end switch + }//end applyTransform() + + /** + * `date` transform: parse the source value with `sourceFormat` (or a + * best-effort `DateTime` parse when omitted) and re-emit it as `targetFormat` + * (default `Y-m-d`). + * + * @param mixed $value The resolved source value. + * @param array $transform The transform block. + * + * @return array{value: mixed, error: ?string} + * + * @SuppressWarnings(PHPMD.StaticAccess) DateTime::createFromFormat is the standard PHP idiom for a strict-format parse. + */ + private function applyDateTransform($value, array $transform): array { + $sourceFormat = $transform['sourceFormat'] ?? null; + $targetFormat = $transform['targetFormat'] ?? 'Y-m-d'; + $stringValue = (string)$value; + + try { + $date = null; + if (is_string($sourceFormat) === true && $sourceFormat !== '') { + $date = DateTime::createFromFormat($sourceFormat, $stringValue); + if ($date === false) { + return [ + 'value' => null, + 'error' => sprintf('Could not parse date "%s" with format "%s"', $stringValue, $sourceFormat), + ]; + } + } + + if ($date === null) { + $date = new DateTime($stringValue); + } + } catch (\Throwable $e) { + return ['value' => null, 'error' => sprintf('Could not parse date "%s": %s', $stringValue, $e->getMessage())]; + } + + return ['value' => $date->format($targetFormat), 'error' => null]; + }//end applyDateTransform() + + /** + * Shared implementation for `bool-map` and `lookup` — both resolve the + * source value through a `map`, with an optional `default` and, absent a + * default, an error on an unresolved key (the literal-leak guard). + * + * @param mixed $value The resolved source value. + * @param array $transform The transform block. + * @param bool $coerceBool Whether to cast the mapped value to bool (bool-map) or return it as-is (lookup). + * + * @return array{value: mixed, error: ?string} + */ + private function applyMapTransform($value, array $transform, bool $coerceBool): array { + $map = $transform['map'] ?? []; + $key = (string)$value; + + if (array_key_exists($key, $map) === true) { + $mapped = $map[$key]; + if ($coerceBool === true) { + $mapped = (bool)$mapped; + } + + return ['value' => $mapped, 'error' => null]; + } + + if (array_key_exists('default', $transform) === true) { + $default = $transform['default']; + if ($coerceBool === true) { + $default = (bool)$default; + } + + return ['value' => $default, 'error' => null]; + } + + // Literal-leak guard: an unresolved map key is a data-quality problem the + // migration operator must see and fix, never a value that silently passes + // through unmapped into the target schema property. + return [ + 'value' => null, + 'error' => sprintf('Value "%s" has no mapping and no default is configured', $key), + ]; + }//end applyMapTransform() + + /** + * `concat` transform: join the primary source value with 0+ additional + * source fields, using `separator` (default a single space). Additional + * fields that resolve to nothing are treated as empty strings — a + * missing *optional* extra field is not itself a literal-leak case, since + * there is no map lookup involved. + * + * @param mixed $value The resolved primary source value. + * @param array $transform The transform block. + * @param array $sourceRow The full source row. + * + * @return array{value: mixed, error: ?string} + */ + private function applyConcatTransform($value, array $transform, array $sourceRow): array { + $separator = $transform['separator'] ?? ' '; + $parts = [(string)$value]; + + foreach (($transform['fields'] ?? []) as $extraSource) { + $extraValue = $this->resolveSource(row: $sourceRow, pointer: (string)$extraSource); + $parts[] = (string)($extraValue ?? ''); + } + + return ['value' => implode($separator, $parts), 'error' => null]; + }//end applyConcatTransform() +}//end class diff --git a/tests/Unit/Support/OpenRegister/PackDefinitionValidator.php b/tests/Unit/Support/OpenRegister/PackDefinitionValidator.php new file mode 100644 index 000000000..6d4afb1ad --- /dev/null +++ b/tests/Unit/Support/OpenRegister/PackDefinitionValidator.php @@ -0,0 +1,383 @@ + + * @copyright 2026 Conduction B.V. + * @license EUPL-1.2 https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 + * + * @link https://OpenRegister.app + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + */ + +declare(strict_types=1); + +/* + * TEST COPY, not loaded in production. Verbatim copy of OpenRegister + * lib/Service/MigrationPack/PackDefinitionValidator.php at version 2.1.34, so the + * CMDB import tests run where OpenRegister is not checked out (CI). Where it is, + * tests/Unit/Support/CmdbTestSupport.php loads the real class instead (set + * OPENREGISTER_DIR). Refresh this copy when OpenRegister changes the pack format. + */ + + +namespace OCA\OpenRegister\Service\MigrationPack; + +use InvalidArgumentException; + +/** + * Structural + business-rule validator for a migration-pack JSON document. + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + * + * @SuppressWarnings(PHPMD.ExcessiveClassComplexity) One small, independently-testable validate*() method + * per pack-document field keeps each check simple; the class total sums them, not any single method. + */ +class PackDefinitionValidator { + /** + * Source formats a pack may declare. + * + * @var string[] + */ + public const ALLOWED_SOURCE_FORMATS = ['csv', 'json', 'excel']; + + /** + * Transform types a field mapping may declare. + * + * @var string[] + */ + public const ALLOWED_TRANSFORM_TYPES = ['trim', 'date', 'bool-map', 'concat', 'lookup', 'const']; + + /** + * IdStrategy types a pack may declare. + * + * @var string[] + */ + public const ALLOWED_ID_STRATEGY_TYPES = ['sourceField', 'generate']; + + /** + * Validate a pack definition document. + * + * @param array $definition The decoded pack definition JSON. + * + * @return string[] List of validation error messages. Empty when valid. + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + */ + public function validate(array $definition): array { + $errors = []; + + $errors = array_merge($errors, $this->validateId(definition: $definition)); + $errors = array_merge($errors, $this->validateName(definition: $definition)); + $errors = array_merge($errors, $this->validateSourceFormat(definition: $definition)); + $errors = array_merge($errors, $this->validateVersion(definition: $definition)); + $errors = array_merge($errors, $this->validateFieldMappings(definition: $definition)); + $errors = array_merge($errors, $this->validateDefaults(definition: $definition)); + $errors = array_merge($errors, $this->validateSkipRows(definition: $definition)); + $errors = array_merge($errors, $this->validateIdStrategy(definition: $definition)); + + return $errors; + }//end validate() + + /** + * Validate and throw on the first structural problem. + * + * @param array $definition The decoded pack definition JSON. + * + * @return void + * + * @throws InvalidArgumentException When the definition is invalid. The message joins every error found. + * + * @spec exclude test copy of an OpenRegister class; its spec is migration-mapping-packs in the openregister repository + */ + public function assertValid(array $definition): void { + $errors = $this->validate(definition: $definition); + if (empty($errors) === false) { + throw new InvalidArgumentException('Invalid migration pack definition: ' . implode('; ', $errors)); + } + }//end assertValid() + + /** + * Validate the `id` field (pack slug, used as the lookup key). + * + * @param array $definition The pack definition. + * + * @return string[] + */ + private function validateId(array $definition): array { + $id = $definition['id'] ?? null; + if (is_string($id) === false || $id === '') { + return ['"id" is required and must be a non-empty string']; + } + + if (preg_match('/^[a-z0-9][a-z0-9-]*$/', $id) !== 1) { + return ['"id" must be a lowercase slug (letters, digits, hyphens), got "' . $id . '"']; + } + + return []; + }//end validateId() + + /** + * Validate the `name` field. + * + * @param array $definition The pack definition. + * + * @return string[] + */ + private function validateName(array $definition): array { + $name = $definition['name'] ?? null; + if (is_string($name) === false || $name === '') { + return ['"name" is required and must be a non-empty string']; + } + + return []; + }//end validateName() + + /** + * Validate the `sourceFormat` field. + * + * @param array $definition The pack definition. + * + * @return string[] + */ + private function validateSourceFormat(array $definition): array { + $format = $definition['sourceFormat'] ?? null; + if (is_string($format) === false || in_array($format, self::ALLOWED_SOURCE_FORMATS, true) === false) { + return [ + '"sourceFormat" must be one of: ' . implode(', ', self::ALLOWED_SOURCE_FORMATS) + . ' (got ' . var_export($format, true) . ')', + ]; + } + + return []; + }//end validateSourceFormat() + + /** + * Validate the `version` field (strict semver: MAJOR.MINOR.PATCH). + * + * @param array $definition The pack definition. + * + * @return string[] + */ + private function validateVersion(array $definition): array { + $version = $definition['version'] ?? null; + if (is_string($version) === false || preg_match('/^\d+\.\d+\.\d+$/', $version) !== 1) { + return ['"version" must be a semver string (MAJOR.MINOR.PATCH), got ' . var_export($version, true)]; + } + + return []; + }//end validateVersion() + + /** + * Validate the `fieldMappings` array and every entry within it. + * + * @param array $definition The pack definition. + * + * @return string[] + * + * @SuppressWarnings(PHPMD.CyclomaticComplexity) Per-transform-type validation requires many branches. + * @SuppressWarnings(PHPMD.NPathComplexity) Per-transform-type validation requires many branches. + */ + private function validateFieldMappings(array $definition): array { + $mappings = $definition['fieldMappings'] ?? null; + if (is_array($mappings) === false || empty($mappings) === true) { + return ['"fieldMappings" is required and must be a non-empty array']; + } + + $errors = []; + foreach ($mappings as $index => $mapping) { + $path = 'fieldMappings[' . $index . ']'; + if (is_array($mapping) === false) { + $errors[] = $path . ' must be an object'; + continue; + } + + $source = $mapping['source'] ?? null; + if (is_string($source) === false || $source === '') { + $errors[] = $path . '.source is required and must be a non-empty string'; + } + + $target = $mapping['target'] ?? null; + if (is_string($target) === false || $target === '') { + $errors[] = $path . '.target is required and must be a non-empty string'; + } + + if (isset($mapping['required']) === true && is_bool($mapping['required']) === false) { + $errors[] = $path . '.required must be a boolean when present'; + } + + if (isset($mapping['transform']) === true) { + $errors = array_merge($errors, $this->validateTransform(transform: $mapping['transform'], path: $path . '.transform')); + } + }//end foreach + + return $errors; + }//end validateFieldMappings() + + /** + * Validate one `transform` block. + * + * @param mixed $transform The transform value to validate. + * @param string $path The error-message path prefix. + * + * @return string[] + * + * @SuppressWarnings(PHPMD.CyclomaticComplexity) Each transform type has its own required-field shape. + * @SuppressWarnings(PHPMD.NPathComplexity) Each transform type has its own required-field shape. + */ + private function validateTransform($transform, string $path): array { + if (is_array($transform) === false) { + return [$path . ' must be an object with a "type" key']; + } + + $type = $transform['type'] ?? null; + if (is_string($type) === false || in_array($type, self::ALLOWED_TRANSFORM_TYPES, true) === false) { + return [ + $path . '.type must be one of: ' . implode(', ', self::ALLOWED_TRANSFORM_TYPES) + . ' (got ' . var_export($type, true) . ')', + ]; + } + + switch ($type) { + case 'date': + if (isset($transform['sourceFormat']) === true && is_string($transform['sourceFormat']) === false) { + return [$path . '.sourceFormat must be a string when present']; + } + + if (isset($transform['targetFormat']) === true && is_string($transform['targetFormat']) === false) { + return [$path . '.targetFormat must be a string when present']; + } + break; + + case 'bool-map': + if (is_array($transform['map'] ?? null) === false || empty($transform['map']) === true) { + return [$path . '.map is required and must be a non-empty object for a bool-map transform']; + } + break; + + case 'lookup': + if (is_array($transform['map'] ?? null) === false || empty($transform['map']) === true) { + return [$path . '.map is required and must be a non-empty object for a lookup transform']; + } + break; + + case 'concat': + if (is_array($transform['fields'] ?? null) === false || empty($transform['fields']) === true) { + return [$path . '.fields is required and must be a non-empty array for a concat transform']; + } + + foreach ($transform['fields'] as $fieldIndex => $field) { + if (is_string($field) === false || $field === '') { + return [$path . '.fields[' . $fieldIndex . '] must be a non-empty string']; + } + } + break; + + case 'const': + if (array_key_exists('value', $transform) === false) { + return [$path . '.value is required for a const transform']; + } + break; + + case 'trim': + default: + // No extra fields required. + break; + }//end switch + + return []; + }//end validateTransform() + + /** + * Validate the optional `defaults` map. + * + * @param array $definition The pack definition. + * + * @return string[] + */ + private function validateDefaults(array $definition): array { + if (isset($definition['defaults']) === false) { + return []; + } + + if (is_array($definition['defaults']) === false) { + return ['"defaults" must be an object of target-property => default value when present']; + } + + return []; + }//end validateDefaults() + + /** + * Validate the optional `skipRows` array. + * + * @param array $definition The pack definition. + * + * @return string[] + */ + private function validateSkipRows(array $definition): array { + if (isset($definition['skipRows']) === false) { + return []; + } + + if (is_array($definition['skipRows']) === false) { + return ['"skipRows" must be an array of row numbers when present']; + } + + foreach ($definition['skipRows'] as $row) { + if (is_int($row) === false || $row < 1) { + return ['"skipRows" entries must be positive integers']; + } + } + + return []; + }//end validateSkipRows() + + /** + * Validate the required `idStrategy` block. + * + * @param array $definition The pack definition. + * + * @return string[] + */ + private function validateIdStrategy(array $definition): array { + $idStrategy = $definition['idStrategy'] ?? null; + if (is_array($idStrategy) === false) { + return ['"idStrategy" is required and must be an object with a "type" key']; + } + + $type = $idStrategy['type'] ?? null; + if (is_string($type) === false || in_array($type, self::ALLOWED_ID_STRATEGY_TYPES, true) === false) { + return [ + 'idStrategy.type must be one of: ' . implode(', ', self::ALLOWED_ID_STRATEGY_TYPES) + . ' (got ' . var_export($type, true) . ')', + ]; + } + + if ($type === 'sourceField' + && (is_string($idStrategy['field'] ?? null) === false || $idStrategy['field'] === '') + ) { + return ['idStrategy.field is required and must be a non-empty string when idStrategy.type is "sourceField"']; + } + + return []; + }//end validateIdStrategy() +}//end class diff --git a/tests/e2e/spec-coverage/cmdb-import.spec.ts b/tests/e2e/spec-coverage/cmdb-import.spec.ts new file mode 100644 index 000000000..29927a1ac --- /dev/null +++ b/tests/e2e/spec-coverage/cmdb-import.spec.ts @@ -0,0 +1,566 @@ +// SPDX-License-Identifier: EUPL-1.2 +// SPDX-FileCopyrightText: 2026 Conduction B.V. +/** + * E2e coverage for openspec/changes/cmdb-export-import (the "CMDB import" + * section of stackiq's Nextcloud admin settings). + * + * Every scenario the spec tags `@e2e tests/e2e/spec-coverage/cmdb-import.spec.ts` + * is driven here through the REAL settings page: the NcSelect municipality + * chooser, the real file input (`setInputFiles`), the Import button and the + * rendered report. The API is used for setup (a run-unique municipality), + * for the "no object written" checks, and for cleanup. + * + * The municipality is created per run (`Gemeente Voorbeeldstad `), + * because the import's match key is scoped to the municipality: a fixed name + * would make the first import of a second run report `unchanged`, not + * `created`. + * + * The anonymised fixtures come from the backend task + * (tests/fixtures/cmdb/, see its README). When one is absent the test that + * needs it is skipped with a message naming the missing file. + * + * Owner contacts the import creates in the admin's Nextcloud address book + * are not removed by the cleanup below; the OpenRegister objects are. + * + * The last test checks, without signing in, that the imported owners are + * not readable anonymously: neither through OpenRegister's objects API nor + * in an OpenCatalogi search hit (skipped when OpenCatalogi is not installed). + */ + +import type { APIRequestContext, Locator, Page, Response } from '@playwright/test' +import type { VoorzieningenConfig } from '../workflows/_fixtures.ts' + +import { expect, request as playwrightRequest, test } from '@playwright/test' +import * as fs from 'fs' +import * as path from 'path' +import { + BASE_URL, + createObject, + deleteObject, + findAll, + newApiContext, + resolveConfig, + RUN_ID, +} from '../workflows/_fixtures.ts' + +const FIXTURES_DIR = path.resolve(__dirname, '../../fixtures/cmdb') +const EXPORT_FIXTURE = path.join(FIXTURES_DIR, 'topdesk-export-anonymised.xlsx') +const MISSING_COLUMN_FIXTURE = path.join(FIXTURES_DIR, 'topdesk-missing-appid.xlsx') +// The owner values the anonymised export holds (tests/fixtures/cmdb/README.md). +const OWNER_VALUES = ['Achternaam', 'Voornaam', 'Teamleider Applicatiebeheer'] + +type UploadFile = Parameters[0] + +const MUNICIPALITY_NAME = `Gemeente Voorbeeldstad ${RUN_ID}` +const IMPORT_PATH = '/index.php/apps/stackiq/api/cmdb-import' +// The page builds its URL with generateUrl(), which drops `/index.php` on an +// instance with pretty URLs, so the browser-side matchers use the path tail. +const IMPORT_ROUTE = '**/apps/stackiq/api/cmdb-import' + +/** + * Whether a response is the answer to the import upload. + * + * @param response The response + */ +function isImportAnswer(response: Response): boolean { + return ( + new URL(response.url()).pathname.endsWith('/apps/stackiq/api/cmdb-import') + && response.request().method() === 'POST' + ) +} + +let config: VoorzieningenConfig +let municipalityUuid = '' + +/** + * Skip the calling test when a fixture is not there yet. + * + * @param file The fixture path + */ +function requireFixture(file: string): void { + test.skip( + !fs.existsSync(file), + `Fixture ${path.relative(process.cwd(), file)} is missing; it is produced by Task 1 of openspec/changes/cmdb-export-import (tests/fixtures/cmdb/build-fixtures.py).`, + ) +} + +/** + * All objects of a schema whose data mentions the run's municipality: its + * usages (consumer), modules (externalKey) and contact persons (organization). + * + * @param ctx The API context + * @param schema The schema id + */ +async function objectsOfMunicipality( + ctx: APIRequestContext, + schema: string, +): Promise>> { + const rows = await findAll(ctx, config.register, schema) + return rows.filter((row) => JSON.stringify(row).includes(municipalityUuid)) +} + +/** + * Count the objects the import can write for the run's municipality. + * + * @param ctx The API context + */ +async function countWritten(ctx: APIRequestContext): Promise<{ + modules: number + usages: number + contactPersons: number + municipalities: number +}> { + const municipalities = ( + await findAll(ctx, config.register, config.organisatie_schema) + ).filter((org) => org.type === 'Municipality') + return { + modules: (await objectsOfMunicipality(ctx, config.module_schema)).length, + usages: (await objectsOfMunicipality(ctx, config.gebruik_schema)).length, + contactPersons: ( + await objectsOfMunicipality(ctx, config.contactpersoon_schema) + ).length, + municipalities: municipalities.length, + } +} + +/** + * Get Nextcloud's own first-run wizard out of the way. + * + * It opens on an admin's first visit to a fresh instance, and its modal mask + * intercepts every click, so the chooser below would time out on a click + * that reads like a broken select. It has no close button and ignores + * Escape until its last slide, so it is marked as seen through its own + * route (what finishing it does) and the page is loaded again. A bounded + * wait, because the wizard mounts after the page. + * + * @param page The page + * @return True when the page was reloaded + */ +async function dismissFirstRunWizard(page: Page): Promise { + const wizard = page.locator('.first-run-wizard[role="dialog"]') + try { + await wizard.waitFor({ state: 'visible', timeout: 3000 }) + } catch { + return false + } + await page.evaluate(async () => { + const oc = ( + window as unknown as { + OC: { requestToken: string; generateUrl: (u: string) => string } + } + ).OC + await fetch(oc.generateUrl('/apps/firstrunwizard/wizard'), { + method: 'DELETE', + headers: { requesttoken: oc.requestToken }, + }) + }) + await page.reload({ waitUntil: 'domcontentloaded' }) + return true +} + +/** + * Open stackiq's admin settings and return the CMDB import section. + * + * @param page The page + */ +async function gotoCmdbSection(page: Page) { + await page.goto('/settings/admin/stackiq', { waitUntil: 'domcontentloaded' }) + const section = page.locator('[data-testid="cmdb-import"]') + await expect(section).toBeVisible({ timeout: 30000 }) + if (await dismissFirstRunWizard(page)) { + await expect(section).toBeVisible({ timeout: 30000 }) + } + await section.scrollIntoViewIfNeeded() + return section +} + +/** + * Pick the run's municipality in the chooser, the way an admin does. + * + * @param page The page + */ +async function chooseMunicipality(page: Page): Promise { + const input = page.locator('#cmdb-import-municipality') + await input.click() + await input.fill(MUNICIPALITY_NAME) + await page + .getByRole('option') + // hasText, not the accessible name: NcSelect splits a long option + // into two spans for its middle ellipsis. + .filter({ hasText: MUNICIPALITY_NAME }) + .first() + .click() + await expect( + page.locator('[data-testid="cmdb-import-municipality"] .vs__selected'), + ).toContainText(MUNICIPALITY_NAME) +} + +/** + * Read one summary count from the rendered report. + * + * @param page The page + * @param key The summary key (created, unchanged, …) + */ +function summaryValue(page: Page, key: string) { + return page.locator( + `[data-testid="cmdb-import-summary-${key}"] .cmdb-import__tile-value`, + ) +} + +/** + * The rendered report rows. + * + * @param page The page + */ +function reportRows(page: Page) { + return page.locator( + '[data-testid="cmdb-import-rows"] [data-testid="cn-object-row"]', + ) +} + +/** + * Choose a file, press Import and wait for the import request to answer. + * + * @param page The page + * @param file The file to upload + */ +async function runImport(page: Page, file: UploadFile) { + await page.locator('[data-testid="cmdb-import-file"]').setInputFiles(file) + const answer = page.waitForResponse(isImportAnswer, { timeout: 120000 }) + await page.locator('[data-testid="cmdb-import-start"]').click() + return await answer +} + +test.describe.serial('CMDB import section', () => { + test.beforeAll(async () => { + const ctx = await newApiContext() + try { + config = await resolveConfig(ctx) + municipalityUuid = await createObject( + ctx, + config.register, + config.organisatie_schema, + { name: MUNICIPALITY_NAME, type: 'Municipality', status: 'Active' }, + ) + } finally { + await ctx.dispose() + } + }) + + test.afterAll(async () => { + if (!municipalityUuid) { + return + } + const ctx = await newApiContext() + try { + for (const schema of [ + config.gebruik_schema, + config.contactpersoon_schema, + config.module_schema, + ]) { + for (const row of await objectsOfMunicipality(ctx, schema)) { + const id = String( + row.id + ?? (row['@self'] as { id?: string } | undefined)?.id + ?? '', + ) + if (id !== '') { + await deleteObject(ctx, config.register, schema, id) + } + } + } + await deleteObject( + ctx, + config.register, + config.organisatie_schema, + municipalityUuid, + ) + } finally { + await ctx.dispose() + } + }) + + // @e2e cmdb-export-import::the-admin-runs-an-import-from-the-settings-page + // @e2e cmdb-export-import::the-admin-picks-an-existing-municipality + // @e2e cmdb-export-import::upload-with-a-per-row-report + test('an admin imports the anonymised export for an existing municipality', async ({ + page, + }) => { + requireFixture(EXPORT_FIXTURE) + const ctx = await newApiContext() + const before = await countWritten(ctx) + + const section = await gotoCmdbSection(page) + await chooseMunicipality(page) + + // Hold the import request for a moment so the running state is + // observable. route.continue() forwards the original multipart body; + // route.fetch() would re-send it without the file. + await page.route(IMPORT_ROUTE, async (route) => { + await new Promise((resolve) => setTimeout(resolve, 1500)) + await route.continue() + }) + await page + .locator('[data-testid="cmdb-import-file"]') + .setInputFiles(EXPORT_FIXTURE) + const answer = page.waitForResponse(isImportAnswer, { timeout: 120000 }) + await page.locator('[data-testid="cmdb-import-start"]').click() + + // A progress bar shows while the import runs. + const progress = section.locator('[data-testid="cmdb-import-progress"]') + await expect(progress).toBeVisible() + await expect(progress.getByRole('progressbar')).toBeVisible() + await expect( + section.locator('[data-testid="cmdb-import-cancel"]'), + ).toBeVisible() + + const response = await answer + expect(response.status()).toBe(200) + await page.unroute(IMPORT_ROUTE) + await expect(progress).toBeHidden({ timeout: 30000 }) + + // Summary: 2 rows read, 2 created. + await expect( + section.locator('[data-testid="cmdb-import-summary"]'), + ).toBeVisible() + await expect(summaryValue(page, 'rowsRead')).toHaveText('2') + await expect(summaryValue(page, 'created')).toHaveText('2') + + // The report lists exactly the two data rows, not the hundreds of + // formatted but empty rows below them, each created with a module link. + const rows = reportRows(page) + await expect(rows).toHaveCount(2) + for (const [sheet, appId, name] of [ + ['Onbeh Applicaties CMDB', '1234', 'Aangetekend Mailen'], + ['Beheerde Applicaties CMDB', '2', 'naamtest123'], + ]) { + const row = rows.filter({ hasText: name }) + await expect(row).toHaveCount(1) + await expect(row).toContainText(sheet) + await expect(row.locator('td').nth(1)).toHaveText('2') + await expect(row.locator('td').nth(2)).toHaveText(appId) + await expect(row.locator('[data-outcome="created"]')).toBeVisible() + await expect( + row.locator('[data-testid="cmdb-import-module-link"]'), + ).toHaveAttribute('href', /\/apps\/stackiq\/modules\/[0-9a-f-]{36}$/) + } + + // Filtering the table on `created` shows the two imported rows. + await section.locator('#cmdb-import-outcome-filter').click() + await page + .getByRole('option') + .filter({ hasText: /^\s*(Created|Aangemaakt)\s*$/ }) + .first() + .click() + await expect(rows).toHaveCount(2) + + // Both usages point at the chosen municipality, and no new + // municipality was created. + const after = await countWritten(ctx) + expect(after.usages - before.usages).toBe(2) + expect(after.modules - before.modules).toBe(2) + expect(after.municipalities).toBe(before.municipalities) + const usages = await objectsOfMunicipality(ctx, config.gebruik_schema) + for (const usage of usages) { + expect(String(usage.consumer)).toContain(municipalityUuid) + } + await ctx.dispose() + }) + + // @e2e cmdb-export-import::re-importing-the-same-export-creates-no-duplicates + test('importing the same export again reports both rows unchanged', async ({ + page, + }) => { + requireFixture(EXPORT_FIXTURE) + const ctx = await newApiContext() + const before = await countWritten(ctx) + test.skip( + before.modules === 0, + 'The first import (previous test) wrote nothing for this municipality, so there is nothing to re-import.', + ) + + const section = await gotoCmdbSection(page) + await chooseMunicipality(page) + const response = await runImport(page, EXPORT_FIXTURE) + expect(response.status()).toBe(200) + + await expect(summaryValue(page, 'rowsRead')).toHaveText('2') + await expect(summaryValue(page, 'created')).toHaveText('0') + await expect(summaryValue(page, 'unchanged')).toHaveText('2') + await expect( + reportRows(page).locator('[data-outcome="unchanged"]'), + ).toHaveCount(2) + await expect( + section.locator('[data-testid="cmdb-import-error"]'), + ).toHaveCount(0) + + // Same number of modules, usages, contact persons and municipalities. + expect(await countWritten(ctx)).toEqual(before) + await ctx.dispose() + }) + + // @e2e cmdb-export-import::a-missing-required-column-is-named-in-the-422-response + test('an export without a required column names the column and the sheet', async ({ + page, + }) => { + requireFixture(MISSING_COLUMN_FIXTURE) + const ctx = await newApiContext() + const before = await countWritten(ctx) + + const section = await gotoCmdbSection(page) + await chooseMunicipality(page) + const response = await runImport(page, MISSING_COLUMN_FIXTURE) + + expect(response.status()).toBe(422) + const body = await response.json() + expect(body.error).toBe('MISSING_COLUMN') + expect(body.details).toEqual({ + sheet: 'Beheerde Applicaties CMDB', + column: 'APPID', + }) + + const error = section.locator('[data-testid="cmdb-import-error"]') + await expect(error).toBeVisible() + await expect(error).toContainText('"Beheerde Applicaties CMDB"') + await expect(error).toContainText('"APPID"') + await expect(error).toContainText('MISSING_COLUMN') + await expect( + section.locator('[data-testid="cmdb-import-report"]'), + ).toHaveCount(0) + + expect(await countWritten(ctx)).toEqual(before) + await ctx.dispose() + }) + + // @e2e cmdb-export-import::a-file-that-is-not-xlsx-is-rejected + test('a CSV, or a text file named .xlsx, is rejected as not xlsx', async ({ + page, + }) => { + const ctx = await newApiContext() + const before = await countWritten(ctx) + const csv = { + name: 'applications.csv', + mimeType: 'text/csv', + buffer: Buffer.from('APPID;Applicatie Naam\n2;naamtest123\n'), + } + const textAsXlsx = { + name: 'export.xlsx', + mimeType: + 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + buffer: Buffer.from('APPID;Applicatie Naam\n2;naamtest123\n'), + } + + // The endpoint answers 400 NOT_XLSX for both. + for (const file of [csv, textAsXlsx]) { + const res = await ctx.post(IMPORT_PATH, { + multipart: { + cmdbFile: file, + municipalityUuid, + }, + }) + expect(res.status(), file.name).toBe(400) + expect((await res.json()).error, file.name).toBe('NOT_XLSX') + } + + // The section shows the NOT_XLSX message for both: for the CSV before + // anything is sent, for the text file from the server's answer. + const section = await gotoCmdbSection(page) + await chooseMunicipality(page) + const error = section.locator('[data-testid="cmdb-import-error"]') + + await page.locator('[data-testid="cmdb-import-file"]').setInputFiles(csv) + await expect(error).toBeVisible() + await expect(error).toContainText('NOT_XLSX') + await expect(error).toContainText('.xlsx') + + const response = await runImport(page, textAsXlsx) + expect(response.status()).toBe(400) + await expect(error).toBeVisible() + await expect(error).toContainText('NOT_XLSX') + + // No module, usage, contact person or municipality was written. + expect(await countWritten(ctx)).toEqual(before) + await ctx.dispose() + }) + + // @e2e cmdb-export-import::imported-owners-are-never-readable-anonymously + test('the imported owners are not readable without signing in', async () => { + requireFixture(EXPORT_FIXTURE) + const ctx = await newApiContext() + const written = await countWritten(ctx) + await ctx.dispose() + test.skip( + written.contactPersons === 0, + 'The first import (first test) wrote no owner for this municipality, so there is nothing to look for.', + ) + + // Inside the test runner a new request context inherits the project's + // `use` options, including the admin storageState; clear it explicitly. + const anonymous = await playwrightRequest.newContext({ + baseURL: BASE_URL, + storageState: { cookies: [], origins: [] }, + }) + try { + // Prove the context is anonymous before trusting an empty answer. + const whoami = await anonymous.get( + '/ocs/v2.php/cloud/user?format=json', + { + headers: { 'OCS-APIRequest': 'true' }, + }, + ) + expect(whoami.status(), 'the context must not be signed in').toBe(401) + + // OpenRegister: no contact person and no usage for an anonymous caller. + for (const schema of [ + config.contactpersoon_schema, + config.gebruik_schema, + ]) { + const res = await anonymous.get( + `/index.php/apps/openregister/api/objects/${config.register}/${schema}?_limit=200`, + ) + if (res.ok()) { + const body = await res.json() + expect( + body.total ?? (body.results ?? []).length, + `schema ${schema}`, + ).toBe(0) + } else { + expect([401, 403], `schema ${schema}`).toContain(res.status()) + } + } + + // OpenCatalogi: a search hit for an imported module names nobody. + const search = await anonymous.get( + '/index.php/apps/opencatalogi/api/search?_search=naamtest123&_limit=50', + ) + test.skip( + search.status() === 404, + 'OpenCatalogi is not installed on this instance.', + ) + expect(search.ok()).toBe(true) + const hits = ((await search.json()).results ?? []) as Array< + Record + > + for (const hit of hits) { + const text = JSON.stringify(hit) + for (const value of OWNER_VALUES) { + expect(text, `search hit ${String(hit.id)}`).not.toContain(value) + } + for (const field of ['contactPerson', 'usages']) { + const value = hit[field] + const ids = Array.isArray(value) ? value : [value] + for (const id of ids) { + expect( + id === null + || id === undefined + || typeof id === 'string', + `${field} of search hit ${String(hit.id)} is an id or empty`, + ).toBe(true) + } + } + } + } finally { + await anonymous.dispose() + } + }) +}) diff --git a/tests/fixtures/cmdb/README.md b/tests/fixtures/cmdb/README.md new file mode 100644 index 000000000..c94830307 --- /dev/null +++ b/tests/fixtures/cmdb/README.md @@ -0,0 +1,45 @@ +# CMDB import fixtures + +Test workbooks for the TOPdesk CMDB import (`openspec/changes/cmdb-export-import`). +They are used by the PHPUnit tests under `tests/Unit/` and by the Playwright test +`tests/e2e/spec-coverage/cmdb-import.spec.ts`. + +The import reads the two CMDB sheets, "Onbeh Applicaties CMDB" and "Beheerde +Applicaties CMDB". Their cells are formulas over the "Invoer" sheets; the import +reads the value Excel cached for each formula. `build-fixtures.py` writes a +placeholder cached value into the formula cells of the mapped columns that the +anonymised export left empty (Roepnaam, Applicatiesoort, the owner's function and +person on "Beheerde", BNN Classificatie, Nickname, …); the list is `CACHED_VALUES` +in the script. The "Invoer" sheets are not read and stay as they are. + +| File | What it is | +|---|---| +| `topdesk-export-anonymised.xlsx` | An anonymised TOPdesk export with one fake data row per CMDB sheet (APPID 1234 on "Onbeh", APPID 2 on "Beheerde"), formatted but empty rows below them, and on "Beheerde" ten formula rows whose cached value is `0` (Excel's result for a reference to an empty cell). Document metadata, custom properties, `customXml/`, the workbook's absolute save path and `xl/connections.xml` are removed. | +| `topdesk-missing-appid.xlsx` | The same, but the "APPID" header of "Beheerde Applicaties CMDB" is renamed, so the required column is missing there. | +| `topdesk-shuffled-columns.xlsx` | The same rows with the columns of both CMDB sheets in reverse order, and the header "Vendor" written as `Vendor⚡`. Reads to the same rows as the original. | +| `topdesk-formula-and-connection.xlsx` | On "Beheerde", "Applicatie Naam" is a formula that would evaluate to `Evaluated` with the cached value `Rekenmodel`, "Roepnaam" is a formula without any cached value, and the package declares a synthetic external web connection to `https://example.invalid/`. | +| `topdesk-no-source-sheet.xlsx` | A minimal workbook with only a sheet "Blad1". | + +## Placeholder data only + +Every person value is a placeholder: `Achternaam, Voornaam`, +`letter.achternaam@gemeente.nl`, `groepsmail.test@gemeente.nl`, personnel +number `123456`, and the function `Teamleider Applicatiebeheer` where the CMDB +sheet shows a function instead of an owner. `tests/Unit/Fixtures/CmdbFixtureHygieneTest.php` +fails when a fixture holds document metadata, an e-mail address, a linked host or +a long number that is not on its placeholder list. Never commit a municipality's +own export, not even temporarily. + +## Rebuilding + +`build-fixtures.py` uses the Python standard library only: + +```bash +# Write the cached values into the committed export (idempotent) and derive the variants +python3 tests/fixtures/cmdb/build-fixtures.py + +# Sanitise a new anonymised export first, then write the cached values and derive the variants +python3 tests/fixtures/cmdb/build-fixtures.py --source path/to/anonymised-export.xlsx +``` + +Run the hygiene test afterwards. diff --git a/tests/fixtures/cmdb/build-fixtures.py b/tests/fixtures/cmdb/build-fixtures.py new file mode 100644 index 000000000..8536f73e7 --- /dev/null +++ b/tests/fixtures/cmdb/build-fixtures.py @@ -0,0 +1,355 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: 2026 Conduction B.V. +# SPDX-License-Identifier: EUPL-1.2 +"""Build the CMDB import test fixtures (cmdb-export-import, Task 1). + +Python standard library only (zipfile, re). openpyxl is deliberately not used: +re-saving through a spreadsheet library would rewrite every part of the +package, and the point of these fixtures is to stay byte-close to a real +TOPdesk export. + +Usage: + + python3 build-fixtures.py --source + Sanitise an already anonymised export into topdesk-export-anonymised.xlsx + (strip document metadata, custom properties, customXml, the workbook's + absolute path and xl/connections.xml), write the placeholder cached + values into the CMDB sheets, then derive the variants. + + python3 build-fixtures.py + Write the placeholder cached values into the committed + topdesk-export-anonymised.xlsx (idempotent) and derive the variants. + +The import reads the two CMDB sheets ("Onbeh Applicaties CMDB", +"Beheerde Applicaties CMDB"). Their cells are formulas that read the "Invoer" +sheets; the import reads the value Excel cached for each formula and never +evaluates one. The anonymised export had empty cached values for several +mapped columns, so CACHED_VALUES below writes a placeholder cached value into +those formula cells (the formula itself is kept). The "Invoer" sheets are not +read and are left as they are. + +Never run this on a municipality's original export: the source must already +carry placeholder values only. tests/Unit/Fixtures/CmdbFixtureHygieneTest.php +fails when a fixture holds metadata or person data that is not a placeholder. +""" + +import argparse +import os +import re +import sys +import zipfile + +HERE = os.path.dirname(os.path.abspath(__file__)) +SANITISED = os.path.join(HERE, 'topdesk-export-anonymised.xlsx') + +# Parts that never belong in a fixture. +DROP_PARTS = ('docProps/custom.xml', 'xl/connections.xml') +DROP_PREFIXES = ('customXml/',) + +ONBEH_SHEET = 'xl/worksheets/sheet2.xml' # "Onbeh Applicaties CMDB" (from AIA) +BEHEERDE_SHEET = 'xl/worksheets/sheet4.xml' # "Beheerde Applicaties CMDB" (from APP) + +# Placeholder cached values for formula cells of the CMDB sheets, per sheet and +# cell. A cell that does not exist yet is appended to its row (columns are in +# order: every cell named here lies right of the row's last cell). +CACHED_VALUES = { + ONBEH_SHEET: { + 'E2': 'Mailen', # Roepnaam + 'AE2': 'Herbeoordeling', # Rappelreden + 'AH2': 'Ja', # Locatie BIOToets + 'AI2': 'Geen', # Software Suite + }, + BEHEERDE_SHEET: { + 'E2': 'Naamtest', # Roepnaam + 'I2': 'Saas', # Applicatiesoort + 'AB2': 'Ja', # Cloud (IF(Applicatiesoort="Saas","Ja","Nee")) + 'L2': 'Teamleider Applicatiebeheer', # Applicatie Eigenaar (Functie) + 'M2': 'Teamleider Applicatiebeheer', # Applicatie Eigenaar (Persoon): no Eigenaar, so the function + 'AF2': 'Herbeoordeling', # Rappelreden + 'AH2': 'BBN2', # BNN Classificatie + 'AI2': 'Ja', # Locatie BIOToets + 'AM2': 'NT123', # Nickname (no cell in the export; appended) + }, +} + + +def read_package(path): + """Return the package as an ordered list of (name, bytes).""" + with zipfile.ZipFile(path) as package: + return [(info.filename, package.read(info.filename)) for info in package.infolist()] + + +def write_package(path, parts): + """Write (name, bytes) parts as a deflated zip, [Content_Types].xml first.""" + parts = sorted(parts, key=lambda item: item[0] != '[Content_Types].xml') + with zipfile.ZipFile(path, 'w', zipfile.ZIP_DEFLATED) as package: + for name, data in parts: + info = zipfile.ZipInfo(name, date_time=(2026, 1, 1, 0, 0, 0)) + info.compress_type = zipfile.ZIP_DEFLATED + package.writestr(info, data) + + +def text(data): + return data.decode('utf-8') + + +def sanitise(parts): + """Remove metadata parts and every reference to them.""" + kept = [] + for name, data in parts: + if name in DROP_PARTS or name.startswith(DROP_PREFIXES): + continue + if name == 'docProps/core.xml': + xml = text(data) + xml = re.sub(r'.*?', '', xml) + xml = re.sub(r'.*?', '', xml) + data = xml.encode('utf-8') + elif name == 'xl/workbook.xml': + # The absolute path of the last save names a user profile directory. + xml = re.sub(r']*>.*?x15ac:absPath.*?', '', text(data)) + data = xml.encode('utf-8') + elif name == '[Content_Types].xml': + xml = text(data) + xml = re.sub(r']*/>', '', xml) + xml = re.sub(r']*/>', '', xml) + xml = re.sub(r']*/>', '', xml) + data = xml.encode('utf-8') + elif name == '_rels/.rels': + xml = re.sub(r']*Target="docProps/custom\.xml"[^>]*/>', '', text(data)) + xml = re.sub(r']*Target="\.\./customXml/[^"]*"[^>]*/>', '', xml) + data = xml.encode('utf-8') + elif name == 'xl/_rels/workbook.xml.rels': + xml = re.sub(r']*Target="connections\.xml"[^>]*/>', '', text(data)) + xml = re.sub(r']*Target="\.\./customXml/[^"]*"[^>]*/>', '', xml) + data = xml.encode('utf-8') + kept.append((name, data)) + return kept + + +def replace_part(parts, name, transform): + return [(n, transform(d) if n == name else d) for n, d in parts] + + +def xml_escape(value): + return value.replace('&', '&').replace('<', '<').replace('>', '>') + + +def set_cached_value(xml, ref, value): + """Give the formula cell `ref` the cached string `value`; append a plain string cell when it is missing.""" + cell = re.search(r']*?)(?:/>|>(.*?))' % ref, xml, flags=re.S) + cached = '%s' % xml_escape(value) + if cell is None: + row = re.match(r'[A-Z]+(\d+)$', ref).group(1) + row_match = re.search(r'(]*>.*?)()' % row, xml, flags=re.S) + if row_match is None: + sys.exit('cached values: row %s not found' % row) + new_cell = '%s' % (ref, xml_escape(value)) + return xml[:row_match.end(1)] + new_cell + xml[row_match.end(1):] + attrs, body = cell.group(1), cell.group(2) or '' + formula = re.search(r']*>.*?|]*/>', body, flags=re.S) + if formula is None: + # Not a formula (an earlier run may have written it): only the value changes. + if ' t="inlineStr"' in attrs: + return xml[:cell.start()] + '%s' % (ref, attrs, xml_escape(value)) + xml[cell.end():] + sys.exit('cached values: %s holds no formula' % ref) + attrs = re.sub(r' t="\w+"', '', attrs) + ' t="str"' + return xml[:cell.start()] + '%s%s' % (ref, attrs, formula.group(0), cached) + xml[cell.end():] + + +def cache_values(parts): + """Write CACHED_VALUES into the CMDB sheets; running it twice changes nothing.""" + for sheet, cells in CACHED_VALUES.items(): + def transform(data, cells=cells): + xml = text(data) + for ref, value in cells.items(): + xml = set_cached_value(xml, ref, value) + return xml.encode('utf-8') + parts = replace_part(parts, sheet, transform) + # Excel recalculates from calcChain on open; the cached values are what matter here. + return parts + + +def missing_appid(parts): + """'Beheerde Applicaties CMDB' loses its APPID header (the column gets another name).""" + def transform(data): + xml = text(data) + new, count = re.subn( + r']*?) t="s"([^>]*)>\d+', + r'Applicatienummer', + xml, + count=1, + ) + if count != 1: + sys.exit('missing-appid: header cell B1 not found on Beheerde Applicaties CMDB') + return new.encode('utf-8') + return replace_part(parts, BEHEERDE_SHEET, transform) + + +def col_to_index(col): + index = 0 + for char in col: + index = index * 26 + (ord(char) - 64) + return index + + +def index_to_col(index): + col = '' + while index > 0: + index, rem = divmod(index - 1, 26) + col = chr(65 + rem) + col + return col + + +def reverse_columns(xml): + """Mirror the column order of a worksheet: the last column becomes A.""" + dim = re.search(r'', xml) + width = col_to_index(dim.group(1)) + # Elements that address columns or ranges; they are not needed by the reader. + for tag in ('cols', 'hyperlinks', 'autoFilter', 'conditionalFormatting', 'dataValidations', 'mergeCells'): + xml = re.sub(r'<%s[ >].*?' % (tag, tag), '', xml, flags=re.S) + xml = re.sub(r'<%s [^>]*/>' % tag, '', xml) + xml = re.sub(r']*/>', '', xml) + + def flip_row(match): + row_open, body = match.group(1), match.group(2) + row_open = re.sub(r' spans="[^"]*"', '', row_open) + cells = re.findall(r']*?(?:/>|>.*?)', body, flags=re.S) + flipped = [] + for cell in cells: + ref = re.match(r'' + + return re.sub(r'(]*>)(.*?)', flip_row, xml, flags=re.S) + + +def shuffled_columns(parts): + """Both CMDB sheets with their columns reversed ("Applicatie Naam" before "APPID").""" + parts = replace_part(parts, ONBEH_SHEET, lambda d: reverse_columns(text(d)).encode('utf-8')) + parts = replace_part(parts, BEHEERDE_SHEET, lambda d: reverse_columns(text(d)).encode('utf-8')) + + # A referenced header with the decoration TOPdesk adds to computed fields. + def decorate(data): + xml = text(data) + xml, count = re.subn(r'Vendor', 'Vendor⚡', xml, count=1) + if count != 1: + sys.exit('shuffled-columns: shared string "Vendor" not found') + return xml.encode('utf-8') + return replace_part(parts, 'xl/sharedStrings.xml', decorate) + + +SYNTHETIC_CONNECTION = ( + '\n' + '' + '' + '' + '' +) + + +def formula_and_connection(parts): + """On "Beheerde Applicaties CMDB": a formula in "Applicatie Naam" whose cached value + differs from its result, a "Roepnaam" formula without any cached value, plus an + external connection.""" + def formula(data): + xml = text(data) + new, count = re.subn( + r']*?) t="str"([^>]*)>[^<]*[^<]*', + r'"Evaluated"Rekenmodel', + xml, + count=1, + ) + if count != 1: + sys.exit('formula-and-connection: formula cell D2 not found on Beheerde Applicaties CMDB') + new, count = re.subn( + r']*?)>([^<]*)[^<]*', + r'\2', + new, + count=1, + ) + if count != 1: + sys.exit('formula-and-connection: formula cell E2 not found on Beheerde Applicaties CMDB') + return new.encode('utf-8') + + parts = replace_part(parts, BEHEERDE_SHEET, formula) + parts = replace_part( + parts, + '[Content_Types].xml', + lambda d: text(d).replace( + '', + '', + ).encode('utf-8'), + ) + parts = replace_part( + parts, + 'xl/_rels/workbook.xml.rels', + lambda d: text(d).replace( + '', + '', + ).encode('utf-8'), + ) + return parts + [('xl/connections.xml', SYNTHETIC_CONNECTION.encode('utf-8'))] + + +def no_source_sheet(): + """A minimal workbook with one sheet "Blad1" and neither CMDB sheet.""" + main = 'http://schemas.openxmlformats.org/spreadsheetml/2006/main' + rel = 'http://schemas.openxmlformats.org/officeDocument/2006/relationships' + pkg = 'http://schemas.openxmlformats.org/package/2006/relationships' + return [ + ('[Content_Types].xml', ( + '\n' + '' + '' + '' + '' + '' + '').encode('utf-8')), + ('_rels/.rels', ( + '\n' + '' + % (pkg, rel)).encode('utf-8')), + ('xl/workbook.xml', ( + '\n' + '' + % (main, rel)).encode('utf-8')), + ('xl/_rels/workbook.xml.rels', ( + '\n' + '' + % (pkg, rel)).encode('utf-8')), + ('xl/worksheets/sheet1.xml', ( + '\n' + 'Naam' + 'Voorbeeld' + % main).encode('utf-8')), + ] + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument('--source', help='anonymised export to sanitise into topdesk-export-anonymised.xlsx') + args = parser.parse_args() + + source = args.source or SANITISED + parts = read_package(source) + if args.source: + parts = sanitise(parts) + write_package(SANITISED, cache_values(parts)) + print('wrote', os.path.relpath(SANITISED, HERE)) + + base = read_package(SANITISED) + variants = { + 'topdesk-missing-appid.xlsx': missing_appid(base), + 'topdesk-shuffled-columns.xlsx': shuffled_columns(base), + 'topdesk-formula-and-connection.xlsx': formula_and_connection(base), + 'topdesk-no-source-sheet.xlsx': no_source_sheet(), + } + for name, parts in variants.items(): + write_package(os.path.join(HERE, name), parts) + print('wrote', name) + + +if __name__ == '__main__': + main() diff --git a/tests/fixtures/cmdb/topdesk-export-anonymised.xlsx b/tests/fixtures/cmdb/topdesk-export-anonymised.xlsx new file mode 100644 index 000000000..d8aac0535 Binary files /dev/null and b/tests/fixtures/cmdb/topdesk-export-anonymised.xlsx differ diff --git a/tests/fixtures/cmdb/topdesk-formula-and-connection.xlsx b/tests/fixtures/cmdb/topdesk-formula-and-connection.xlsx new file mode 100644 index 000000000..6189ace83 Binary files /dev/null and b/tests/fixtures/cmdb/topdesk-formula-and-connection.xlsx differ diff --git a/tests/fixtures/cmdb/topdesk-missing-appid.xlsx b/tests/fixtures/cmdb/topdesk-missing-appid.xlsx new file mode 100644 index 000000000..a88486e5c Binary files /dev/null and b/tests/fixtures/cmdb/topdesk-missing-appid.xlsx differ diff --git a/tests/fixtures/cmdb/topdesk-no-source-sheet.xlsx b/tests/fixtures/cmdb/topdesk-no-source-sheet.xlsx new file mode 100644 index 000000000..fb7b2fe99 Binary files /dev/null and b/tests/fixtures/cmdb/topdesk-no-source-sheet.xlsx differ diff --git a/tests/fixtures/cmdb/topdesk-shuffled-columns.xlsx b/tests/fixtures/cmdb/topdesk-shuffled-columns.xlsx new file mode 100644 index 000000000..eb0acb6db Binary files /dev/null and b/tests/fixtures/cmdb/topdesk-shuffled-columns.xlsx differ