fix(cmdb-import): map the CMDB export as the file has it

docs/features/cmdb-import.md

@@ -117,9 +117,10 @@ External data connections, Power Query queries and links in the workbook are
 never opened. Macro-enabled workbooks (`.xlsm`), old Excel files (`.xls`) and
 CSV files are not accepted.
 
-**Placeholder values.** The CMDB sheets fill some empty cells with a
-placeholder. These are read as empty: `NB` in `BNN Classificatie`, and the
-date 2036-01-01 (Excel serial 49675) in `End-of-Life Functioneel`.
+**Placeholder values.** The CMDB sheets fill an empty `BNN Classificatie`
+with `NB` ("niet bekend"); that is read as empty. Dates are stored as the
+file has them: the CMDB's 2036-01-01 in `End-of-Life Functioneel` is imported
+as 2036-01-01.
 
 ### Columns and where they go
 
@@ -130,14 +131,14 @@ date 2036-01-01 (Excel serial 49675) in `End-of-Life Functioneel`.
 | Applicatie Code | module external id | reference only; it can change in TOPdesk, so it is not the match key |
 | Roepnaam, Nickname | module short description | Roepnaam when filled, otherwise Nickname (only on `Beheerde Applicaties CMDB`) |
 | Functionele Omschrijving | module long description | |
-| Applicatiesoort | module hosting model (`cloudDienstverleningsmodel`) | `Saas` → SaaS, `PaaS` → PaaS, `IaaS` → IaaS, `On-premise(s)` → On-premises (self-managed); another value is dropped with a warning |
-| BNN Classificatie | module BBN level | `BBN1`/`BBN 1`/`BNN1` etc. become `BBN1`, `BBN2`, `BBN3`; `NB` is empty; another value is dropped with a warning |
+| Applicatiesoort | module application type (`applicationType`) and hosting model (`cloudDienstverleningsmodel`) | stored as is as the application type (Webapplicatie, Client/server, Saas, …); `Saas` → SaaS, `PaaS` → PaaS, `IaaS` → IaaS, `On-premise(s)` → On-premises (self-managed) also set the hosting model, any other kind leaves it empty |
+| BNN Classificatie | module BBN level | `1`, `2`, `2+`, `3` (and `BBN1`/`BBN 1`/`BNN1` etc.) become `BBN1`, `BBN2`, `BBN2+`, `BBN3`; `NB` is empty; another value is dropped with a warning |
 | Datum | module external creation date | Excel date |
 | Referentie datum wijziging | module external modification date | Excel date |
 | Vendor | Supplier organisation, set as provider on the module and the usage | one organisation per name, see below |
-| Applicatie Status | usage status | In productie → In production, In voorraad → Planned, In ontwikkeling → Acquisition, Uit te faseren → To be phased out, Uitgefaseerd → Phased out; another value is dropped with a warning |
-| Classificatie | usage TIME classification | Tolereren/Tolerate, Investeren/Invest, Migreren/Migrate, Elimineren/Eliminate |
-| End-of-Life Functioneel | usage phase-out date | Excel date; 2036-01-01 is empty |
+| Applicatie Status | usage status | In productie → In production, In voorraad → Planned, In ontwikkeling → Acquisition, Uit te faseren and Moet verwijderd worden → To be phased out, Uitgefaseerd and Verwijderd → Phased out, Besteld and Wordt getest → Acquisition, Stand-by voor continuïteit → In production; another value is dropped with a warning |
+| Classificatie | usage TIME classification | Tolereren/Tolerate (also `1. Tolereren (wordt ingelezen)`), Investeren/Invest, Migreren/Migrate, Elimineren/Eliminate |
+| End-of-Life Functioneel | usage phase-out date | Excel date, stored as is |
 | (the sheet), Cluster, Applicatie Eigenaar (Afdeling) | usage internal annotation | `Beheer geregeld: ja` or `nee`, the cluster and the department, joined with ` / `; written only when the usage is new or the note is empty |
 | Applicatie Eigenaar (Persoon), Applicatie Eigenaar (Functie) | usage business owner (contact person) | see [Owners](#owners) |
 

lib/Service/CmdbExportImportService.php

@@ -624,7 +624,10 @@ private function map(string $target, array $values, int $rowNumber, array &$warn
 			}
 		}
 
-		$data = ($result['data'] ?? []);
+		// A lookup whose default is null maps a known "no value" (an Applicatiesoort
+		// that is not a hosting model) to null; that field is left out, as an empty
+		// cell is, so an update never blanks what the field already holds.
+		$data = array_filter(($result['data'] ?? []), static fn ($value): bool => $value !== null);
 		unset($data['id']);
 
 		return ['data' => $data, 'missing' => array_values(array_unique($missing))];

lib/Settings/cmdb-import/topdesk-module.json

@@ -1,7 +1,7 @@
 {
   "id": "stackiq-topdesk-module",
   "name": "TOPdesk CMDB export to stackiq module",
-  "description": "One application row of a CMDB sheet becomes a stackiq module. A mapping marked required that fails skips the row; any other failing mapping drops that field with a warning. Nickname and Roepnaam both feed shortDescription; Roepnaam wins when both are filled.",
+  "description": "One application row of a CMDB sheet becomes a stackiq module. A mapping marked required that fails skips the row; any other failing mapping drops that field with a warning. Nickname and Roepnaam both feed shortDescription; Roepnaam wins when both are filled. Applicatiesoort is kept as is in applicationType; only its SaaS/PaaS/IaaS/on-premises values also set the hosting model, every other kind leaves that empty without a warning.",
   "sourceFormat": "excel",
   "version": "2.0.0",
   "fieldMappings": [
@@ -11,6 +11,7 @@
     { "source": "Nickname", "target": "shortDescription", "transform": { "type": "trim" } },
     { "source": "Roepnaam", "target": "shortDescription", "transform": { "type": "trim" } },
     { "source": "Functionele Omschrijving", "target": "longDescription", "transform": { "type": "trim" } },
+    { "source": "Applicatiesoort", "target": "applicationType", "transform": { "type": "trim" } },
     {
       "source": "Applicatiesoort",
       "target": "cloudDienstverleningsmodel",
@@ -21,7 +22,8 @@
           "Paas": ["PaaS"], "PaaS": ["PaaS"], "PAAS": ["PaaS"],
           "Iaas": ["IaaS"], "IaaS": ["IaaS"], "IAAS": ["IaaS"],
           "On-premise": ["On-premises (self-managed)"], "On-premises": ["On-premises (self-managed)"], "On premise": ["On-premises (self-managed)"]
-        }
+        },
+        "default": null
       }
     },
     {
@@ -32,7 +34,9 @@
         "map": {
           "BBN1": "BBN1", "BBN 1": "BBN1", "bbn1": "BBN1", "bbn 1": "BBN1", "BNN1": "BBN1", "BNN 1": "BBN1",
           "BBN2": "BBN2", "BBN 2": "BBN2", "bbn2": "BBN2", "bbn 2": "BBN2", "BNN2": "BBN2", "BNN 2": "BBN2",
-          "BBN3": "BBN3", "BBN 3": "BBN3", "bbn3": "BBN3", "bbn 3": "BBN3", "BNN3": "BBN3", "BNN 3": "BBN3"
+          "BBN3": "BBN3", "BBN 3": "BBN3", "bbn3": "BBN3", "bbn 3": "BBN3", "BNN3": "BBN3", "BNN 3": "BBN3",
+          "1": "BBN1", "2": "BBN2", "2+": "BBN2+", "3": "BBN3",
+          "BBN2+": "BBN2+", "BBN 2+": "BBN2+", "BNN2+": "BBN2+", "BNN 2+": "BBN2+"
         }
       }
     },

lib/Settings/cmdb-import/topdesk-profile.json

@@ -22,8 +22,7 @@
   "dateColumns": ["Datum", "Referentie datum wijziging", "End-of-Life Functioneel"],
   "idColumns": ["APPID"],
   "emptyValues": {
-    "BNN Classificatie": ["NB"],
-    "End-of-Life Functioneel": ["49675"]
+    "BNN Classificatie": ["NB"]
   },
   "externalKeyPrefix": "topdesk",
   "packs": {

lib/Settings/cmdb-import/topdesk-usage.json

@@ -15,7 +15,12 @@
           "In voorraad": "Planned",
           "In ontwikkeling": "Acquisition",
           "Uit te faseren": "To be phased out",
-          "Uitgefaseerd": "Phased out"
+          "Uitgefaseerd": "Phased out",
+          "Moet verwijderd worden": "To be phased out",
+          "Verwijderd": "Phased out",
+          "Besteld": "Acquisition",
+          "Wordt getest": "Acquisition",
+          "Stand-by voor continuïteit": "In production"
         }
       }
     },
@@ -28,7 +33,8 @@
           "Tolerate": "Tolerate", "Tolereren": "Tolerate",
           "Invest": "Invest", "Investeren": "Invest",
           "Migrate": "Migrate", "Migreren": "Migrate",
-          "Eliminate": "Eliminate", "Elimineren": "Eliminate"
+          "Eliminate": "Eliminate", "Elimineren": "Eliminate",
+          "1. Tolereren (wordt ingelezen)": "Tolerate"
         }
       }
     },

lib/Settings/register.d/topdesk-cmdb-import.json

@@ -2,7 +2,7 @@
   "components": {
     "schemas": {
       "module": {
-        "version": "0.3.5",
+        "version": "0.3.7",
         "properties": {
           "externalId": {
             "type": "string",
@@ -51,6 +51,18 @@
             "visible": true,
             "facetable": false,
             "order": 64
+          },
+          "applicationType": {
+            "type": "string",
+            "title": "Application type",
+            "description": "The kind of application as the source system records it, such as the TOPdesk Applicatiesoort (Webapplicatie, Client/server, Saas). Kept as the source has it.",
+            "maxLength": 100,
+            "visible": true,
+            "facetable": true,
+            "order": 65
+          },
+          "bbnLevel": {
+            "enum": ["BBN2+"]
           }
         }
       }

openspec/changes/cmdb-export-import/design.md

@@ -107,7 +107,7 @@ External connections, the Power Query package and hyperlinks are never resolved:
 
 `CmdbRowNormaliser` turns reader output into the flat `column => string` row the engine expects:
 
-- Values listed in the profile's `emptyValues` for their column become empty, compared case-insensitively before any conversion: `NB` in "BNN Classificatie" (the CMDB sheet's "niet bekend") and `49675` (2036-01-01) in "End-of-Life Functioneel" (the CMDB sheet's placeholder for "no end-of-life date"; its formula turns an empty date, or TOPdesk's 2099-12-31, into 49675).
+- Values listed in the profile's `emptyValues` for their column become empty, compared case-insensitively before any conversion: `NB` in "BNN Classificatie" (the CMDB sheet's "niet bekend"). Dates are stored as the file has them: "End-of-Life Functioneel" `49675` (2036-01-01) is imported as that date, even though the CMDB sheet's formula writes it for an empty date or TOPdesk's 2099-12-31; the municipality decided to keep the file's value (2026-10-02).
 - Columns listed in the profile's `dateColumns` ("Datum", "Referentie datum wijziging", "End-of-Life Functioneel"): a numeric value is converted with `PhpOffice\PhpSpreadsheet\Shared\Date::excelToDateTimeObject()` in UTC and written as `Y-m-d`. For example, `45111.38…` becomes `2023-07-04` and `53359` becomes `2046-02-01`. A non-numeric value stays as it is, so the pack's `date` transform (`sourceFormat: Y-m-d`) either accepts it or reports a warning.
 - Columns listed in `idColumns` ("APPID"): a whole number becomes a string without a decimal part (`1234.0` becomes `"1234"`).
 - The constants of the row's sheet are added before mapping (`Beheer` = `Beheer geregeld: nee` on "Onbeh", `ja` on "Beheerde"), so the usage pack can map the sheet like a column.
@@ -206,14 +206,14 @@ Source columns of "Onbeh Applicaties CMDB" and "Beheerde Applicaties CMDB" and w
 | Roepnaam | both | module.shortDescription | trim; wins over Nickname |
 | Nickname | Beheerde | module.shortDescription | trim; used when Roepnaam is empty; listed as absent on Onbeh |
 | Functionele Omschrijving | both | module.longDescription | trim |
-| Applicatiesoort | both | module.cloudDienstverleningsmodel | lookup: `Saas`/`SaaS` → `["SaaS"]`, `PaaS` → `["PaaS"]`, `IaaS` → `["IaaS"]`, `On-premise(s)` → `["On-premises (self-managed)"]`; another value (such as `Webapplicatie`): warning, field dropped |
-| BNN Classificatie | both | module.bbnLevel | `NB` is empty; lookup "BBN1"/"BBN 1"/"BNN1" etc. to `BBN1`/`BBN2`/`BBN3`; unknown value: warning |
+| Applicatiesoort | both | module.applicationType and module.cloudDienstverleningsmodel | applicationType: as is; hosting model lookup: `Saas`/`SaaS` → `["SaaS"]`, `PaaS` → `["PaaS"]`, `IaaS` → `["IaaS"]`, `On-premise(s)` → `["On-premises (self-managed)"]`; any other kind (`Webapplicatie`, `Client/server`, …) has default `null`: no hosting model, no warning |
+| BNN Classificatie | both | module.bbnLevel | `NB` is empty; lookup `1`/`2`/`2+`/`3` and "BBN1"/"BBN 1"/"BNN1" etc. to `BBN1`/`BBN2`/`BBN2+`/`BBN3` (the fragment adds `BBN2+` to the enum); unknown value: warning |
 | Datum | both | module.externalCreatedAt | Excel serial to date |
 | Referentie datum wijziging | both | module.externalModifiedAt | Excel serial to date |
 | Vendor | both | organization (Supplier) via module.provider and usage.provider | dedup on normalised name (D7) |
-| Applicatie Status | both | usage.status | lookup: In productie → In production, In voorraad → Planned, In ontwikkeling → Acquisition, Uit te faseren → To be phased out, Uitgefaseerd → Phased out; unknown value: warning |
-| Classificatie | both | usage.timeClassification | lookup Tolereren/Tolerate, Investeren/Invest, Migreren/Migrate, Elimineren/Eliminate |
-| End-of-Life Functioneel | both | usage.startDateOutPhased | `49675` (2036-01-01) is empty; Excel serial to date |
+| Applicatie Status | both | usage.status | lookup: In productie → In production, In voorraad → Planned, In ontwikkeling → Acquisition, Uit te faseren and Moet verwijderd worden → To be phased out, Uitgefaseerd and Verwijderd → Phased out, Besteld and Wordt getest → Acquisition, Stand-by voor continuïteit → In production; unknown value: warning |
+| Classificatie | both | usage.timeClassification | lookup Tolereren/Tolerate (also `1. Tolereren (wordt ingelezen)`), Investeren/Invest, Migreren/Migrate, Elimineren/Eliminate |
+| End-of-Life Functioneel | both | usage.startDateOutPhased | Excel serial to date, stored as is (2036-01-01 included) |
 | (sheet constant `Beheer`), Cluster, Applicatie Eigenaar (Afdeling) | both | usage.interneAnnotation | concat with " / ", empty parts dropped, create-only; `Beheer geregeld: nee` (Onbeh) or `ja` (Beheerde) |
 | Applicatie Eigenaar (Persoon), Applicatie Eigenaar (Functie) | both | usage.businessOwner (contactPerson + Nextcloud contact; role = Functie) | D8; the person column may hold a function |
 | Hostingpartij | both | not mapped | follow-up; "Leverancier" (where the municipality buys the software) is not on the CMDB sheets |
@@ -414,8 +414,8 @@ The seeds show the new properties in a fresh install. They carry no `publication
 - [Owner contacts land in the importing admin's address book] → `StackiqContactSyncService` writes to the first writable address book of the acting user, the same as every other stackiq contact path. The docs say so. A dedicated system address book is a follow-up.
 - [Long synchronous request] → Per-row progress, cancel, and "unchanged" rows skip the save. About 1,100 rows is expected to fit. A background job is a follow-up if it does not.
 - [OpenRegister internals (`MappingEngine`, `PackDefinitionValidator`, PhpSpreadsheet) change shape] → Guarded resolution with 503, and a contract test that maps the fixture through the real engine in the dev environment.
-- [Provisional lookups for Applicatiesoort and BNN Classificatie] → The values of the first real import were not kept (the report lives in the progress cache for an hour). The maps hold the values the anonymised export and the CMDB formulas show (`Saas`, `Webapplicatie`, `NB`) plus the usual spellings. An unknown value is a warning, never a wrong value; once the municipality lists its values, the maps in the JSON are extended, with no code change.
-- [CMDB placeholders] → "End-of-Life Functioneel" `2036-01-01` and "BNN Classificatie" `NB` are read as empty. A real end-of-life date of exactly 2036-01-01 would be lost; the municipality confirms. "Classificatie" defaults to `Tolereren` on "Beheerde" when TOPdesk has none; that cannot be told apart from a real `Tolereren` and is imported as Tolerate.
+- [Lookups from the real export] → The maps hold the values the municipality's export of 2026-09-22 contains (2026-10-02 import report): "Applicatiesoort" is an application kind, kept as is in `applicationType`; "BNN Classificatie" holds `NB`, `1`, `2` and `2+`; "Applicatie Status" adds five Dutch statuses; "Classificatie" one numbered form. A new value is a warning, never a wrong value, and is added to the JSON map with no code change. A lookup `default` of `null` means "known, no value": the service leaves the field out.
+- [CMDB placeholders] → "BNN Classificatie" `NB` is read as empty. "End-of-Life Functioneel" and "Classificatie" are stored as the file has them: the sheet's 2036-01-01 (written for an empty date) is imported as that date, and the `Tolereren` the "Beheerde" formula writes when TOPdesk has none is imported as Tolerate.
 - [An application moves between the sheets] → Same APPID, so the same module and usage; the usage note (create-only) keeps its old `Beheer geregeld` line when it is not empty.
 - [An unknown status on create falls back to the usage schema's default "In production"] → Accepted. The warning in the report makes it visible.
 - [The fragment version is overwritten by merge order] → Filename ordering plus a unit test on the merged version (Mixed-spec rationale).
@@ -426,7 +426,5 @@ No data migration. The register fragment deploys with the existing repair-step r
 
 ## Open Questions
 
-- Which "Applicatiesoort" and "BNN Classificatie" values occur in the municipality's real export, and which hosting model does each mean (lookup maps)?
-- Is 2036-01-01 in "End-of-Life Functioneel" always the placeholder, and should a "Beheerde" row without a TIME class really be Tolerate?
 - Should the maintenance status update the usage note on re-import (it is create-only today), or get a field of its own?
 - Should OpenRegister promote `MigrationPack\MappingEngine` to its `Contract` namespace?