From ecaba9cdfe3d66169e0a32f5166c980cade258f2 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:20:37 +0200 Subject: [PATCH 01/54] Added ADR folder along with ADRs existing so far --- docs/adr/001-architecture-symfony-docker.md | 87 ++++++++++++ docs/adr/002-publish-to-a-context-broker.md | 104 +++++++++++++++ docs/adr/003-ngsi-ld-representation.md | 70 ++++++++++ docs/adr/004-coordinate-reference-system.md | 112 ++++++++++++++++ .../005-smart-data-models-as-vocabulary.md | 126 ++++++++++++++++++ .../006-onstreetparking-over-parkinggroup.md | 104 +++++++++++++++ docs/adr/CLAUDE.md | 78 +++++++++++ docs/adr/README.md | 20 +++ 8 files changed, 701 insertions(+) create mode 100644 docs/adr/001-architecture-symfony-docker.md create mode 100644 docs/adr/002-publish-to-a-context-broker.md create mode 100644 docs/adr/003-ngsi-ld-representation.md create mode 100644 docs/adr/004-coordinate-reference-system.md create mode 100644 docs/adr/005-smart-data-models-as-vocabulary.md create mode 100644 docs/adr/006-onstreetparking-over-parkinggroup.md create mode 100644 docs/adr/CLAUDE.md create mode 100644 docs/adr/README.md diff --git a/docs/adr/001-architecture-symfony-docker.md b/docs/adr/001-architecture-symfony-docker.md new file mode 100644 index 0000000..c0f8708 --- /dev/null +++ b/docs/adr/001-architecture-symfony-docker.md @@ -0,0 +1,87 @@ +# 001: Architecture — Symfony 8 on the ITK Dev Docker template + +| Field | Value | +|--------------------|----------------------------------------| +| **Created By** | Jeppe Krogh | +| **Date** | 2026-08-24 | +| **Decision Maker** | ITK Dev team | +| **Stakeholders** | ITK Dev developers, future maintainers | +| **Status** | Draft | + +## Context + +The adapter reads open data sets, converts them to a standard smart-city +representation, and publishes them to a context broker. It needs a runtime, an +HTTP client, a console for running imports, and a local development environment +including a broker to import into. It has no web UI and no domain data of its +own. + +ITK Dev maintains a fleet of PHP services with an established Docker-based +development convention, expressed as versioned project templates with shared CI +and coding-standards configuration. A new application either adopts that or +diverges from it. + +This ADR serves to decide the runtime, framework and development +environment the application is built on. + +### Drivers + +- **Functional:** scheduled console commands; outbound HTTP; a local broker. + No database and no HTTP surface of its own. +- **Non-functional:** minimal onboarding cost; shared tooling rather than + reimplemented tooling; reproducible across developers and CI; long-term + vendor support. + +### Options Considered + +1. **PHP 8.4 / Symfony 8 on the ITK Dev `symfony-8` template.** Matches the + organisation's existing stack, so CI, coding standards and task runner come + for free; the console component suits scheduled imports. Provisions a web + server, database and mail catcher this application never uses, and its PHP + version runs ahead of developer hosts, making containers mandatory. +2. **Minimal framework project without the template, run on the host.** No + unused services, no container requirement for the application — but shared + CI and coding-standards config would be reimplemented by hand, and a local + broker needs containers anyway, so the dependency is moved rather than + removed. +3. **A second entry point in an existing internal application.** One + deployment to operate, but couples a batch importer's release cycle to a + user-facing application and inherits dependencies it has no use for. +4. **A different language ecosystem on a bespoke setup.** Richer geospatial + libraries in some ecosystems, but no internal expertise and no shared + tooling. The transformations needed are available as mature libraries in the + established stack too. + +## Decision + +**PHP 8.4 + Symfony 8** on the ITK Dev `symfony-8` template, as its **own +deployable service**, with a containerised broker overlay for local development. + +- Standardising costs less over the application's lifetime than trimming unused + services. A second toolchain must be learned and patched; idle containers + cost only disk. +- A batch importer and a user-facing application have different lifecycles and + failure modes, so they stay separate services. +- No domain persistence is needed — the broker is the system of record — so the + template's database service is left unused rather than removed, keeping + template updates a clean diff. +- Local development includes a real broker, so imports are verified end to end + rather than only as serialised output. + +## Consequences + +### Positive + +- Onboarding cost close to zero; CI and coding standards work from the first + commit. +- No schema, no migrations, no state to keep consistent with the broker. + +### Negative / Trade-offs + +- **Containers are mandatory.** The template's PHP runs ahead of developer + hosts, so dependency management, console commands and tests cannot run + natively. Most likely source of first-run confusion. +- A web server, database and mail catcher are provisioned and never used. +- Broker images are not published for every CPU architecture, so local start-up + may be slow under emulation. +- The application follows the template's choices; deviating later has a cost. diff --git a/docs/adr/002-publish-to-a-context-broker.md b/docs/adr/002-publish-to-a-context-broker.md new file mode 100644 index 0000000..7372479 --- /dev/null +++ b/docs/adr/002-publish-to-a-context-broker.md @@ -0,0 +1,104 @@ +# 002: Publication mechanism — publish to a context broker + +| Field | Value | +|--------------------|--------------------------------------------------------| +| **Created By** | Jeppe Krogh | +| **Date** | 2026-08-24 | +| **Decision Maker** | ITK Dev team | +| **Stakeholders** | ITK Dev developers, data consumers, future maintainers | +| **Status** | Draft | + +## Context + +The adapter makes data available to consumers the organisation does +not control and cannot brief. The data already exists in operational systems, +with heterogeneous formats, coordinate systems and access methods. What has to +be decided is the mechanism by which it is published. The source systems remain +authoritative; the published copy is not a system of record. + +This ADR serves to decide the mechanism by which data is published to +consumers. + +### Drivers + +- **Functional:** query by location and attribute, not bulk download only; + several data sets reaching one consumer-facing surface; change notification; + adding a data set without changing consumer integrations. +- **Non-functional:** interpretable by consumers we have never spoken to; + operational cost proportionate to the data and the number of consumers; + existing client tooling rather than clients we supply; an interface that + outlives individual data sets. + +### Options Considered + +1. **An NGSI-LD context broker.** Provides geospatial and attribute queries, + pagination, subscriptions and a temporal interface without implementing + them; payloads carry a vocabulary reference, so they are self-describing; + many producers converge on one consumer surface. Substantial operational + weight — typically a database and message bus alongside the broker — and the + strictness of a particular implementation is inherited. +2. **Static file export** on a web server or object store. Near-zero + operational cost, trivially cacheable, readable by anything. No query, so + consumers download everything and filter client-side; no change + notification; conventions must be documented in prose because nothing in the + file declares its own meaning. +3. **A bespoke REST API** over our own datastore. Exact fit, full control of + the query surface and semantics. Every capability is ours to build and + maintain — geo-queries, filtering, pagination, notifications, documentation, + clients, versioning — and consumers must learn an interface that exists + nowhere else. +4. **Direct database access or a read replica.** No API layer, powerful ad-hoc + querying. Exposes internal schema as a public contract, requires per-consumer + credentials and network access, and is unusable by browser-based consumers. + +## Decision + +Publish to an **NGSI-LD context broker**. + +- Consumers of geographic data need "everything within this area" and + "everything of this kind" more often than the whole data set. A broker + provides that as a standard interface rather than a per-data-set feature. +- Publishing structure without a vocabulary reference requires every consumer + to have our documentation. A broker payload carries the reference. +- For a single small data set a static export would be cheaper and better. Once + several heterogeneous data sets must be published, the fixed operational cost + is paid once while the per-data-set cost approaches zero, and consumers + integrate once rather than once per source. +- New consumers require no change to the adapter, and new data sets require no + change to consumers. +- Existing viewers, dashboards and connectors speak this interface; a bespoke + API would mean supplying clients indefinitely. + +The broker's value is interoperability and query, not storage. If no consumer +reads the data through its interface, a static export would have been the better +decision. Revisit once data sets have been published long enough for consumers +to appear. + +## Consequences + +### Positive + +- Geospatial and attribute queries, pagination, subscriptions and a temporal + interface, none of which are implemented here. +- Payloads reference a shared vocabulary, so they need no bespoke + documentation. +- Additional data sets reach every existing consumer with no integration work. +- The adapter has no database and no read surface of its own. + +### Negative / Trade-offs + +- **Operational weight out of proportion to a small data set.** A broker + deployment is several services to run, patch, monitor and back up. For a + static data set a file on a web server would serve the same need. +- **Broker implementations impose constraints beyond the standard.** Those + encountered include accepting only one spelling of a UTC timestamp while + rejecting an equivalent one, requiring the vocabulary reference on read + requests — with omission returning an empty success rather than an error — + and collapsing single-element lists to scalars. +- **Vocabulary documents may be fetched over the network during writes**, so + third-party availability becomes part of the import path. +- **No delete semantics.** Upsert creates and updates but never removes, so + records that disappear upstream persist until reconciliation is built. +- JSON-LD is a learning curve for maintainers and consumers. +- Fitting data to a shared vocabulary costs effort that publishing as-is + would not. diff --git a/docs/adr/003-ngsi-ld-representation.md b/docs/adr/003-ngsi-ld-representation.md new file mode 100644 index 0000000..8adcb7e --- /dev/null +++ b/docs/adr/003-ngsi-ld-representation.md @@ -0,0 +1,70 @@ +# 003: NGSI-LD representation — normalized form and batch upsert + +| Field | Value | +|--------------------|--------------------------------------------------------| +| **Created By** | Jeppe Krogh | +| **Date** | 2026-08-24 | +| **Decision Maker** | ITK Dev team | +| **Stakeholders** | ITK Dev developers, data consumers, future maintainers | +| **Status** | Draft | + +## Context + +ADR 002 chooses an NGSI-LD context broker. That leaves two representation +choices open: how attributes are shaped, and which write operation is used. + +NGSI v2, the older FIWARE API generation, was not considered viable: it has no +`@context`, so a shared vocabulary cannot be expressed, and the +organisation operates no v2 broker. + +This ADR serves to decide how attributes are shaped and which write +operation is used. + +### Drivers + +- **Functional:** the broker must accept the payload on write; attribute-level + metadata must be expressible; repeated imports must not duplicate entities. +- **Non-functional:** idempotency; payload size; readability for consumers. + +### Options Considered + +#### Attribute form + +1. **Normalized** — every attribute an object naming its kind. Verbose, but + the form brokers accept on write and the only one carrying metadata such as + `observedAt`. +2. **Key-values** — flat `name: value`. Much smaller and easier to read, but + read-only and cannot carry attribute metadata. + +#### Write operation + +1. **Batch upsert** — creates or updates. Idempotent when identifiers are + derived from source keys. +2. **Create** — fails with `409` for identifiers that already exist, so a + re-import errors rather than refreshing. +3. **Batch replace** — silently drops attributes absent from the payload, + making partial payloads destructive. + +## Decision + +Publish **normalized** NGSI-LD with `Content-Type: application/ld+json`, via +**batch upsert**. + +- Normalized is the only form accepted on write, so key-values is not a real + option for a producer. +- Upsert makes imports idempotent: identifiers derive from each source's + primary key, so re-running updates in place. + +## Consequences + +### Positive + +- Re-imports produce no duplicates and need no prior state. +- Attribute metadata remains available if a source ever supplies it. + +### Negative / Trade-offs + +- Payloads are considerably larger than key-values. +- Upsert never deletes, so records removed upstream persist until + reconciliation is built (see ADR 002). +- Consumers unfamiliar with JSON-LD face a learning curve. diff --git a/docs/adr/004-coordinate-reference-system.md b/docs/adr/004-coordinate-reference-system.md new file mode 100644 index 0000000..177d6b7 --- /dev/null +++ b/docs/adr/004-coordinate-reference-system.md @@ -0,0 +1,112 @@ +# 004: Coordinate reference system — publish WGS84 (EPSG:4326) + +| Field | Value | +|--------------------|----------------------------------------------------------| +| **Created By** | Jeppe Krogh | +| **Date** | 2026-08-27 | +| **Decision Maker** | ITK Dev team | +| **Stakeholders** | ITK Dev developers, broker consumers, future maintainers | +| **Status** | Draft | + +## Context + +Every entity the adapter publishes carries a `location` GeoProperty, so the +coordinate reference system is a cross-cutting concern rather than a per-source +detail. + +Input data arrives in whatever CRS its publisher uses. Danish municipal data is +commonly projected — typically EPSG:25832 (ETRS89 / UTM zone 32N), as eastings +and northings in metres. Other inputs may already be geographic, or use a +different projection. The adapter cannot assume one input CRS. + +Projected coordinates are sometimes delivered inside a GeoJSON envelope, which +states a geometry type but not units. The data models specify only that +`location` is GeoJSON and make no reference to a coordinate system; the +constraint comes from GeoJSON itself. + +This ADR serves to decide which coordinate reference system is published, +and at what precision. + +### Drivers + +- **Functional:** consumers must be able to interpret `location` unbriefed; + geo-queries must return correct results; clients must render without + preprocessing; one rule must hold for every input. +- **Non-functional:** self-description; conformance; uniformity across inputs; + precision no worse than the input. + +### Options Considered + +1. **Normalise everything to WGS84, reprojecting in the adapter.** Conforms to + RFC 7946; self-describing; geo-queries work; one rule however many input + CRSs accumulate. Requires a reprojection dependency, and each input must + declare its CRS. +2. **Pass each input's native CRS through unchanged.** No transformation, no + dependency — but produces invalid GeoJSON with nowhere to declare the CRS, + makes entities from different inputs mutually incomparable, and breaks + geo-queries because distances are read as degrees. Every failure is silent. +3. **Publish WGS84 and also retain original coordinates in an extra + attribute.** Avoids a round trip for consumers wanting native coordinates, + but the attribute cannot have a stable shape: each input brings its own CRS + and geometry type, and it would be absent for inputs already in WGS84. A + consumer cannot code against that, so it would go unused. +4. **Pass native CRSs through under RFC 7946's "prior arrangement" clause, + documenting each out of band.** Permitted by the RFC, but the clause + requires all parties to have agreed — incompatible with a broker whose + consumers are unknown by design. + +## Decision + +Every `location` the adapter emits is **EPSG:4326 (WGS84) +longitude/latitude**, at **full precision — coordinates are not rounded**. Each +input declares its own CRS; reprojection happens at the boundary between reading +an input and building an entity, and nowhere else. + +- **The specification is unambiguous.** RFC 7946 §4: "The coordinate reference + system for all GeoJSON coordinates is a geographic coordinate reference + system, using the World Geodetic System 1984 (WGS 84) datum, with longitude + and latitude units of decimal degrees." NGSI-LD GeoProperty values are + GeoJSON, so the requirement is inherited. +- **There is no way to declare otherwise.** RFC 7946 Appendix B.1: + "Specification of coordinate reference systems has been removed, i.e., the + 'crs' member of [GJ2008] is no longer used." Publishing projected coordinates + means publishing an undeclarable assumption. Inputs may still carry that + deprecated member; it can be read, but not passed on. +- Entities from different inputs are queried together, so a query spanning two + inputs published in different CRSs returns meaningless results. +- A broker given projected coordinates accepts them, answers geo-queries + incorrectly, and renders points in the wrong location. No error is raised at + any stage. +- **The conversion is lossless at full precision.** A projected-to-geographic + round trip returns the input exactly when no rounding is applied. Rounding + trades accuracy for a marginal reduction in payload size. +- `source` and `seeAlso` can reference the originating export, which states its + own CRS. A coordinate copied into an extra attribute states nothing. + +## Consequences + +### Positive + +- Payloads are valid GeoJSON and NGSI-LD; geo-queries work and are comparable + across inputs; any client renders them unmodified. +- One rule for every present and future input. +- Reprojection is isolated in one component with its own tests, verified + against independently known reference coordinates, so a regression fails + loudly instead of silently relocating data. + +### Negative / Trade-offs + +- Adds a reprojection dependency. National grid definitions are not always + shipped and may need registering explicitly, making them load-bearing + project code. +- **Datum shifts are approximated.** ETRS89-based grids are treated as + equivalent to WGS84 via a null datum transformation. The two were coincident + in 1989 and have diverged by roughly 0.5–1 m since, at about 2.5 cm per year. + What is published is therefore ETRS89 labelled WGS84. This is standard + practice in web GIS, but it is the largest error in the pipeline — greater + than the source's own positional accuracy — so a consumer using a rigorous + transformation with an explicit epoch will land about a metre away. +- Consumers with natively projected stacks must convert. +- Every new input must declare its CRS, and unsupported ones need adding. +- Only point geometries were implemented initially; line and area geometries + were added later. diff --git a/docs/adr/005-smart-data-models-as-vocabulary.md b/docs/adr/005-smart-data-models-as-vocabulary.md new file mode 100644 index 0000000..42f11ec --- /dev/null +++ b/docs/adr/005-smart-data-models-as-vocabulary.md @@ -0,0 +1,126 @@ +# 005: Vocabulary — adopt Smart Data Models + +| Field | Value | +|--------------------|--------------------------------------------------------| +| **Created By** | Jeppe Krogh | +| **Date** | 2026-08-31 | +| **Decision Maker** | ITK Dev team | +| **Stakeholders** | ITK Dev developers, data consumers, future maintainers | +| **Status** | Draft | + +## Context + +NGSI-LD defines how attributes are carried and how to reference a vocabulary. +It does not define entity types or attribute names. Without a vocabulary the +JSON-LD context resolves to nothing, entity types are local strings, and +consumers still need our documentation to interpret anything. + +The choice also determines the cost of onboarding each data set, since mapping +a source onto an existing model takes more effort than exposing its fields +verbatim. + +This ADR serves to decide which vocabulary supplies entity types and +attribute names, and the rules for mapping sources onto it. + +### Drivers + +- **Functional:** types and attributes interpretable without our + documentation; expressible as a JSON-LD context a broker can resolve; + coverage across the domains in scope. +- **Non-functional:** a vocabulary consumers plausibly already know; governed + and maintained by someone else; mapping cost that does not dominate + onboarding. + +### Options Considered + +1. **Smart Data Models.** Purpose-built for NGSI-LD and the reference + vocabulary of that ecosystem; publishes JSON-LD context documents per + domain; broad coverage; each model ships a JSON schema and examples, giving + an objective conformance target; open governance. Model depth varies; + many models assume real-time sensing, so static inventory leaves attributes + unset; required attributes occasionally presuppose a hierarchy the source + lacks; enum spellings sometimes disagree between a model's schema and its + examples; versioning is loose. +2. **A vocabulary of our own, with self-hosted context documents.** Exact fit, + no required attributes we cannot satisfy, full control of naming and + versioning. Nobody else speaks it, so consumers return to reading our + documentation; governance, documentation and versioning become ours + indefinitely; no existing tooling recognises the types. +3. **A general-purpose web vocabulary.** Widely recognised, stable governance, + adequate for names, addresses and descriptions. No NGSI-LD conventions for + geometry or relationships, and no domain-specific terms, so the domains in + scope would remain unmodelled. + +## Decision + +Adopt **Smart Data Models**, referencing the relevant domain context documents +alongside the NGSI-LD core context. + +Two rules follow, and they matter more than the choice itself: + +1. **Use an existing model; do not invent a type.** An imperfect standard type + is more useful to a consumer than a perfect private one. +2. **Never fabricate a value to satisfy a model.** An absent attribute is + honest; a fabricated one is indistinguishable from a measured one. Applied: + + - Attributes the source cannot fill are left unset, not approximated, + defaulted or inferred. + - Where a model *requires* an attribute the source cannot supply, choose a + different model rather than inventing the value. Required relationships + are the common case: inventing a related entity yields something + schema-valid and factually wrong that must then be maintained + indefinitely. A sibling model without the requirement is the better + choice even if its terms are less precise. + - In a hierarchy — a root site, subdivisions beneath it, individual units + beneath those, each lower level requiring a relationship upward — publish + at the highest level the source can populate. Static inventory typically + describes a location and a count of units without describing what the + location is part of, so the site level is usually correct. + +The concrete model chosen for a given data set is recorded in its own ADR; this +one states policy only. + +Rationale: + +- The context must resolve to terms a consumer recognises, or publishing gains + nothing over a file. +- Smart Data Models is the vocabulary the surrounding ecosystem uses and ships + the context documents needed to reference it, so adoption is a URL rather + than a project. +- Shipped schemas and examples make modelling disagreements checkable against a + specification instead of settled by preference. +- Mandatory relationships propagate downward: choosing a subdivision level + schedules the need for a parent rather than avoiding it, because the + individual-unit level requires a site as well. +- The costs are asymmetric. Publishing at site level and later finding real + sites exist means a one-off migration. Publishing at subdivision level and + never acquiring real sites means maintaining an invented entity + indefinitely, with every consumer that follows the relationship receiving + something meaningless. + +## Consequences + +### Positive + +- Types and attributes resolve to shared global identifiers. +- Consumers may already have code for the types published. +- Modelling decisions have an external reference point. +- Later data sets are likely already covered, so onboarding does not start with + vocabulary design. +- Published entities are self-contained, with nothing invented to keep in sync. +- Finer granularity can be added later beneath what is already published. + +### Negative / Trade-offs + +- **Many attributes will always be empty.** Models built around real-time + sensing carry availability, occupancy and detection attributes that static + inventory cannot fill. +- **Model choice is embedded in entity identifiers.** Changing model later + means deleting and re-publishing rather than updating in place, so selection + deserves attention before a data set is first published. +- **Enum values must be read from the schema, not the examples.** Where the two + disagree the schema is authoritative, and equivalent-looking values differ + between sibling models, so they must not be copied across. +- **Loose versioning.** A model can change without an obvious signal. +- Mapping a source to a model takes longer than exposing its fields verbatim, + and occasionally the fit is poor. diff --git a/docs/adr/006-onstreetparking-over-parkinggroup.md b/docs/adr/006-onstreetparking-over-parkinggroup.md new file mode 100644 index 0000000..8b04272 --- /dev/null +++ b/docs/adr/006-onstreetparking-over-parkinggroup.md @@ -0,0 +1,104 @@ +# 006: Model selection — OnStreetParking over ParkingGroup + +| Field | Value | +|--------------------|--------------------------------------------------------| +| **Created By** | Jeppe Krogh | +| **Date** | 2026-08-31 | +| **Decision Maker** | ITK Dev team | +| **Stakeholders** | ITK Dev developers, data consumers, future maintainers | +| **Status** | Draft | + +## Context + +The parking domain is organised as a hierarchy: + +```text +OnStreetParking / OffStreetParking site — no parent, requires id, type, location + └── ParkingGroup subdivision — requires refParkingSite + └── ParkingSpot individual unit — requires refParkingSite, status, category +``` + +This decision applies where a data set provides a count of units per location +with a point geometry, no reference to a containing site, and neither per-unit +geometry nor occupancy. + +Entity identifiers embed the type by convention, so the choice must be made +before first publication: changing it afterwards means deleting and +re-publishing. + +This ADR serves to decide which model in the parking hierarchy is published +under those conditions. + +### Drivers + +- **Functional:** every mandatory relationship must point at an entity that + exists; the restriction on who may park must be expressible unambiguously; + finer granularity addable later without restructuring what is published. +- **Non-functional:** nothing invented purely to satisfy a schema; a choice + that is cheap to reverse in preference to one that is not. + +### Options Considered + +1. **`ParkingGroup`, creating the missing parent site.** `category` offers + `onlyDisabled`, which by name states exclusivity, and the model's reference + example for disabled parking sits at this level. But `refParkingSite` is + mandatory and no value is available for it, so a parent must be invented; + one spanning the whole administrative area asserts a false containment, and + its own mandatory geometry would carry no meaning. + `ParkingSpot` also requires a site, so adding per-unit data later would force + the invented entity into existence after entities had been published against + it. +2. **`ParkingGroup`, omitting `refParkingSite`.** Nothing invented, smallest + change — but knowingly non-conformant, and a schema validator flags every + entity. +3. **`OnStreetParking`.** Requires only `id`, `type` and `location`, all of + which are available. It is the entity both `ParkingGroup` and + `ParkingSpot` are required to reference, so finer granularity can be + attached beneath it, and migration down to `ParkingGroup` stays possible if + real site data appears. `category` offers only `forDisabled`, which does not + state exclusivity as plainly. +4. **`ParkingSpot`.** Models an individual unit, but only a count per location + is available, `status` is mandatory with no occupancy data, and a parent site + is required. Rejected outright. + +## Decision + +Publish each record as an **`OnStreetParking`** entity, with +`category: ["forDisabled"]` and no `refParkingSite`. + +- It is the only option that invents nothing; everything the model requires is + available. +- Mandatory relationships propagate downward, so choosing `ParkingGroup` would + schedule the invented parent rather than avoid it — `ParkingSpot` requires a + site too. +- The costs are asymmetric. Site level, then finding real sites exist, is a + one-off migration. Subdivision level, then never acquiring real sites, means + maintaining an invented entity indefinitely. + +The model's reference example does use `ParkingGroup` for disabled parking, but +points at a real street-address site. It shows what to do when a site exists, +not when none does. + +## Consequences + +### Positive + +- No dangling relationship; every entity is self-contained. +- Nothing invented to create, document or keep in sync. +- Conformant to the model's schema without exceptions. +- `ParkingGroup` or `ParkingSpot` entities can be attached beneath these later + without changing them. + +### Negative / Trade-offs + +- **`forDisabled` does not state exclusivity.** The schema documents `category` + only as "Street parking category" with an enum list and defines no individual + value. The two models' enums use inconsistent prefixes for what appear to be + the same concepts — `forDisabled` / `forResidents` against `onlyDisabled` / + `onlyResidents` — while both carry `onlyWithPermit`, so the spelling cannot + be relied on to carry exclusivity. +- Values must not be copied between the two models' `category` enums. +- Entity type is embedded in identifiers, so any later change means deleting + and re-publishing. +- Domain experts may find a single address described as a "site" + counter-intuitive. diff --git a/docs/adr/CLAUDE.md b/docs/adr/CLAUDE.md new file mode 100644 index 0000000..679a5cd --- /dev/null +++ b/docs/adr/CLAUDE.md @@ -0,0 +1,78 @@ +# CLAUDE.md — writing ADRs in this project + +Guidance for Claude Code when creating or editing files in `docs/adr/`. + +An ADR here **decides a specific choice on general grounds**. The choice is +concrete; the reasoning must hold beyond any one data set, developer or machine. + +## Never include + +- **Local project or repository names.** These are organisation-level documents; + someone's working copies have no place in them. +- **Specific data sources, their fields, or their quirks.** Not field names, not + envelope shapes, not "the source provides…". State the *condition* the + decision applies under instead: "This decision applies where a data set + provides a count of units per location…". +- **Measurements or counts taken from a data set.** No record counts, no + "measured across N records", no byte or centimetre figures derived from one + export. Standards identifiers (EPSG codes, RFC numbers, framework versions) + and general domain facts are fine. +- **Educational or self-referential framing.** No "two things make this easy to + get wrong", no "note that", no "the real lesson is", no bolded aphorism + followed by the actual fact, and no commentary on the ADR's own importance + ("this is the hardest to reverse"). State the fact and stop. + +## Structure + +```text +# NNN: Area — the choice + +| Field | Value | Created By, Date, Decision Maker, Stakeholders, Status + +## Context the situation; what is undecided + ends with: "This ADR serves to decide …" +### Drivers Functional / Non-functional, brief +### Options Considered one paragraph each, upsides then downsides +## Decision the choice, then terse rationale bullets +## Consequences +### Positive +### Negative / Trade-offs the honest costs, including ones found by hitting them +``` + +The Context section **must end with a single "This ADR serves to …" sentence** +so the purpose is unmissable. + +Status is `Draft` while it needs review, then `Accepted`. Other values: +`Rejected`, `Deprecated by NNN`, `Supersedes NNN`. + +## Numbering + +Numbering follows **dependency order**: every ADR may reference only +lower-numbered ones. Check for forward references after adding or renumbering. +Dates therefore need not run in the same order as numbers. + +Renumbering is only acceptable while nothing external cites the numbers. Once +cited, supersede instead. + +## Before writing, ask whether it earns its place + +- **Is there a real alternative?** An ADR comparing one viable candidate against + a dead end documents a decision that made itself, and dilutes the ones + recording genuine trade-offs. Reframe it around the decisions that did have + alternatives, or fold it into a neighbour. +- **Would folding it into an existing ADR be better?** Prefer fewer, shorter + documents. Consolidation has been chosen over adding a document more than + once. +- **Are the rules distinct?** One principle restated against several targets is + one rule with sub-points, not several rules. + +## Before finishing + +- Grep for source field names, model names in the general ADRs, project names, + record counts and measured figures. +- Grep for `Note that`, `worth noting`, `is the lesson`, `easy to get wrong`, + `Two things`, `earns its`, `is the point`. +- Check no ADR cites a higher number than its own. +- `task coding-standards:markdown:check` — recurring failures are line length + over 120 (MD013), misaligned table pipes (MD060), and bold used as a heading + (MD036); use `####` for sub-headings inside a section. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000..1ce7c5e --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,20 @@ +# Architecture Decision Records + +This directory contains Architecture Decision Records (ADRs) for this project. + +See [adr.github.io](https://adr.github.io/) for background on the format. + +| Number | Title | Status | Date | +| ----------------------------------------------- | ------------------------------------------------------- | -------- | ---------- | +| [001](001-architecture-symfony-docker.md) | Architecture — Symfony 8 on the ITK Dev Docker template | Accepted | 2026-08-24 | +| [002](002-publish-to-a-context-broker.md) | Publication mechanism — publish to a context broker | Accepted | 2026-08-24 | +| [003](003-ngsi-ld-representation.md) | NGSI-LD representation — normalized form, batch upsert | Accepted | 2026-08-24 | +| [004](004-coordinate-reference-system.md) | Coordinate reference system — publish WGS84 | Accepted | 2026-08-27 | +| [005](005-smart-data-models-as-vocabulary.md) | Vocabulary — adopt Smart Data Models | Accepted | 2026-08-31 | +| [006](006-onstreetparking-over-parkinggroup.md) | Model selection — OnStreetParking over ParkingGroup | Accepted | 2026-08-31 | + +Numbering follows dependency order: each ADR relies only on lower-numbered +ones. Dates therefore do not run in the same order as numbers. + +All ADRs state general policy and name no data set. Concrete per-data-set +mappings are recorded in the README and in the source classes themselves. From 9aa1a0bf7bc18375a924a3ac5473def7da1286c2 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:20:49 +0200 Subject: [PATCH 02/54] Updated README --- README.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/README.md b/README.md index 95afbea..4052b40 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ We use [DDEV](https://ddev.com/) and [Task](https://taskfile.dev/) for developme ``` shell task site:install +``` ``` shell task site:update @@ -12,6 +13,44 @@ ddev launch Run `task` to see what cool task are available. Running `ddev` can help with other stuff. +## Adapter + +Takes an Aarhus open-data set, converts it to [NGSI-LD], and upserts it into the +context broker. + +``` text +source feed (JSON) + → SourceInterface implementation maps fields, fixes quirks, picks the data model + → NgsiEntity normalized NGSI-LD: Property / GeoProperty / Relationship + → NgsiLdBroker POST /ngsi-ld/v1/entityOperations/upsert + → context broker +``` + +| Class | Responsibility | +| ---------------------------------- | ------------------------------------------------ | +| `App\Source\SourceInterface` | Contract for one input data set | +| `App\Source\HandicapParkingSource` | Disabled parking bays → `OnStreetParking` | +| `App\Source\FeedReader` | Path or URL → decoded JSON | +| `App\Geo\Wgs84Transformer` | Any registered CRS → WGS84, any GeoJSON geometry | +| `App\Ngsi\NgsiEntity` | Builds normalized NGSI-LD entities | +| `App\Broker\NgsiLdBroker` | Batch upsert to the broker | +| `App\Command\ImportCommand` | `app:import` | + +``` shell +task import # list the available sources +task import -- MTM-handicap-parking # import one +task import -- MTM-handicap-parking --dry-run --limit 5 # print the payload instead +task broker:entities -- OnStreetParking 10 # read back what landed +``` + +Adding a data set means adding one `SourceInterface` implementation. It is +discovered through `#[AutoconfigureTag('app.source')]` and shows up as an +`app:import` argument with no further wiring. + +Design decisions are recorded in [docs/adr](docs/adr/README.md). + +[NGSI-LD]: https://www.etsi.org/committee/cim + ## Broker A [Scorpio Broker](https://scorpio.readthedocs.io/) is part of the development setup. From 9dadf6294ec6b36b4ff2cfe03f33ad9a212dd2be Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:21:21 +0200 Subject: [PATCH 03/54] Updated CHANGELOG --- CHANGELOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index bf48d94..2e140da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,4 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- `app:import` command with `--dry-run` and `--limit`, exposed as `task import`. +- `SourceInterface`: extension point for further data sets, discovered through + `#[AutoconfigureTag('app.source')]`. +- `Wgs84Transformer`: reprojects coordinates from any registered CRS to WGS84, + for a single position or a whole GeoJSON geometry of any type. +- `FeedReader`: reads a feed from a filesystem path or an http(s) URL and decodes + it, without interpreting its shape. +- `NgsiEntity`: builds normalized NGSI-LD entities. +- `NgsiLdBroker`: idempotent batch upsert to an NGSI-LD context broker. +- `task broker:entities` for reading entities back out of the broker. +- Architecture Decision Records under `docs/adr`. +- Added test suite + [Unreleased]: https://github.com/itk-dev/enter From 6de59da0dc51fca7efc660f5a69ca5811c0eae67 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:21:35 +0200 Subject: [PATCH 04/54] Added pointers to .env --- .env | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.env b/.env index 999fa0f..2dfb180 100644 --- a/.env +++ b/.env @@ -32,4 +32,18 @@ DEFAULT_URI=http://localhost ###> app ### APP_BROKER_BASE_URI=http://scorpio.local:9090/ + +# JSON-LD contexts attached to every entity, outermost last so the ETSI core +# context resolves the NGSI-LD terms and the domain context resolves the +# Smart Data Models ones. +ENTER_NGSI_CONTEXT_URLS='https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld,https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld' + +# A single domain context, for the Link header that read requests need. Only +# used by `task broker:entities`, and it must be the context defining the type +# being read — Parking while that is the only model published. +ENTER_NGSI_DOMAIN_CONTEXT=https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld + +# Where the disabled-parking export is read from: the live SpatialMap export. +# Accepts any http(s) URL, or a path relative to the project directory. +ENTER_HANDICAP_PARKING_SOURCE='https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap' ###< app ### From a56f14cfc5b217fb124c44b7c458668037d75c7d Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:22:05 +0200 Subject: [PATCH 05/54] Added a few commands to Taskfile --- Taskfile.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/Taskfile.yml b/Taskfile.yml index 49f5e72..3b30260 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -36,6 +36,23 @@ tasks: test:integration: *test_task test:application: *test_task + import: + desc: 'Import a source into the broker, e.g. task import -- MTM-handicap-parking' + cmd: ddev console app:import {{.CLI_ARGS}} + + # The broker itself needs no start task: Scorpio is a DDEV custom service + # (.ddev/docker-compose.scorpio.yaml), so it comes up with the rest of the + # site. Read requests need the domain context in a Link header, which the + # script supplies; the env values come from the dotenv block above, and are + # passed in explicitly because the container shell does not read .env. + broker:entities: + desc: 'List broker entities of a type, e.g. task broker:entities -- OnStreetParking 10' + cmd: >- + ddev exec sh -c + "APP_BROKER_BASE_URI='$APP_BROKER_BASE_URI' + ENTER_NGSI_DOMAIN_CONTEXT='$ENTER_NGSI_DOMAIN_CONTEXT' + sh task/scripts/broker-entities {{.CLI_ARGS}}" + coding-standards:apply: desc: 'Apply coding standards' cmds: From ff7146dad1bca1536d92a6e47052e71146bda4d6 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:24:04 +0200 Subject: [PATCH 06/54] Added wgs84 transformer class --- src/Geo/Wgs84Transformer.php | 160 +++++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 src/Geo/Wgs84Transformer.php diff --git a/src/Geo/Wgs84Transformer.php b/src/Geo/Wgs84Transformer.php new file mode 100644 index 0000000..9f64596 --- /dev/null +++ b/src/Geo/Wgs84Transformer.php @@ -0,0 +1,160 @@ + + */ + private const DEFINITIONS = [ + // ETRS89 / UTM zone 32N — most of Denmark. + 'EPSG:25832' => '+proj=utm +zone=32 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs', + // ETRS89 / UTM zone 33N — Bornholm and eastwards. + 'EPSG:25833' => '+proj=utm +zone=33 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs', + // Web Mercator, in case an input arrives as tile coordinates. + 'EPSG:3857' => '+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +nadgrids=@null +no_defs', + self::TARGET_SRID => '+proj=longlat +datum=WGS84 +no_defs', + ]; + + private readonly Proj4php $proj4; + + /** @var array */ + private readonly array $definitions; + + /** @var array */ + private array $projections = []; + + /** + * @param array $definitions additional PROJ definitions, keyed by SRID + */ + public function __construct(array $definitions = []) + { + $this->definitions = [...self::DEFINITIONS, ...$definitions]; + + $this->proj4 = new Proj4php(); + foreach ($this->definitions as $srid => $definition) { + $this->proj4->addDef($srid, $definition); + } + } + + /** + * Coordinates are not rounded. Downstream use is unknown and may include + * planning work, so the transformed value is published as computed. + * + * @param string $srid source CRS, e.g. "EPSG:25832" + * + * @return array{float, float} GeoJSON coordinate order: [longitude, latitude] + */ + public function toWgs84(string $srid, float $x, float $y): array + { + if (self::TARGET_SRID === $srid) { + return [$x, $y]; + } + + $transformed = $this->proj4->transform( + $this->projection(self::TARGET_SRID), + new Point($x, $y, $this->projection($srid)) + ); + + return [(float) $transformed->x, (float) $transformed->y]; + } + + /** + * @param string $srid source CRS, e.g. "EPSG:25832" + * + * @return array{type: string, coordinates: array{float, float}} GeoJSON Point + */ + public function point(string $srid, float $x, float $y): array + { + return [ + 'type' => 'Point', + 'coordinates' => $this->toWgs84($srid, $x, $y), + ]; + } + + /** + * Reprojects a whole GeoJSON geometry, whatever its type. + * + * Handles Point, LineString, Polygon, MultiPoint, MultiLineString and + * MultiPolygon — every type an NGSI-LD GeoProperty accepts. GeometryCollection + * is not supported, because it carries `geometries` rather than `coordinates`. + * + * A third ordinate (elevation) is dropped: the inputs are two-dimensional, + * and vertical datums are a separate concern this class does not model. + * + * @param string $srid source CRS, e.g. "EPSG:25832" + * @param array $geometry GeoJSON geometry object + * + * @return array{type: string, coordinates: mixed} + */ + public function geometry(string $srid, array $geometry): array + { + $type = $geometry['type'] ?? null; + + if (!\is_string($type) || !\array_key_exists('coordinates', $geometry)) { + throw new \InvalidArgumentException('Not a GeoJSON geometry: "type" and "coordinates" are both required.'); + } + + return [ + 'type' => $type, + 'coordinates' => $this->transformCoordinates($srid, $geometry['coordinates']), + ]; + } + + /** + * GeoJSON nests coordinates to a depth that depends on the geometry type: + * a bare position for Point, an array of positions for LineString, an array + * of those for Polygon, and so on. Recursing until the first element is + * numeric handles every depth without enumerating the types. + * + * @return array + */ + private function transformCoordinates(string $srid, mixed $coordinates): array + { + if (!\is_array($coordinates) || [] === $coordinates) { + throw new \InvalidArgumentException('GeoJSON coordinates must be a non-empty array.'); + } + + if (is_numeric($coordinates[0] ?? null)) { + if (!is_numeric($coordinates[1] ?? null)) { + throw new \InvalidArgumentException('A GeoJSON position needs at least two ordinates.'); + } + + return $this->toWgs84($srid, (float) $coordinates[0], (float) $coordinates[1]); + } + + return array_map( + fn (mixed $nested): array => $this->transformCoordinates($srid, $nested), + array_values($coordinates) + ); + } + + private function projection(string $srid): Proj + { + if (!isset($this->definitions[$srid])) { + throw new \InvalidArgumentException(\sprintf('Unknown CRS "%s". Register a PROJ definition for it before use. Known: %s.', $srid, implode(', ', array_keys($this->definitions)))); + } + + return $this->projections[$srid] ??= new Proj($srid, $this->proj4); + } +} From 0386a7d2fb69484464b91a81725b4a89b7c0cdac Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:24:38 +0200 Subject: [PATCH 07/54] Added a generic way to retrieve data from broker via terminal --- task/scripts/broker-entities | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 task/scripts/broker-entities diff --git a/task/scripts/broker-entities b/task/scripts/broker-entities new file mode 100755 index 0000000..280bb6c --- /dev/null +++ b/task/scripts/broker-entities @@ -0,0 +1,34 @@ +#!/bin/sh +# List entities of a given type from the NGSI-LD broker. +# +# Read requests need a domain context supplied in a Link header; without it the +# broker cannot expand a short type name into the full term the entity was +# written under. The context comes from ENTER_NGSI_DOMAIN_CONTEXT, which must +# be the one defining the type being asked for. +# +# Usage: broker-entities [limit] + +set -eu + +if [ $# -lt 1 ]; then + echo "Usage: broker-entities [limit]" >&2 + echo " NGSI-LD entity type, e.g. OnStreetParking" >&2 + echo " [limit] maximum entities to return (default 100)" >&2 + exit 64 +fi + +TYPE="$1" +LIMIT="${2:-100}" + +RESPONSE=$(curl -sS \ + -H "Link: <${ENTER_NGSI_DOMAIN_CONTEXT}>; rel=\"http://www.w3.org/ns/json-ld#context\"; type=\"application/ld+json\"" \ + "${APP_BROKER_BASE_URI%/}/ngsi-ld/v1/entities?type=${TYPE}&limit=${LIMIT}") + +printf '%s\n' "$RESPONSE" + +# A misspelled type, or one whose context is not the one configured, returns an +# empty list with HTTP 200 — indistinguishable from an empty broker or a failed +# import. Say what was asked for so the difference is visible. +if [ "$(printf '%s' "$RESPONSE" | tr -d '[:space:]')" = "[]" ]; then + printf '\nNo entities of type "%s". A type that does not match, or one defined in a context other than the configured ENTER_NGSI_DOMAIN_CONTEXT, returns an empty list rather than an error — check both before concluding the import failed.\n' "$TYPE" >&2 +fi From 492d105e055f1494d7daa1a356b82600aaaa1a9a Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:32:57 +0200 Subject: [PATCH 08/54] Added feedReader --- src/Source/FeedReader.php | 58 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 src/Source/FeedReader.php diff --git a/src/Source/FeedReader.php b/src/Source/FeedReader.php new file mode 100644 index 0000000..6f8677c --- /dev/null +++ b/src/Source/FeedReader.php @@ -0,0 +1,58 @@ + the decoded document + * + * @throws \RuntimeException when the location is not an http(s) URL, cannot + * be fetched, does not contain valid JSON, or + * does not decode to an array + */ + public function read(string $location): array + { + if (!str_starts_with($location, 'http://') && !str_starts_with($location, 'https://')) { + throw new \RuntimeException(\sprintf('Feed location must be an http(s) URL, got "%s".', $location)); + } + + $json = $this->fetch($location); + + try { + $decoded = json_decode($json, true, 512, \JSON_THROW_ON_ERROR); + } catch (\JsonException $exception) { + throw new \RuntimeException(\sprintf('Invalid JSON in "%s": %s', $location, $exception->getMessage()), previous: $exception); + } + + // A JSON document may legally be a scalar. Every feed we consume is a + // list or an object, and a scalar here means the location is wrong + // rather than that the feed is empty. + if (!\is_array($decoded)) { + throw new \RuntimeException(\sprintf('Expected a JSON array or object in "%s", got %s.', $location, get_debug_type($decoded))); + } + + return $decoded; + } + + private function fetch(string $url): string + { + try { + return $this->client->request('GET', $url)->getContent(); + } catch (\Throwable $exception) { + throw new \RuntimeException(\sprintf('Could not fetch "%s": %s', $url, $exception->getMessage()), previous: $exception); + } + } +} From 6a2ebea1610ae03478860810ac0d88babe61954a Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:43:57 +0200 Subject: [PATCH 09/54] Added import command --- src/Command/ImportCommand.php | 158 ++++++++++++++++++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100644 src/Command/ImportCommand.php diff --git a/src/Command/ImportCommand.php b/src/Command/ImportCommand.php new file mode 100644 index 0000000..1dab064 --- /dev/null +++ b/src/Command/ImportCommand.php @@ -0,0 +1,158 @@ + $sources + */ + public function __construct( + #[AutowireIterator('app.source')] + private readonly iterable $sources, + private readonly NgsiLdBroker $broker, + #[Autowire(env: 'ENTER_NGSI_CONTEXT_URLS')] + private readonly string $contextUrls, + ) { + parent::__construct(); + } + + protected function configure(): void + { + $this + ->addArgument('source', InputArgument::OPTIONAL, 'Source to import. Omit to list the available sources.') + ->addOption('dry-run', null, InputOption::VALUE_NONE, 'Print the NGSI-LD payload instead of sending it.') + ->addOption('limit', 'l', InputOption::VALUE_REQUIRED, 'Import at most this many entities.'); + } + + protected function execute(InputInterface $input, OutputInterface $output): int + { + $io = new SymfonyStyle($input, $output); + + $sources = []; + foreach ($this->sources as $source) { + $sources[$source->key()] = $source; + } + + if ([] === $sources) { + $io->error('No data sources are registered.'); + $io->listing([ + 'A source must implement App\Source\SourceInterface.', + 'Implementations are picked up automatically — check the class exists and is not excluded from the container.', + ]); + + return Command::FAILURE; + } + + $key = $input->getArgument('source'); + + if (null === $key) { + // Options only make sense together with a source. Listing the + // sources and exiting successfully would look like an import ran. + if ($input->getOption('dry-run') || null !== $input->getOption('limit')) { + $io->error(\sprintf( + 'No source given. Available: %s.', + implode(', ', array_keys($sources)) + )); + + return Command::INVALID; + } + + $io->section('Available sources'); + $io->listing(array_keys($sources)); + + return Command::SUCCESS; + } + + if (!isset($sources[$key])) { + $io->error(\sprintf('Unknown source "%s". Available: %s.', $key, implode(', ', array_keys($sources)))); + + return Command::INVALID; + } + + $limit = null !== $input->getOption('limit') ? max(1, (int) $input->getOption('limit')) : null; + $contexts = $this->contexts(); + + $payload = []; + foreach ($sources[$key]->entities() as $entity) { + $payload[] = $entity->toArray($contexts); + + if (null !== $limit && \count($payload) >= $limit) { + break; + } + } + + // A source that yields nothing is almost always misconfigured rather + // than genuinely empty, and it fails silently by construction: a + // record skipped for a missing field looks exactly like a feed with no + // records. Fail loudly so it cannot be mistaken for a successful run. + if ([] === $payload) { + $io->error(\sprintf('Source "%s" produced no entities.', $key)); + $io->text( + 'The source ran to completion without raising an exception, so every record was ' + .'discarded by the source\'s own guards rather than failing. Verbosity flags will ' + .'not reveal more: there is no exception to show.' + ); + $io->listing([ + 'Does the configured path or URL point at the intended document?', + 'Does the document match the shape the source expects — envelope, nesting, field names?', + 'Which guard returns early — a missing identifier, or a missing geometry?', + ]); + + return Command::FAILURE; + } + + if ($input->getOption('dry-run')) { + $output->writeln(json_encode($payload, self::JSON_FLAGS)); + $io->note(\sprintf('Dry run: %d entities were not sent.', \count($payload))); + + return Command::SUCCESS; + } + + try { + $status = $this->broker->upsert($payload); + } catch (\Throwable $exception) { + $io->error($exception->getMessage()); + + return Command::FAILURE; + } + + $io->success(\sprintf( + 'Upserted %d entities into %s (HTTP %d).', + \count($payload), + $this->broker->brokerUrl(), + $status + )); + + return Command::SUCCESS; + } + + /** + * @return list + */ + private function contexts(): array + { + return array_values(array_filter(array_map('trim', explode(',', $this->contextUrls)))); + } +} From 1fdbf473fd67ef6e02ad9dd7988ae3f572aaf0a6 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 13:46:27 +0200 Subject: [PATCH 10/54] Added source interface for picking up new source adapters --- src/Source/SourceInterface.php | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 src/Source/SourceInterface.php diff --git a/src/Source/SourceInterface.php b/src/Source/SourceInterface.php new file mode 100644 index 0000000..978c5ce --- /dev/null +++ b/src/Source/SourceInterface.php @@ -0,0 +1,30 @@ + + */ + public function entities(): iterable; +} From d518d7540e35f341e09804d366430a28878a3595 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 14:00:51 +0200 Subject: [PATCH 11/54] Added mtm spatialmaps handicap parking source adapter --- src/Source/MtmSpatialMaps/HandicapParking.php | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 src/Source/MtmSpatialMaps/HandicapParking.php diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php new file mode 100644 index 0000000..62414b5 --- /dev/null +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -0,0 +1,106 @@ +reader->read($this->location)['features'] ?? [] as $feature) { + if (\is_array($feature) && null !== $entity = $this->toEntity($feature)) { + yield $entity; + } + } + } + + /** + * @param array $feature GeoJSON Feature + */ + private function toEntity(array $feature): ?NgsiEntity + { + // A Feature keeps its attributes under `properties` and its geometry + // beside them, so neither is at the feature's top level. + $row = $feature['properties'] ?? null; + $geometry = $feature['geometry'] ?? null; + + if (!\is_array($row) || !\is_array($geometry)) { + return null; + } + + // mi_prinx is the feed's stable primary key. Without it there is no + // way to address the same bay again on the next import, and an upsert + // would create duplicates instead of updating. + $key = $row['mi_prinx'] ?? null; + if (null === $key || '' === $key) { + return null; + } + + $entity = new NgsiEntity( + \sprintf('urn:ngsi-ld:OnStreetParking:aarhus-handicap-%s', $key), + 'OnStreetParking' + ); + + // `forDisabled` rather than ParkingGroup's `onlyDisabled`: the two + // models have separate category enums, so values are not interchangeable. + // `onStreet` is dropped because the entity type already states it. + return $entity + ->property('name', $this->address($row)) + ->property('description', trim((string) ($row['bemrk'] ?? ''))) + ->property('category', ['forDisabled']) + ->property('totalSpotNumber', (int) ($row['invalidepladser'] ?? 0)) + ->property('source', $this->location) + ->geoProperty('location', $this->transformer->geometry(self::SOURCE_SRID, $geometry)); + } + + /** + * @param array $row + */ + private function address(array $row): string + { + return trim(\sprintf( + '%s %s', + trim((string) ($row['vejnavn'] ?? '')), + trim((string) ($row['husnnr'] ?? '')) + )); + } +} From 1cde88ac7f06b6d281c35e69630fe176ce396d1f Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 14:04:10 +0200 Subject: [PATCH 12/54] Added NgsiEntity for normalized NGSI-LD output --- src/Ngsi/NgsiEntity.php | 81 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 src/Ngsi/NgsiEntity.php diff --git a/src/Ngsi/NgsiEntity.php b/src/Ngsi/NgsiEntity.php new file mode 100644 index 0000000..6f38391 --- /dev/null +++ b/src/Ngsi/NgsiEntity.php @@ -0,0 +1,81 @@ +> */ + private array $attributes = []; + + public function __construct( + private readonly string $id, + private readonly string $type, + ) { + } + + public function id(): string + { + return $this->id; + } + + /** + * Null and empty-string values are dropped rather than emitted as null, + * because the source data uses "" for "not filled in" and a broker would + * otherwise store the emptiness as a fact. + */ + public function property(string $name, mixed $value, ?string $observedAt = null): self + { + if (null === $value || '' === $value || [] === $value) { + return $this; + } + + $attribute = ['type' => 'Property', 'value' => $value]; + + if (null !== $observedAt) { + $attribute['observedAt'] = $observedAt; + } + + $this->attributes[$name] = $attribute; + + return $this; + } + + /** + * @param array{type: string, coordinates: mixed} $geoJson + */ + public function geoProperty(string $name, array $geoJson): self + { + $this->attributes[$name] = ['type' => 'GeoProperty', 'value' => $geoJson]; + + return $this; + } + + public function relationship(string $name, string $object): self + { + $this->attributes[$name] = ['type' => 'Relationship', 'object' => $object]; + + return $this; + } + + /** + * @param list $contextUrls + * + * @return array + */ + public function toArray(array $contextUrls): array + { + return [ + 'id' => $this->id, + 'type' => $this->type, + ...$this->attributes, + '@context' => $contextUrls, + ]; + } +} From 20798f263aed751a8b93af7bfd3861ef1f8c5749 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 14:05:21 +0200 Subject: [PATCH 13/54] Added NgsiLdBroker for idempotent batch upserts --- src/Broker/NgsiLdBroker.php | 69 +++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 src/Broker/NgsiLdBroker.php diff --git a/src/Broker/NgsiLdBroker.php b/src/Broker/NgsiLdBroker.php new file mode 100644 index 0000000..2ce7651 --- /dev/null +++ b/src/Broker/NgsiLdBroker.php @@ -0,0 +1,69 @@ +> $entities + * + * @return int the broker's HTTP status code + */ + public function upsert(array $entities): int + { + if ([] === $entities) { + return 204; + } + + $response = $this->client->request( + 'POST', + rtrim($this->brokerUrl, '/').self::UPSERT_PATH, + [ + 'headers' => ['Content-Type' => self::CONTENT_TYPE], + 'json' => $entities, + ] + ); + + $status = $response->getStatusCode(); + + if ($status >= 400) { + throw new \RuntimeException(\sprintf('Broker rejected the upsert with HTTP %d: %s', $status, $response->getContent(false))); + } + + return $status; + } + + public function brokerUrl(): string + { + return $this->brokerUrl; + } +} From 61f6190d154f4ddbf243d83f86c7d0287f12a53f Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 14:05:33 +0200 Subject: [PATCH 14/54] Added tests --- tests/Command/ImportCommandTest.php | 148 ++++++++++++++ tests/Geo/Wgs84TransformerTest.php | 182 ++++++++++++++++++ tests/Source/FeedReaderTest.php | 114 +++++++++++ .../MtmSpatialMaps/HandicapParkingTest.php | 129 +++++++++++++ 4 files changed, 573 insertions(+) create mode 100644 tests/Command/ImportCommandTest.php create mode 100644 tests/Geo/Wgs84TransformerTest.php create mode 100644 tests/Source/FeedReaderTest.php create mode 100644 tests/Source/MtmSpatialMaps/HandicapParkingTest.php diff --git a/tests/Command/ImportCommandTest.php b/tests/Command/ImportCommandTest.php new file mode 100644 index 0000000..23830aa --- /dev/null +++ b/tests/Command/ImportCommandTest.php @@ -0,0 +1,148 @@ + $sources + */ + private function tester(iterable $sources): CommandTester + { + return new CommandTester(new ImportCommand( + $sources, + new NgsiLdBroker(new MockHttpClient(), 'http://broker.invalid'), + 'https://example.com/context.jsonld', + )); + } + + private function source(string $key, NgsiEntity ...$entities): SourceInterface + { + return new class($key, $entities) implements SourceInterface { + /** @param list $entities */ + public function __construct( + private readonly string $key, + private readonly array $entities, + ) { + } + + public function key(): string + { + return $this->key; + } + + public function entities(): iterable + { + yield from $this->entities; + } + }; + } + + /** + * The important one: a source yielding nothing used to exit successfully + * with a warning, which is indistinguishable from a working import. + */ + public function testItFailsWhenASourceProducesNothing(): void + { + $tester = $this->tester([$this->source('empty-source')]); + + $status = $tester->execute(['source' => 'empty-source', '--dry-run' => true]); + + $this->assertSame(Command::FAILURE, $status); + $this->assertStringContainsString('produced no entities', $tester->getDisplay()); + } + + public function testItSuggestsCausesWhenASourceProducesNothing(): void + { + $tester = $this->tester([$this->source('empty-source')]); + $tester->execute(['source' => 'empty-source', '--dry-run' => true]); + + $display = $tester->getDisplay(); + + $this->assertStringContainsString('path or URL', $display); + $this->assertStringContainsString('envelope, nesting, field names', $display); + } + + public function testItFailsWhenNoSourcesAreRegistered(): void + { + $tester = $this->tester([]); + + $status = $tester->execute([]); + + $this->assertSame(Command::FAILURE, $status); + $this->assertStringContainsString('No data sources are registered', $tester->getDisplay()); + } + + /** + * Passing --dry-run with no source used to print the source listing and + * exit successfully, silently ignoring the flag. + */ + public function testItRejectsOptionsWithoutASource(): void + { + $tester = $this->tester([$this->source('some-source')]); + + $status = $tester->execute(['--dry-run' => true]); + + $this->assertSame(Command::INVALID, $status); + $this->assertStringContainsString('No source given', $tester->getDisplay()); + } + + public function testItStillListsSourcesWhenCalledBare(): void + { + $tester = $this->tester([$this->source('some-source')]); + + $status = $tester->execute([]); + + $this->assertSame(Command::SUCCESS, $status); + $this->assertStringContainsString('some-source', $tester->getDisplay()); + } + + public function testItRejectsAnUnknownSource(): void + { + $tester = $this->tester([$this->source('some-source')]); + + $status = $tester->execute(['source' => 'nope']); + + $this->assertSame(Command::INVALID, $status); + $this->assertStringContainsString('Unknown source "nope"', $tester->getDisplay()); + } + + public function testDryRunPrintsThePayloadAndSendsNothing(): void + { + $entity = (new NgsiEntity('urn:ngsi-ld:Example:1', 'Example')) + ->property('name', 'Example'); + + $tester = $this->tester([$this->source('one-entity', $entity)]); + + $status = $tester->execute(['source' => 'one-entity', '--dry-run' => true]); + $display = $tester->getDisplay(); + + $this->assertSame(Command::SUCCESS, $status); + $this->assertStringContainsString('"urn:ngsi-ld:Example:1"', $display); + $this->assertStringContainsString('1 entities were not sent', $display); + } + + public function testLimitCapsThePayload(): void + { + $entities = []; + foreach (range(1, 5) as $i) { + $entities[] = new NgsiEntity(\sprintf('urn:ngsi-ld:Example:%d', $i), 'Example'); + } + + $tester = $this->tester([$this->source('many', ...$entities)]); + $tester->execute(['source' => 'many', '--dry-run' => true, '--limit' => 2]); + + $this->assertStringContainsString('2 entities were not sent', $tester->getDisplay()); + } +} diff --git a/tests/Geo/Wgs84TransformerTest.php b/tests/Geo/Wgs84TransformerTest.php new file mode 100644 index 0000000..7e3d823 --- /dev/null +++ b/tests/Geo/Wgs84TransformerTest.php @@ -0,0 +1,182 @@ +transformer = new Wgs84Transformer(); + } + + public function testItReprojectsFromUtm32ToWgs84(): void + { + [$longitude, $latitude] = $this->transformer->toWgs84( + self::UTM32, + self::REFERENCE_EASTING, + self::REFERENCE_NORTHING + ); + + $this->assertEqualsWithDelta(10.2152, $longitude, 0.001); + $this->assertEqualsWithDelta(56.1592, $latitude, 0.001); + } + + public function testItReturnsLongitudeLatitudeOrder(): void + { + $point = $this->transformer->point(self::UTM32, 574108.2557507273, 6222343.6199512165); + + $this->assertSame('Point', $point['type']); + + // Longitude first, per GeoJSON. A swapped pair is easy to spot here: + // latitude cannot be 10.19. + $this->assertGreaterThan(9.0, $point['coordinates'][0]); + $this->assertLessThan(11.0, $point['coordinates'][0]); + $this->assertGreaterThan(55.0, $point['coordinates'][1]); + $this->assertLessThan(57.0, $point['coordinates'][1]); + } + + public function testItPassesThroughCoordinatesAlreadyInWgs84(): void + { + $this->assertSame( + [10.199512, 56.149753], + $this->transformer->toWgs84(Wgs84Transformer::TARGET_SRID, 10.199512, 56.149753) + ); + } + + public function testItSupportsAnotherRegisteredCrs(): void + { + // UTM zone 33N: same northing, easting near the zone's central + // meridian, so the result must land east of zone 32's output. + [$longitude] = $this->transformer->toWgs84('EPSG:25833', 500000.0, 6224460.0); + + $this->assertEqualsWithDelta(15.0, $longitude, 0.001); + } + + public function testItRejectsAnUnregisteredCrs(): void + { + $this->expectException(\InvalidArgumentException::class); + $this->expectExceptionMessageMatches('/Unknown CRS "EPSG:31700"/'); + + $this->transformer->toWgs84('EPSG:31700', 500000.0, 6224460.0); + } + + public function testItAcceptsAdditionalDefinitions(): void + { + $transformer = new Wgs84Transformer([ + 'EPSG:23032' => '+proj=utm +zone=32 +ellps=intl +units=m +no_defs', + ]); + + [$longitude, $latitude] = $transformer->toWgs84('EPSG:23032', self::REFERENCE_EASTING, self::REFERENCE_NORTHING); + + // Different datum, so not identical to EPSG:25832 — but the same + // corner of the world, which proves the definition was registered. + $this->assertEqualsWithDelta(10.2152, $longitude, 0.01); + $this->assertEqualsWithDelta(56.1592, $latitude, 0.01); + } + + public function testItReprojectsAPointGeometry(): void + { + $geometry = $this->transformer->geometry(self::UTM32, [ + 'type' => 'Point', + 'coordinates' => [self::REFERENCE_EASTING, self::REFERENCE_NORTHING], + ]); + + $this->assertSame('Point', $geometry['type']); + $this->assertEqualsWithDelta(10.2152, $geometry['coordinates'][0], 0.001); + } + + public function testItReprojectsALineString(): void + { + $geometry = $this->transformer->geometry(self::UTM32, [ + 'type' => 'LineString', + 'coordinates' => [ + [574108.2557507273, 6222343.6199512165], + [self::REFERENCE_EASTING, self::REFERENCE_NORTHING], + ], + ]); + + $this->assertSame('LineString', $geometry['type']); + $this->assertCount(2, $geometry['coordinates']); + $this->assertEqualsWithDelta(10.1926, $geometry['coordinates'][0][0], 0.001); + $this->assertEqualsWithDelta(10.2152, $geometry['coordinates'][1][0], 0.001); + } + + public function testItReprojectsAPolygonPreservingNesting(): void + { + $geometry = $this->transformer->geometry(self::UTM32, [ + 'type' => 'Polygon', + 'coordinates' => [ + [ + [574108.0, 6222343.0], + [574208.0, 6222343.0], + [574208.0, 6222443.0], + [574108.0, 6222343.0], + ], + ], + ]); + + $this->assertSame('Polygon', $geometry['type']); + $this->assertCount(1, $geometry['coordinates']); + $this->assertCount(4, $geometry['coordinates'][0]); + $this->assertIsFloat($geometry['coordinates'][0][0][0]); + } + + public function testItReprojectsAMultiPolygon(): void + { + $ring = [ + [574108.0, 6222343.0], + [574208.0, 6222343.0], + [574208.0, 6222443.0], + [574108.0, 6222343.0], + ]; + + $geometry = $this->transformer->geometry(self::UTM32, [ + 'type' => 'MultiPolygon', + 'coordinates' => [[$ring], [$ring]], + ]); + + $this->assertCount(2, $geometry['coordinates']); + $this->assertCount(4, $geometry['coordinates'][1][0]); + } + + public function testItRejectsAGeometryWithoutCoordinates(): void + { + $this->expectException(\InvalidArgumentException::class); + + $this->transformer->geometry(self::UTM32, ['type' => 'Point']); + } + + public function testItRejectsAGeometryCollection(): void + { + $this->expectException(\InvalidArgumentException::class); + + $this->transformer->geometry(self::UTM32, [ + 'type' => 'GeometryCollection', + 'geometries' => [], + ]); + } + + public function testItRejectsAPositionWithASingleOrdinate(): void + { + $this->expectException(\InvalidArgumentException::class); + + $this->transformer->geometry(self::UTM32, ['type' => 'Point', 'coordinates' => [574108.0]]); + } +} diff --git a/tests/Source/FeedReaderTest.php b/tests/Source/FeedReaderTest.php new file mode 100644 index 0000000..d5487f3 --- /dev/null +++ b/tests/Source/FeedReaderTest.php @@ -0,0 +1,114 @@ +reader($client)->read('https://example.com/feed.json'); + + $this->assertSame([['example' => 1]], $document); + } + + public function testItAcceptsABareArrayDocument(): void + { + $client = new MockHttpClient(new MockResponse('[1, 2, 3]')); + + $this->assertSame([1, 2, 3], $this->reader($client)->read('http://example.com/feed.json')); + } + + /** + * The envelope is feed-specific knowledge, so it must survive intact for + * the caller to interpret. Returning the features list directly would push + * that knowledge into the wrong class. + */ + public function testItDoesNotUnwrapTheEnvelope(): void + { + $client = new MockHttpClient(new MockResponse(self::FEATURE_COLLECTION)); + + $document = $this->reader($client)->read('https://example.com/feed.json'); + + $this->assertArrayHasKey('features', $document); + $this->assertArrayNotHasKey(0, $document); + $this->assertCount(2, $document['features']); + } + + public function testItRejectsALocationThatIsNotAUrl(): void + { + $client = new MockHttpClient(); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessageMatches('/must be an http\(s\) URL/'); + + try { + $this->reader($client)->read('data/feed.json'); + } finally { + // Rejection has to happen before the request, or a mistyped + // location becomes an opaque transport error instead. + $this->assertSame(0, $client->getRequestsCount()); + } + } + + /** + * A string that merely begins with "http" is not a URL. + */ + public function testItRejectsAPathThatMerelyStartsWithHttp(): void + { + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessageMatches('/must be an http\(s\) URL/'); + + $this->reader()->read('https-export.json'); + } + + public function testItFailsOnInvalidJson(): void + { + $client = new MockHttpClient(new MockResponse('{ not json')); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessageMatches('/Invalid JSON/'); + + $this->reader($client)->read('https://example.com/feed.json'); + } + + public function testItFailsWhenTheDocumentIsAScalar(): void + { + $client = new MockHttpClient(new MockResponse('"just a string"')); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessageMatches('/Expected a JSON array or object/'); + + $this->reader($client)->read('https://example.com/feed.json'); + } + + public function testItWrapsTransportFailures(): void + { + $client = new MockHttpClient(new MockResponse('', ['http_code' => 500])); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessageMatches('/Could not fetch/'); + + $this->reader($client)->read('https://example.com/feed.json'); + } +} diff --git a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php new file mode 100644 index 0000000..88fc89b --- /dev/null +++ b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php @@ -0,0 +1,129 @@ +> */ + private array $entities; + + protected function setUp(): void + { + $source = new HandicapParking( + new Wgs84Transformer(), + // Never used: the fixture is read from disk, not over HTTP. + new MockHttpClient(), + 'data/handicapparkering.json', + self::PARKING_SITE_URN, + \dirname(__DIR__, 2), + ); + + $this->entities = array_map( + static fn (NgsiEntity $entity): array => $entity->toArray(['https://example.com/context.jsonld']), + iterator_to_array($source->entities(), false) + ); + } + + public function testItReadsEveryRecordInTheFixture(): void + { + $this->assertCount(10, $this->entities); + } + + public function testItBuildsAParkingGroupWithAStableId(): void + { + $first = $this->entities[0]; + + // The id is derived from mi_prinx so that re-importing upserts the + // same entity instead of creating a duplicate. + $this->assertSame('urn:ngsi-ld:ParkingGroup:aarhus-handicap-261', $first['id']); + $this->assertSame('ParkingGroup', $first['type']); + } + + public function testItJoinsStreetAndHouseNumberIntoName(): void + { + $this->assertSame('P.P. Ørums Gade 2', $this->entities[0]['name']['value']); + } + + public function testItOmitsTheHouseNumberWhenBlank(): void + { + $brammersgade = $this->entityById('urn:ngsi-ld:ParkingGroup:aarhus-handicap-378'); + + // husnnr is "" for this row, so the name must not end in a space. + $this->assertSame('Brammersgade', $brammersgade['name']['value']); + } + + public function testItMarksEveryGroupAsOnStreetDisabledParking(): void + { + foreach ($this->entities as $entity) { + $this->assertSame(['onStreet', 'onlyDisabled'], $entity['category']['value']); + } + } + + public function testItCarriesTheBayCountAsTotalSpotNumber(): void + { + $this->assertSame(2, $this->entities[0]['totalSpotNumber']['value']); + $this->assertSame(1, $this->entities[1]['totalSpotNumber']['value']); + } + + /** + * The feed stores dates as Excel serial numbers with a decimal comma + * ("43543,6190690972"), which has to become a real timestamp. + */ + public function testItConvertsExcelSerialDatesToIso8601(): void + { + $observedAt = $this->entities[0]['totalSpotNumber']['observedAt']; + + $this->assertMatchesRegularExpression('/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/', $observedAt); + $this->assertStringStartsWith('2019-03-', $observedAt); + } + + public function testItEmitsLocationAsAGeoProperty(): void + { + $location = $this->entities[0]['location']; + + $this->assertSame('GeoProperty', $location['type']); + $this->assertSame('Point', $location['value']['type']); + } + + public function testItDropsEmptyDescriptions(): void + { + // bemrk is "" for the first row and "Navitas" for mi_prinx 215. + $this->assertArrayNotHasKey('description', $this->entities[0]); + $this->assertSame( + 'Navitas', + $this->entityById('urn:ngsi-ld:ParkingGroup:aarhus-handicap-215')['description']['value'] + ); + } + + public function testItRelatesEveryGroupToTheSyntheticParkingSite(): void + { + foreach ($this->entities as $entity) { + $this->assertSame('Relationship', $entity['refParkingSite']['type']); + $this->assertSame(self::PARKING_SITE_URN, $entity['refParkingSite']['object']); + } + } + + /** + * @return array + */ + private function entityById(string $id): array + { + foreach ($this->entities as $entity) { + if ($id === $entity['id']) { + return $entity; + } + } + + $this->fail(\sprintf('No entity with id "%s".', $id)); + } +} From db7575b30682875ccc40032ac28995628efde362 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 14:06:09 +0200 Subject: [PATCH 15/54] Renaming paths to new source location structure --- .env | 4 ++-- README.md | 22 +++++++++++----------- Taskfile.yml | 2 +- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/.env b/.env index 2dfb180..5e53eba 100644 --- a/.env +++ b/.env @@ -44,6 +44,6 @@ ENTER_NGSI_CONTEXT_URLS='https://raw.githubusercontent.com/smart-data-models/dat ENTER_NGSI_DOMAIN_CONTEXT=https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld # Where the disabled-parking export is read from: the live SpatialMap export. -# Accepts any http(s) URL, or a path relative to the project directory. -ENTER_HANDICAP_PARKING_SOURCE='https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap' +# Must be an http(s) URL — feeds are read where they live, never from a copy. +ENTER_MTM_SPATIALMAPS_HANDICAP_PARKING_SOURCE='https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap' ###< app ### diff --git a/README.md b/README.md index 4052b40..b6a8d5f 100644 --- a/README.md +++ b/README.md @@ -26,20 +26,20 @@ source feed (JSON) → context broker ``` -| Class | Responsibility | -| ---------------------------------- | ------------------------------------------------ | -| `App\Source\SourceInterface` | Contract for one input data set | -| `App\Source\HandicapParkingSource` | Disabled parking bays → `OnStreetParking` | -| `App\Source\FeedReader` | Path or URL → decoded JSON | -| `App\Geo\Wgs84Transformer` | Any registered CRS → WGS84, any GeoJSON geometry | -| `App\Ngsi\NgsiEntity` | Builds normalized NGSI-LD entities | -| `App\Broker\NgsiLdBroker` | Batch upsert to the broker | -| `App\Command\ImportCommand` | `app:import` | +| Class | Responsibility | +| ------------------------------------------- | ------------------------------------------------ | +| `App\Source\SourceInterface` | Contract for one input data set | +| `App\Source\FeedReader` | Feed URL → decoded JSON | +| `App\Source\MtmSpatialMaps\HandicapParking` | Disabled parking bays → `OnStreetParking` | +| `App\Geo\Wgs84Transformer` | Any registered CRS → WGS84, any GeoJSON geometry | +| `App\Ngsi\NgsiEntity` | Builds normalized NGSI-LD entities | +| `App\Broker\NgsiLdBroker` | Batch upsert to the broker | +| `App\Command\ImportCommand` | `app:import` | ``` shell task import # list the available sources -task import -- MTM-handicap-parking # import one -task import -- MTM-handicap-parking --dry-run --limit 5 # print the payload instead +task import -- mtm_spatialmaps-handicap-parking # import one +task import -- mtm_spatialmaps-handicap-parking --dry-run --limit 5 # print the payload instead task broker:entities -- OnStreetParking 10 # read back what landed ``` diff --git a/Taskfile.yml b/Taskfile.yml index 3b30260..6ac1694 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -37,7 +37,7 @@ tasks: test:application: *test_task import: - desc: 'Import a source into the broker, e.g. task import -- MTM-handicap-parking' + desc: 'Import a source into the broker, e.g. task import -- mtm_spatialmaps-handicap-parking' cmd: ddev console app:import {{.CLI_ARGS}} # The broker itself needs no start task: Scorpio is a DDEV custom service From 8e4f7ca049ed80907c4046190acd1cef0bb9f4d3 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 14:07:15 +0200 Subject: [PATCH 16/54] Added pull request template --- .github/PULL_REQUEST_TEMPLATE.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 .github/PULL_REQUEST_TEMPLATE.md diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..e39b258 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,19 @@ + Date: Tue, 1 Sep 2026 14:23:32 +0200 Subject: [PATCH 17/54] Coding standards --- src/Broker/NgsiLdBroker.php | 10 +++++----- src/Command/ImportCommand.php | 3 +-- src/Geo/Wgs84Transformer.php | 4 ++-- src/Source/FeedReader.php | 4 ++-- src/Source/MtmSpatialMaps/HandicapParking.php | 10 +++++----- tests/Command/ImportCommandTest.php | 8 ++++---- tests/Geo/Wgs84TransformerTest.php | 6 +++--- tests/Source/FeedReaderTest.php | 2 +- tests/Source/MtmSpatialMaps/HandicapParkingTest.php | 2 +- 9 files changed, 24 insertions(+), 25 deletions(-) diff --git a/src/Broker/NgsiLdBroker.php b/src/Broker/NgsiLdBroker.php index 2ce7651..806a99f 100644 --- a/src/Broker/NgsiLdBroker.php +++ b/src/Broker/NgsiLdBroker.php @@ -15,21 +15,21 @@ * import idempotent, which matters because the entity ids are derived from * the source's own primary key. */ -final class NgsiLdBroker +final readonly class NgsiLdBroker { - private const UPSERT_PATH = '/ngsi-ld/v1/entityOperations/upsert'; + private const string UPSERT_PATH = '/ngsi-ld/v1/entityOperations/upsert'; /** * The payload carries its own @context, so it must be sent as * application/ld+json. Sending application/json instead requires the * context in a Link header, and brokers reject the mismatch. */ - private const CONTENT_TYPE = 'application/ld+json'; + private const string CONTENT_TYPE = 'application/ld+json'; public function __construct( - private readonly HttpClientInterface $client, + private HttpClientInterface $client, #[Autowire(env: 'APP_BROKER_BASE_URI')] - private readonly string $brokerUrl, + private string $brokerUrl, ) { } diff --git a/src/Command/ImportCommand.php b/src/Command/ImportCommand.php index 1dab064..b0c6a46 100644 --- a/src/Command/ImportCommand.php +++ b/src/Command/ImportCommand.php @@ -6,7 +6,6 @@ use App\Broker\NgsiLdBroker; use App\Source\SourceInterface; -use MongoDB\Driver\Command; use Symfony\Component\Console\Attribute\AsCommand; use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Input\InputArgument; @@ -153,6 +152,6 @@ protected function execute(InputInterface $input, OutputInterface $output): int */ private function contexts(): array { - return array_values(array_filter(array_map('trim', explode(',', $this->contextUrls)))); + return array_values(array_filter(array_map(trim(...), explode(',', $this->contextUrls)))); } } diff --git a/src/Geo/Wgs84Transformer.php b/src/Geo/Wgs84Transformer.php index 9f64596..4894555 100644 --- a/src/Geo/Wgs84Transformer.php +++ b/src/Geo/Wgs84Transformer.php @@ -15,7 +15,7 @@ */ final class Wgs84Transformer { - public const TARGET_SRID = 'EPSG:4326'; + public const string TARGET_SRID = 'EPSG:4326'; /** * PROJ definitions for coordinate systems this application reads. @@ -26,7 +26,7 @@ final class Wgs84Transformer * * @var array */ - private const DEFINITIONS = [ + private const array DEFINITIONS = [ // ETRS89 / UTM zone 32N — most of Denmark. 'EPSG:25832' => '+proj=utm +zone=32 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs', // ETRS89 / UTM zone 33N — Bornholm and eastwards. diff --git a/src/Source/FeedReader.php b/src/Source/FeedReader.php index 6f8677c..6a9c280 100644 --- a/src/Source/FeedReader.php +++ b/src/Source/FeedReader.php @@ -9,10 +9,10 @@ /** * Fetches a feed over HTTP and decodes it. */ -final class FeedReader +final readonly class FeedReader { public function __construct( - private readonly HttpClientInterface $client, + private HttpClientInterface $client, ) { } diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index 62414b5..296340b 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -21,18 +21,18 @@ * * @see https://github.com/smart-data-models/dataModel.Parking/tree/master/OnStreetParking */ -final class HandicapParking implements SourceInterface +final readonly class HandicapParking implements SourceInterface { /** * The CRS(coordinate reference system) this feed publishes - SRID (Spatial reference identifier). */ - private const SOURCE_SRID = 'EPSG:25832'; + private const string SOURCE_SRID = 'EPSG:25832'; public function __construct( - private readonly FeedReader $reader, - private readonly Wgs84Transformer $transformer, + private FeedReader $reader, + private Wgs84Transformer $transformer, #[Autowire(env: 'ENTER_MTM_SPATIALMAPS_HANDICAP_PARKING_SOURCE')] - private readonly string $location, + private string $location, ) { } diff --git a/tests/Command/ImportCommandTest.php b/tests/Command/ImportCommandTest.php index 23830aa..ae858a3 100644 --- a/tests/Command/ImportCommandTest.php +++ b/tests/Command/ImportCommandTest.php @@ -29,11 +29,11 @@ private function tester(iterable $sources): CommandTester private function source(string $key, NgsiEntity ...$entities): SourceInterface { - return new class($key, $entities) implements SourceInterface { + return new readonly class($key, $entities) implements SourceInterface { /** @param list $entities */ public function __construct( - private readonly string $key, - private readonly array $entities, + private string $key, + private array $entities, ) { } @@ -120,7 +120,7 @@ public function testItRejectsAnUnknownSource(): void public function testDryRunPrintsThePayloadAndSendsNothing(): void { - $entity = (new NgsiEntity('urn:ngsi-ld:Example:1', 'Example')) + $entity = new NgsiEntity('urn:ngsi-ld:Example:1', 'Example') ->property('name', 'Example'); $tester = $this->tester([$this->source('one-entity', $entity)]); diff --git a/tests/Geo/Wgs84TransformerTest.php b/tests/Geo/Wgs84TransformerTest.php index 7e3d823..aacf039 100644 --- a/tests/Geo/Wgs84TransformerTest.php +++ b/tests/Geo/Wgs84TransformerTest.php @@ -9,15 +9,15 @@ class Wgs84TransformerTest extends TestCase { - private const UTM32 = 'EPSG:25832'; + private const string UTM32 = 'EPSG:25832'; /** * Reference coordinates for a building on the Aarhus harbourfront, known * independently to sit at roughly 56.1592 N, 10.2152 E. This is the * ground truth that catches a broken or missing CRS definition. */ - private const REFERENCE_EASTING = 575477.5441628407; - private const REFERENCE_NORTHING = 6224460.129141892; + private const float REFERENCE_EASTING = 575477.5441628407; + private const float REFERENCE_NORTHING = 6224460.129141892; private Wgs84Transformer $transformer; diff --git a/tests/Source/FeedReaderTest.php b/tests/Source/FeedReaderTest.php index d5487f3..8d925b8 100644 --- a/tests/Source/FeedReaderTest.php +++ b/tests/Source/FeedReaderTest.php @@ -16,7 +16,7 @@ class FeedReaderTest extends TestCase * properties, because nothing this class does depends on what the feed * contains — only on the envelope surviving. */ - private const FEATURE_COLLECTION = '{"type":"FeatureCollection","features":[{"example":1},{"example":2}]}'; + private const string FEATURE_COLLECTION = '{"type":"FeatureCollection","features":[{"example":1},{"example":2}]}'; private function reader(?MockHttpClient $client = null): FeedReader { diff --git a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php index 88fc89b..67200f2 100644 --- a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php +++ b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php @@ -12,7 +12,7 @@ class HandicapParkingTest extends TestCase { - private const PARKING_SITE_URN = 'urn:ngsi-ld:ParkingSite:aarhus-on-street'; + private const string PARKING_SITE_URN = 'urn:ngsi-ld:ParkingSite:aarhus-on-street'; /** @var list> */ private array $entities; From b04cb634ceb00c66b7308c952b962ee7e03b48e3 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 1 Sep 2026 14:39:47 +0200 Subject: [PATCH 18/54] Converted all ADRs in readme to draft --- docs/adr/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/adr/README.md b/docs/adr/README.md index 1ce7c5e..57f6629 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -4,14 +4,14 @@ This directory contains Architecture Decision Records (ADRs) for this project. See [adr.github.io](https://adr.github.io/) for background on the format. -| Number | Title | Status | Date | -| ----------------------------------------------- | ------------------------------------------------------- | -------- | ---------- | -| [001](001-architecture-symfony-docker.md) | Architecture — Symfony 8 on the ITK Dev Docker template | Accepted | 2026-08-24 | -| [002](002-publish-to-a-context-broker.md) | Publication mechanism — publish to a context broker | Accepted | 2026-08-24 | -| [003](003-ngsi-ld-representation.md) | NGSI-LD representation — normalized form, batch upsert | Accepted | 2026-08-24 | -| [004](004-coordinate-reference-system.md) | Coordinate reference system — publish WGS84 | Accepted | 2026-08-27 | -| [005](005-smart-data-models-as-vocabulary.md) | Vocabulary — adopt Smart Data Models | Accepted | 2026-08-31 | -| [006](006-onstreetparking-over-parkinggroup.md) | Model selection — OnStreetParking over ParkingGroup | Accepted | 2026-08-31 | +| Number | Title | Status | Date | +| ----------------------------------------------- | ------------------------------------------------------- | ------ | ---------- | +| [001](001-architecture-symfony-docker.md) | Architecture — Symfony 8 on the ITK Dev Docker template | Draft | 2026-08-24 | +| [002](002-publish-to-a-context-broker.md) | Publication mechanism — publish to a context broker | Draft | 2026-08-24 | +| [003](003-ngsi-ld-representation.md) | NGSI-LD representation — normalized form, batch upsert | Draft | 2026-08-24 | +| [004](004-coordinate-reference-system.md) | Coordinate reference system — publish WGS84 | Draft | 2026-08-27 | +| [005](005-smart-data-models-as-vocabulary.md) | Vocabulary — adopt Smart Data Models | Draft | 2026-08-31 | +| [006](006-onstreetparking-over-parkinggroup.md) | Model selection — OnStreetParking over ParkingGroup | Draft | 2026-08-31 | Numbering follows dependency order: each ADR relies only on lower-numbered ones. Dates therefore do not run in the same order as numbers. From 22d50170465069d59f90477df075adad3e8342ea Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Fri, 4 Sep 2026 12:24:39 +0200 Subject: [PATCH 19/54] Added ADR 007: data set metadata in a committed source manifest --- docs/adr/007-source-manifest.md | 122 ++++++++++++++++++++++++++++++++ docs/adr/README.md | 4 +- 2 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 docs/adr/007-source-manifest.md diff --git a/docs/adr/007-source-manifest.md b/docs/adr/007-source-manifest.md new file mode 100644 index 0000000..241eb8f --- /dev/null +++ b/docs/adr/007-source-manifest.md @@ -0,0 +1,122 @@ +# 007: Data set metadata — a committed source manifest + +| Field | Value | +|--------------------|--------------------------------------------------------| +| **Created By** | Jeppe Krogh | +| **Date** | 2026-09-02 | +| **Decision Maker** | ITK Dev team | +| **Stakeholders** | ITK Dev developers, data consumers, future maintainers | +| **Status** | Draft | + +## Context + +Each published data set carries facts the import needs — where the feed is read +from, the coordinate reference system its coordinates are in, and the model it +is published as — and facts only people need: who owns the data, on what terms +it may be republished, how often it changes, and which of its fields are +deliberately not published, with the reason for each. + +The first group must be readable by code. The second is what a public data +portal requires at registration, and what a data owner asks for when +establishing what happened to their data. Both grow with the number of data +sets. + +Recording the two groups separately produces a machine-readable value and a +written description of the same value, which can then disagree. Recording only +the first leaves the rest unwritten, and the questions it answers are then +answered from memory. + +This ADR serves to decide where a data set's own facts are recorded, and which +of them the code reads. + +### Drivers + +- **Functional:** the code reads the facts it needs from the same record a + person reads; a data set is registerable on a public portal without a fresh + survey; an incomplete record fails the import that needs it rather than + producing an incomplete publication. +- **Non-functional:** adding a data set requires no deployment change; the + record is reviewable as a diff; no fact exists in two places. + +### Options Considered + +1. **One environment variable per data set.** Follows the convention that + configuration belongs in the environment, and lets a value differ per + environment. But variable names grow with the catalogue, so each new data + set becomes a deployment change; the environment carries strings only, + leaving metadata beyond an address nowhere to live; and values are invisible + in review, so a wrong one is found by running the import. +2. **Every fact in the class that maps the data set.** Nothing can diverge, + there being one copy, and the language enforces its presence. But metadata + is then readable only by opening code, extracting it for portal registration + requires writing an extractor, and correcting a licence or a contact becomes + a code change reviewed as one. +3. **A committed manifest the code reads.** One record per data set, keyed by + the identifier the import selects it with, holding the facts the code needs + beside those it does not, in a shape a catalogue profile can be generated + from. Validating the record becomes work of our own, and the values are + identical in every environment. +4. **An external catalogue or registry service.** The eventual home of + published metadata, with search and harvesting already built. But it has to + be running for an import to work, it is a second system to operate, and it + must be populated before anything can be published from it — from records + that would have to live somewhere else in the meantime. + +## Decision + +Record each data set in a **committed manifest**, keyed by the identifier the +import selects it with, and read from it every fact the code needs. + +Four rules follow: + +1. **Record only what the code cannot state.** How a feed's fields map onto the + model, and every quirk of its shape, stay in the class that performs the + mapping. Restating them in the manifest recreates the divergence the + manifest exists to prevent. +2. **A fact both the code and a reader need is read from the manifest.** It is + not also written in code, in a comment, or in the README. +3. **An incomplete or malformed record is an error.** Fields an import cannot + run without are required, and their absence raises rather than defaulting. A + fact that is unknown is recorded as unknown, so the gap stays visible. +4. **Name the fields after the catalogue profile the data will be registered + under** — DCAT-AP — so that publication is a translation rather than a + redesign. + +Rationale: + +- What has to be prevented is a value diverging from its description, so the + two belong to the same record. +- Portal registration and answering a data owner need the same fields, and + neither can be derived from mapping code. +- A record is reviewed alongside the class that consumes it, so a reviewer sees + the address, the reference system and the model together with the mapping + that assumes them. +- Where a feed is read from is a fact about the data set, not about the machine + running the import, so the environment is the wrong place for it. + +## Consequences + +### Positive + +- One record per data set, reviewable as a diff and versioned with the code. +- Adding a data set is one class and one record, with no deployment change. +- Registration on a public portal is a translation of records that exist. +- Licence, ownership and what was withheld have a single answer, and an + unanswered question shows as an empty value rather than as nothing at all. +- The same records can generate catalogue entities later without introducing a + second source of truth. + +### Negative / Trade-offs + +- **Values are identical in every environment.** Pointing a data set at a copy + for testing means editing a committed file. That is consistent with reading + feeds where they live, but it removes an escape hatch the environment offered. +- **A wrong reference system or model in the manifest is as damaging as a wrong + one in code, while looking less like code.** A wrong reference system yields + coordinates that are well-formed and in the wrong place. +- **Changing the published model changes entity identifiers.** Editing one + value can therefore orphan everything already published; see ADR 005 and ADR + 006. +- **Fields no code reads have nothing keeping them current.** Until catalogue + entities are generated from them, only review does. +- Validating the record is work the environment did not require. diff --git a/docs/adr/README.md b/docs/adr/README.md index 57f6629..0560124 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -12,9 +12,11 @@ See [adr.github.io](https://adr.github.io/) for background on the format. | [004](004-coordinate-reference-system.md) | Coordinate reference system — publish WGS84 | Draft | 2026-08-27 | | [005](005-smart-data-models-as-vocabulary.md) | Vocabulary — adopt Smart Data Models | Draft | 2026-08-31 | | [006](006-onstreetparking-over-parkinggroup.md) | Model selection — OnStreetParking over ParkingGroup | Draft | 2026-08-31 | +| [007](007-source-manifest.md) | Data set metadata — a committed source manifest | Draft | 2026-09-02 | Numbering follows dependency order: each ADR relies only on lower-numbered ones. Dates therefore do not run in the same order as numbers. All ADRs state general policy and name no data set. Concrete per-data-set -mappings are recorded in the README and in the source classes themselves. +facts are recorded in `config/sources.yaml`, and the mappings in the source +classes themselves. From 8eb2a3709afea782ac5c0a84896bc4b0a06ed4ee Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Fri, 4 Sep 2026 12:24:46 +0200 Subject: [PATCH 20/54] Added source manifest and catalog reader One record per data set in config/sources.yaml, DCAT-AP field names. SourceCatalog validates lazily and fails loudly on incomplete records. --- config/sources.yaml | 46 ++++++++ src/Source/SourceCatalog.php | 169 +++++++++++++++++++++++++++++ src/Source/SourceDescriptor.php | 43 ++++++++ tests/Source/SourceCatalogTest.php | 144 ++++++++++++++++++++++++ 4 files changed, 402 insertions(+) create mode 100644 config/sources.yaml create mode 100644 src/Source/SourceCatalog.php create mode 100644 src/Source/SourceDescriptor.php create mode 100644 tests/Source/SourceCatalogTest.php diff --git a/config/sources.yaml b/config/sources.yaml new file mode 100644 index 0000000..8081a03 --- /dev/null +++ b/config/sources.yaml @@ -0,0 +1,46 @@ +# The data sets this application publishes, one entry per source key. +# +# The facts the import needs — where the feed is read from, the coordinate +# reference system it publishes, the Smart Data Model it is published as — are +# read from here, so they exist in exactly one place. +# +# The remaining fields record what no code can state: who owns the data, on +# what terms, how often it changes, and which source fields are deliberately +# not published. The rule is to record only what the code cannot tell you, so +# the field mapping itself stays in the source class, which is the only place +# that knows the feed's shape. +# +# Field names follow DCAT-AP, the metadata profile European data portals +# harvest, so registering a data set is a translation of its entry rather than +# a new survey. See docs/adr/007-source-manifest.md. + +sources: + mtm_spatialmaps-handicap-parking: + title: 'Handicapparkering, Aarhus Kommune' + description: >- + Disabled parking bays in Aarhus Municipality, with the number of + reserved bays per location. + publisher: 'Aarhus Kommune' + contact: ppg@aarhus.dk + landing_page: 'https://www.opendata.dk/city-of-aarhus/parkering-i-aarhus-kommune' + access_url: 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap' + media_type: application/geo+json + crs: 'EPSG:25832' + model: OnStreetParking + update_frequency: continuous + + # The portal states no licence for this data set. DCAT-AP requires + # one, so it has to be settled with the data owner before the + # catalogue can be registered anywhere. + licence: ~ + + # Fields the feed carries that are not published. Recorded here + # because the source class shows what is mapped but cannot show what + # was left out, or why. + omitted_fields: + ident: 'Single-letter code; its meaning is not documented and not confirmed by the data owner.' + oprettet_af: 'Directory username of the municipal employee who created the record — personal data.' + rettet_af: 'Directory username of the municipal employee who last edited the record — personal data.' + oprettet_dato: 'Describes the register record rather than the parking bay.' + rettet_dato: 'Describes the register record rather than the parking bay. A candidate for observedAt, not yet mapped.' + mi_style: 'MapInfo rendering style, empty throughout the export.' diff --git a/src/Source/SourceCatalog.php b/src/Source/SourceCatalog.php new file mode 100644 index 0000000..cdcd9f9 --- /dev/null +++ b/src/Source/SourceCatalog.php @@ -0,0 +1,169 @@ +|null */ + private ?array $descriptors = null; + + public function __construct( + #[Autowire('%kernel.project_dir%/config/sources.yaml')] + private readonly string $manifest, + ) { + } + + /** + * @throws \RuntimeException when the manifest cannot be read, or carries no entry for the key + */ + public function get(string $key): SourceDescriptor + { + $descriptors = $this->all(); + + if (!isset($descriptors[$key])) { + throw new \RuntimeException(\sprintf('No entry for source "%s" in %s. Entries: %s.', $key, $this->manifest, [] === $descriptors ? 'none' : implode(', ', array_keys($descriptors)))); + } + + return $descriptors[$key]; + } + + /** + * @return array keyed by source key + * + * @throws \RuntimeException when the manifest cannot be read + */ + public function all(): array + { + return $this->descriptors ??= $this->load(); + } + + /** + * @return array + */ + private function load(): array + { + if (!is_file($this->manifest)) { + throw new \RuntimeException(\sprintf('Source manifest "%s" does not exist.', $this->manifest)); + } + + try { + $parsed = Yaml::parseFile($this->manifest); + } catch (ParseException $exception) { + throw new \RuntimeException(\sprintf('Source manifest "%s" is not valid YAML: %s', $this->manifest, $exception->getMessage()), previous: $exception); + } + + // Without this the wrong shape yields an empty catalogue, which reads + // as "no data sets are registered" rather than as a broken file. + $sources = \is_array($parsed) ? $parsed['sources'] ?? null : null; + if (!\is_array($sources)) { + throw new \RuntimeException(\sprintf('Source manifest "%s" must contain a "sources" mapping at the top level.', $this->manifest)); + } + + $descriptors = []; + foreach ($sources as $key => $entry) { + $key = (string) $key; + + if (!\is_array($entry)) { + throw new \RuntimeException(\sprintf('Entry "%s" in %s must be a mapping, got %s.', $key, $this->manifest, get_debug_type($entry))); + } + + $descriptors[$key] = $this->descriptor($key, $entry); + } + + return $descriptors; + } + + /** + * @param array $entry + */ + private function descriptor(string $key, array $entry): SourceDescriptor + { + return new SourceDescriptor( + key: $key, + title: $this->required($key, $entry, 'title'), + accessUrl: $this->required($key, $entry, 'access_url'), + crs: $this->required($key, $entry, 'crs'), + model: $this->required($key, $entry, 'model'), + description: $this->optional($key, $entry, 'description'), + publisher: $this->optional($key, $entry, 'publisher'), + contact: $this->optional($key, $entry, 'contact'), + landingPage: $this->optional($key, $entry, 'landing_page'), + mediaType: $this->optional($key, $entry, 'media_type'), + updateFrequency: $this->optional($key, $entry, 'update_frequency'), + licence: $this->optional($key, $entry, 'licence'), + omittedFields: $this->omittedFields($key, $entry), + ); + } + + /** + * @param array $entry + */ + private function required(string $key, array $entry, string $field): string + { + return $this->optional($key, $entry, $field) + ?? throw new \RuntimeException(\sprintf('Entry "%s" in %s is missing the required field "%s"; an import cannot run without it.', $key, $this->manifest, $field)); + } + + /** + * An empty value means "not filled in", the same as an absent key, so both + * become null rather than an empty string. + * + * @param array $entry + */ + private function optional(string $key, array $entry, string $field): ?string + { + $value = $entry[$field] ?? null; + + if (null === $value || '' === $value) { + return null; + } + + if (!is_scalar($value)) { + throw new \RuntimeException(\sprintf('Field "%s" of entry "%s" in %s must be a single value, got %s.', $field, $key, $this->manifest, get_debug_type($value))); + } + + return trim((string) $value); + } + + /** + * @param array $entry + * + * @return array + */ + private function omittedFields(string $key, array $entry): array + { + $omitted = $entry['omitted_fields'] ?? []; + + if (!\is_array($omitted)) { + throw new \RuntimeException(\sprintf('Field "omitted_fields" of entry "%s" in %s must map each field to the reason it is not published.', $key, $this->manifest)); + } + + $reasons = []; + foreach ($omitted as $field => $reason) { + // The reason is the half of the record that cannot be recovered + // from the code, so a bare list of names is not accepted. + if (!\is_string($reason) || '' === trim($reason)) { + throw new \RuntimeException(\sprintf('Omitted field "%s" of entry "%s" in %s needs a reason.', $field, $key, $this->manifest)); + } + + $reasons[(string) $field] = trim($reason); + } + + return $reasons; + } +} diff --git a/src/Source/SourceDescriptor.php b/src/Source/SourceDescriptor.php new file mode 100644 index 0000000..83b2b14 --- /dev/null +++ b/src/Source/SourceDescriptor.php @@ -0,0 +1,43 @@ + $omittedFields source field name => why it is not published + */ + public function __construct( + public string $key, + public string $title, + public string $accessUrl, + public string $crs, + public string $model, + public ?string $description = null, + public ?string $publisher = null, + public ?string $contact = null, + public ?string $landingPage = null, + public ?string $mediaType = null, + public ?string $updateFrequency = null, + public ?string $licence = null, + public array $omittedFields = [], + ) { + } +} diff --git a/tests/Source/SourceCatalogTest.php b/tests/Source/SourceCatalogTest.php new file mode 100644 index 0000000..e5f1dbd --- /dev/null +++ b/tests/Source/SourceCatalogTest.php @@ -0,0 +1,144 @@ + */ + private array $written = []; + + protected function tearDown(): void + { + foreach ($this->written as $path) { + if (is_file($path)) { + unlink($path); + } + } + + $this->written = []; + } + + public function testTheShippedManifestIsUsable(): void + { + $catalog = new SourceCatalog(\dirname(__DIR__, 2).'/config/sources.yaml'); + + $this->assertNotSame([], $catalog->all(), 'The manifest registers no data sets.'); + } + + /** + * A wrong URL scheme or a CRS the transformer does not know only surfaces + * mid-import otherwise, after the feed has been fetched. + */ + public function testEveryShippedEntryCanBeImportedFrom(): void + { + $catalog = new SourceCatalog(\dirname(__DIR__, 2).'/config/sources.yaml'); + + foreach ($catalog->all() as $key => $descriptor) { + $this->assertSame($key, $descriptor->key); + $this->assertMatchesRegularExpression('#^https?://#', $descriptor->accessUrl, $key); + $this->assertMatchesRegularExpression('/^EPSG:\d+$/', $descriptor->crs, $key); + $this->assertNotSame('', $descriptor->model, $key); + } + } + + public function testItNamesTheKnownEntriesWhenAskedForAnUnknownOne(): void + { + $catalog = new SourceCatalog($this->manifest(<<<'YAML' + sources: + a-source: + title: A source + access_url: https://example.com/feed.json + crs: EPSG:25832 + model: Example + YAML)); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('Entries: a-source.'); + + $catalog->get('no-such-source'); + } + + public function testItRejectsAManifestWithoutASourcesMapping(): void + { + $catalog = new SourceCatalog($this->manifest("data_sets:\n a-source: {}\n")); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('must contain a "sources" mapping'); + + $catalog->all(); + } + + public function testItRejectsAnEntryMissingAFieldTheImportNeeds(): void + { + $catalog = new SourceCatalog($this->manifest(<<<'YAML' + sources: + a-source: + title: A source + access_url: https://example.com/feed.json + model: Example + YAML)); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('missing the required field "crs"'); + + $catalog->all(); + } + + public function testItRejectsAnOmittedFieldWithoutAReason(): void + { + $catalog = new SourceCatalog($this->manifest(<<<'YAML' + sources: + a-source: + title: A source + access_url: https://example.com/feed.json + crs: EPSG:25832 + model: Example + omitted_fields: + some_field: ~ + YAML)); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('needs a reason'); + + $catalog->all(); + } + + public function testItReportsAManifestThatIsNotThere(): void + { + $catalog = new SourceCatalog('/no/such/sources.yaml'); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('does not exist'); + + $catalog->all(); + } + + public function testItReportsUnparsableYaml(): void + { + $catalog = new SourceCatalog($this->manifest("sources:\n - [unbalanced\n")); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('is not valid YAML'); + + $catalog->all(); + } + + private function manifest(string $yaml): string + { + $path = tempnam(sys_get_temp_dir(), 'sources-'); + + if (false === $path) { + $this->fail('Could not create a temporary manifest.'); + } + + file_put_contents($path, $yaml); + $this->written[] = $path; + + return $path; + } +} From 8e3683218d6d6cd5feb048021059a957bae421a5 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Fri, 4 Sep 2026 12:24:55 +0200 Subject: [PATCH 21/54] Moved handicap parking feed config from .env to the source manifest Removes ENTER_MTM_SPATIALMAPS_HANDICAP_PARKING_SOURCE; the feed URL, CRS and model now come from the manifest entry, and the entity type and URN prefix both derive from its model field. Rewrote the test, which called a constructor signature that no longer existed and read a fixture file not in the repo. --- .env | 6 +- src/Source/MtmSpatialMaps/HandicapParking.php | 43 ++--- .../MtmSpatialMaps/HandicapParkingTest.php | 169 ++++++++++++------ 3 files changed, 141 insertions(+), 77 deletions(-) diff --git a/.env b/.env index 5e53eba..a9d7b27 100644 --- a/.env +++ b/.env @@ -43,7 +43,7 @@ ENTER_NGSI_CONTEXT_URLS='https://raw.githubusercontent.com/smart-data-models/dat # being read — Parking while that is the only model published. ENTER_NGSI_DOMAIN_CONTEXT=https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld -# Where the disabled-parking export is read from: the live SpatialMap export. -# Must be an http(s) URL — feeds are read where they live, never from a copy. -ENTER_MTM_SPATIALMAPS_HANDICAP_PARKING_SOURCE='https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap' +# Where each feed is read from is not configured here. It belongs to the data +# set rather than to the environment, and lives in config/sources.yaml — see +# docs/adr/007-source-manifest.md. ###< app ### diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index 296340b..72e0653 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -7,47 +7,50 @@ use App\Geo\Wgs84Transformer; use App\Ngsi\NgsiEntity; use App\Source\FeedReader; +use App\Source\SourceCatalog; +use App\Source\SourceDescriptor; use App\Source\SourceInterface; -use Symfony\Component\DependencyInjection\Attribute\Autowire; /** * Disabled parking bays in Aarhus Municipality, exported from SpatialMap. - * https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap. * - * Published at site level as OnStreetParking rather than as a ParkingGroup - * subdivision: the feed describes locations with a count of reserved bays and - * nothing above them, and ParkingGroup requires a parent site this source does - * not contain. See ADR 006, and ADR 005 rule 2 for the principle behind it. + * Where the feed is read from, the CRS it publishes and the model it is + * published as come from this key's manifest entry; this class owns only the + * field mapping. * + * Published at site level rather than as a ParkingGroup subdivision: the feed + * describes locations with a count of reserved bays and nothing above them, + * and ParkingGroup requires a parent site this source does not contain. See + * ADR 006, and ADR 005 rule 2 for the principle behind it. + * + * @see config/sources.yaml * @see https://github.com/smart-data-models/dataModel.Parking/tree/master/OnStreetParking */ final readonly class HandicapParking implements SourceInterface { - /** - * The CRS(coordinate reference system) this feed publishes - SRID (Spatial reference identifier). - */ - private const string SOURCE_SRID = 'EPSG:25832'; + private const string KEY = 'mtm_spatialmaps-handicap-parking'; public function __construct( private FeedReader $reader, private Wgs84Transformer $transformer, - #[Autowire(env: 'ENTER_MTM_SPATIALMAPS_HANDICAP_PARKING_SOURCE')] - private string $location, + private SourceCatalog $catalog, ) { } public function key(): string { - return 'mtm_spatialmaps-handicap-parking'; + return self::KEY; } public function entities(): iterable { + $source = $this->catalog->get(self::KEY); + // The export is a GeoJSON FeatureCollection, so the records live under // `features`. Iterating the document itself would walk its two // top-level keys instead. - foreach ($this->reader->read($this->location)['features'] ?? [] as $feature) { - if (\is_array($feature) && null !== $entity = $this->toEntity($feature)) { + foreach ($this->reader->read($source->accessUrl)['features'] ?? [] as $feature) { + if (\is_array($feature) && null !== $entity = $this->toEntity($feature, $source)) { yield $entity; } } @@ -56,7 +59,7 @@ public function entities(): iterable /** * @param array $feature GeoJSON Feature */ - private function toEntity(array $feature): ?NgsiEntity + private function toEntity(array $feature, SourceDescriptor $source): ?NgsiEntity { // A Feature keeps its attributes under `properties` and its geometry // beside them, so neither is at the feature's top level. @@ -76,8 +79,8 @@ private function toEntity(array $feature): ?NgsiEntity } $entity = new NgsiEntity( - \sprintf('urn:ngsi-ld:OnStreetParking:aarhus-handicap-%s', $key), - 'OnStreetParking' + \sprintf('urn:ngsi-ld:%s:aarhus-handicap-%s', $source->model, $key), + $source->model ); // `forDisabled` rather than ParkingGroup's `onlyDisabled`: the two @@ -88,8 +91,8 @@ private function toEntity(array $feature): ?NgsiEntity ->property('description', trim((string) ($row['bemrk'] ?? ''))) ->property('category', ['forDisabled']) ->property('totalSpotNumber', (int) ($row['invalidepladser'] ?? 0)) - ->property('source', $this->location) - ->geoProperty('location', $this->transformer->geometry(self::SOURCE_SRID, $geometry)); + ->property('source', $source->accessUrl) + ->geoProperty('location', $this->transformer->geometry($source->crs, $geometry)); } /** diff --git a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php index 67200f2..e0065a7 100644 --- a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php +++ b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php @@ -6,27 +6,41 @@ use App\Geo\Wgs84Transformer; use App\Ngsi\NgsiEntity; +use App\Source\FeedReader; use App\Source\MtmSpatialMaps\HandicapParking; +use App\Source\SourceCatalog; +use App\Source\SourceDescriptor; use PHPUnit\Framework\TestCase; use Symfony\Component\HttpClient\MockHttpClient; +use Symfony\Component\HttpClient\Response\MockResponse; +/** + * Runs against the manifest the project ships, not a fixture of one, so the + * entry this source depends on is covered too. Only the feed is mocked. + */ class HandicapParkingTest extends TestCase { - private const string PARKING_SITE_URN = 'urn:ngsi-ld:ParkingSite:aarhus-on-street'; + private const string KEY = 'mtm_spatialmaps-handicap-parking'; + + private SourceDescriptor $source; + + private string $requestedUrl; /** @var list> */ private array $entities; protected function setUp(): void { - $source = new HandicapParking( - new Wgs84Transformer(), - // Never used: the fixture is read from disk, not over HTTP. - new MockHttpClient(), - 'data/handicapparkering.json', - self::PARKING_SITE_URN, - \dirname(__DIR__, 2), - ); + $catalog = new SourceCatalog(\dirname(__DIR__, 3).'/config/sources.yaml'); + $this->source = $catalog->get(self::KEY); + + $client = new MockHttpClient(function (string $method, string $url): MockResponse { + $this->requestedUrl = $url; + + return new MockResponse(json_encode($this->feed(), \JSON_THROW_ON_ERROR)); + }); + + $source = new HandicapParking(new FeedReader($client), new Wgs84Transformer(), $catalog); $this->entities = array_map( static fn (NgsiEntity $entity): array => $entity->toArray(['https://example.com/context.jsonld']), @@ -34,96 +48,143 @@ protected function setUp(): void ); } - public function testItReadsEveryRecordInTheFixture(): void + public function testItReadsTheFeedTheManifestPointsAt(): void + { + $this->assertSame($this->source->accessUrl, $this->requestedUrl); + } + + public function testItSkipsRecordsWithoutAnIdentifierOrGeometry(): void { - $this->assertCount(10, $this->entities); + // Four features, of which one has no mi_prinx and one no geometry. + $this->assertCount(2, $this->entities); } - public function testItBuildsAParkingGroupWithAStableId(): void + public function testItTakesIdentifierAndTypeFromTheManifestModel(): void { $first = $this->entities[0]; // The id is derived from mi_prinx so that re-importing upserts the - // same entity instead of creating a duplicate. - $this->assertSame('urn:ngsi-ld:ParkingGroup:aarhus-handicap-261', $first['id']); - $this->assertSame('ParkingGroup', $first['type']); + // same bay instead of creating a duplicate. + $this->assertSame( + \sprintf('urn:ngsi-ld:%s:aarhus-handicap-172', $this->source->model), + $first['id'] + ); + $this->assertSame($this->source->model, $first['type']); } public function testItJoinsStreetAndHouseNumberIntoName(): void { - $this->assertSame('P.P. Ørums Gade 2', $this->entities[0]['name']['value']); + $this->assertSame('Domkirkeplads/Bispegade 1', $this->entities[0]['name']['value']); } public function testItOmitsTheHouseNumberWhenBlank(): void { - $brammersgade = $this->entityById('urn:ngsi-ld:ParkingGroup:aarhus-handicap-378'); - // husnnr is "" for this row, so the name must not end in a space. - $this->assertSame('Brammersgade', $brammersgade['name']['value']); + $this->assertSame('Brammersgade', $this->entities[1]['name']['value']); } - public function testItMarksEveryGroupAsOnStreetDisabledParking(): void + public function testItMarksEveryEntityAsDisabledParking(): void { foreach ($this->entities as $entity) { - $this->assertSame(['onStreet', 'onlyDisabled'], $entity['category']['value']); + $this->assertSame(['forDisabled'], $entity['category']['value']); } } public function testItCarriesTheBayCountAsTotalSpotNumber(): void { - $this->assertSame(2, $this->entities[0]['totalSpotNumber']['value']); + $this->assertSame(6, $this->entities[0]['totalSpotNumber']['value']); $this->assertSame(1, $this->entities[1]['totalSpotNumber']['value']); } - /** - * The feed stores dates as Excel serial numbers with a decimal comma - * ("43543,6190690972"), which has to become a real timestamp. - */ - public function testItConvertsExcelSerialDatesToIso8601(): void + public function testItDropsEmptyDescriptions(): void { - $observedAt = $this->entities[0]['totalSpotNumber']['observedAt']; - - $this->assertMatchesRegularExpression('/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/', $observedAt); - $this->assertStringStartsWith('2019-03-', $observedAt); + // bemrk is null for the first row and filled in for the second. + $this->assertArrayNotHasKey('description', $this->entities[0]); + $this->assertSame('Ved indgangen', $this->entities[1]['description']['value']); } - public function testItEmitsLocationAsAGeoProperty(): void + public function testItReprojectsLocationIntoWgs84(): void { $location = $this->entities[0]['location']; $this->assertSame('GeoProperty', $location['type']); $this->assertSame('Point', $location['value']['type']); - } - public function testItDropsEmptyDescriptions(): void - { - // bemrk is "" for the first row and "Navitas" for mi_prinx 215. - $this->assertArrayNotHasKey('description', $this->entities[0]); - $this->assertSame( - 'Navitas', - $this->entityById('urn:ngsi-ld:ParkingGroup:aarhus-handicap-215')['description']['value'] - ); + // The feed publishes metres in the manifest's CRS, so degrees within + // Denmark are the evidence that the reprojection ran. + [$longitude, $latitude] = $location['value']['coordinates']; + $this->assertGreaterThan(8.0, $longitude); + $this->assertLessThan(13.0, $longitude); + $this->assertGreaterThan(54.5, $latitude); + $this->assertLessThan(57.8, $latitude); } - public function testItRelatesEveryGroupToTheSyntheticParkingSite(): void + public function testItRecordsTheManifestUrlAsTheEntitySource(): void { - foreach ($this->entities as $entity) { - $this->assertSame('Relationship', $entity['refParkingSite']['type']); - $this->assertSame(self::PARKING_SITE_URN, $entity['refParkingSite']['object']); - } + $this->assertSame($this->source->accessUrl, $this->entities[0]['source']['value']); } /** + * The first feature is a record from the live export, kept verbatim. The + * rest are constructed to exercise a blank house number and the two guards + * that discard a record. + * * @return array */ - private function entityById(string $id): array + private function feed(): array { - foreach ($this->entities as $entity) { - if ($id === $entity['id']) { - return $entity; - } - } - - $this->fail(\sprintf('No entity with id "%s".', $id)); + return [ + 'type' => 'FeatureCollection', + 'crs' => ['type' => 'name', 'properties' => ['name' => 'EPSG:25832']], + 'features' => [ + [ + 'type' => 'Feature', + 'geometry' => ['type' => 'Point', 'coordinates' => [575153.9524951308, 6224260.609753487]], + 'properties' => [ + 'vejnavn' => 'Domkirkeplads/Bispegade', + 'husnnr' => '1', + 'invalidepladser' => 6, + 'bemrk' => null, + 'ident' => 'P', + 'oprettet_af' => 'ADM\\aztnbnd', + 'oprettet_dato' => '2019-03-19 14:51:23.91', + 'rettet_af' => 'ADM\\aztnbnd', + 'rettet_dato' => '2019-03-19 14:51:23.91', + 'mi_style' => null, + 'mi_prinx' => 172, + ], + ], + [ + 'type' => 'Feature', + 'geometry' => ['type' => 'Point', 'coordinates' => [574000.0, 6223000.0]], + 'properties' => [ + 'vejnavn' => 'Brammersgade', + 'husnnr' => '', + 'invalidepladser' => 1, + 'bemrk' => 'Ved indgangen', + 'mi_prinx' => 378, + ], + ], + [ + 'type' => 'Feature', + 'geometry' => ['type' => 'Point', 'coordinates' => [574100.0, 6223100.0]], + 'properties' => [ + 'vejnavn' => 'Uden nøgle', + 'husnnr' => '3', + 'invalidepladser' => 2, + ], + ], + [ + 'type' => 'Feature', + 'geometry' => null, + 'properties' => [ + 'vejnavn' => 'Uden geometri', + 'husnnr' => '5', + 'invalidepladser' => 2, + 'mi_prinx' => 999, + ], + ], + ], + ]; } } From 0bd7178b40eac6b54c88f2bcb9198f1a1cb732fa Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Fri, 4 Sep 2026 12:25:03 +0200 Subject: [PATCH 22/54] Updated README and changelog for the source manifest --- CHANGELOG.md | 17 +++++++++++++++++ README.md | 43 +++++++++++++++++++++++++++++++------------ 2 files changed, 48 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e140da..5758a47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,5 +21,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `task broker:entities` for reading entities back out of the broker. - Architecture Decision Records under `docs/adr`. - Added test suite +- `config/sources.yaml`: one record per data set, holding the feed URL, its CRS + and the model it is published as alongside the metadata no code can state — + owner, contact, licence, update frequency, and the source fields deliberately + left unpublished with the reason for each. Field names follow DCAT-AP. +- `SourceCatalog` and `SourceDescriptor`: read and validate the manifest, + failing loudly on a missing or malformed record rather than defaulting. + +### Changed + +- Sources read their feed URL, CRS and model from `config/sources.yaml` instead + of holding them in an environment variable and a class constant. + +### Removed + +- `ENTER_MTM_SPATIALMAPS_HANDICAP_PARKING_SOURCE`, and with it the pattern of + one environment variable per data set. Where a feed is read from is a fact + about the data set, not about the environment. [Unreleased]: https://github.com/itk-dev/enter diff --git a/README.md b/README.md index b6a8d5f..511ead3 100644 --- a/README.md +++ b/README.md @@ -26,15 +26,17 @@ source feed (JSON) → context broker ``` -| Class | Responsibility | -| ------------------------------------------- | ------------------------------------------------ | -| `App\Source\SourceInterface` | Contract for one input data set | -| `App\Source\FeedReader` | Feed URL → decoded JSON | -| `App\Source\MtmSpatialMaps\HandicapParking` | Disabled parking bays → `OnStreetParking` | -| `App\Geo\Wgs84Transformer` | Any registered CRS → WGS84, any GeoJSON geometry | -| `App\Ngsi\NgsiEntity` | Builds normalized NGSI-LD entities | -| `App\Broker\NgsiLdBroker` | Batch upsert to the broker | -| `App\Command\ImportCommand` | `app:import` | +| Class | Responsibility | +| ------------------------------------------- | ------------------------------------------------- | +| `App\Source\SourceInterface` | Contract for one input data set | +| `App\Source\SourceCatalog` | Reads the source manifest | +| `App\Source\SourceDescriptor` | One manifest entry: what a data set is and where | +| `App\Source\FeedReader` | Feed URL → decoded JSON | +| `App\Source\MtmSpatialMaps\HandicapParking` | Disabled parking bays → `OnStreetParking` | +| `App\Geo\Wgs84Transformer` | Any registered CRS → WGS84, any GeoJSON geometry | +| `App\Ngsi\NgsiEntity` | Builds normalized NGSI-LD entities | +| `App\Broker\NgsiLdBroker` | Batch upsert to the broker | +| `App\Command\ImportCommand` | `app:import` | ``` shell task import # list the available sources @@ -43,13 +45,30 @@ task import -- mtm_spatialmaps-handicap-parking --dry-run --limit 5 # print t task broker:entities -- OnStreetParking 10 # read back what landed ``` -Adding a data set means adding one `SourceInterface` implementation. It is -discovered through `#[AutoconfigureTag('app.source')]` and shows up as an -`app:import` argument with no further wiring. +### Source manifest + +Every data set is recorded in [config/sources.yaml](config/sources.yaml), keyed +by the identifier `app:import` takes as its argument. The feed URL, the +coordinate reference system it publishes and the Smart Data Model it is +published as are read from there, so each exists in one place only. The rest of +an entry is what no code can state: owner, contact, licence, update frequency, +and the source fields deliberately left unpublished with the reason for each. +The field mapping stays in the source class, which is the only place that knows +the feed's shape. + +Field names follow [DCAT-AP], the metadata profile European data portals +harvest, so registering a data set is a translation of its entry rather than a +new survey. See [ADR 007](docs/adr/007-source-manifest.md). + +Adding a data set means adding one `SourceInterface` implementation and one +manifest entry. The class is discovered through +`#[AutoconfigureTag('app.source')]` and shows up as an `app:import` argument +with no further wiring. Design decisions are recorded in [docs/adr](docs/adr/README.md). [NGSI-LD]: https://www.etsi.org/committee/cim +[DCAT-AP]: https://semiceu.github.io/DCAT-AP/ ## Broker From 1d1e303562728098ad0dbf0470ba66a3368040cc Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 10:18:13 +0200 Subject: [PATCH 23/54] Cleaned up sources.yaml --- config/sources.yaml | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/config/sources.yaml b/config/sources.yaml index 8081a03..b011401 100644 --- a/config/sources.yaml +++ b/config/sources.yaml @@ -1,15 +1,5 @@ # The data sets this application publishes, one entry per source key. # -# The facts the import needs — where the feed is read from, the coordinate -# reference system it publishes, the Smart Data Model it is published as — are -# read from here, so they exist in exactly one place. -# -# The remaining fields record what no code can state: who owns the data, on -# what terms, how often it changes, and which source fields are deliberately -# not published. The rule is to record only what the code cannot tell you, so -# the field mapping itself stays in the source class, which is the only place -# that knows the feed's shape. -# # Field names follow DCAT-AP, the metadata profile European data portals # harvest, so registering a data set is a translation of its entry rather than # a new survey. See docs/adr/007-source-manifest.md. @@ -39,8 +29,8 @@ sources: # was left out, or why. omitted_fields: ident: 'Single-letter code; its meaning is not documented and not confirmed by the data owner.' - oprettet_af: 'Directory username of the municipal employee who created the record — personal data.' - rettet_af: 'Directory username of the municipal employee who last edited the record — personal data.' - oprettet_dato: 'Describes the register record rather than the parking bay.' - rettet_dato: 'Describes the register record rather than the parking bay. A candidate for observedAt, not yet mapped.' + oprettet_af: 'Directory username of the municipal employee who created the record.' + rettet_af: 'Directory username of the municipal employee who last edited the record.' + oprettet_dato: 'Describes the register record.' + rettet_dato: 'Describes the register record.' mi_style: 'MapInfo rendering style, empty throughout the export.' From fd623fc8511addaad9348a375f68c150643aacfa Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 10:25:03 +0200 Subject: [PATCH 24/54] Cleaned up ADR 007 --- docs/adr/007-source-manifest.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/adr/007-source-manifest.md b/docs/adr/007-source-manifest.md index 241eb8f..d644c8f 100644 --- a/docs/adr/007-source-manifest.md +++ b/docs/adr/007-source-manifest.md @@ -26,8 +26,7 @@ written description of the same value, which can then disagree. Recording only the first leaves the rest unwritten, and the questions it answers are then answered from memory. -This ADR serves to decide where a data set's own facts are recorded, and which -of them the code reads. +This ADR serves to decide where a data set's own facts are recorded. ### Drivers @@ -114,9 +113,6 @@ Rationale: - **A wrong reference system or model in the manifest is as damaging as a wrong one in code, while looking less like code.** A wrong reference system yields coordinates that are well-formed and in the wrong place. -- **Changing the published model changes entity identifiers.** Editing one - value can therefore orphan everything already published; see ADR 005 and ADR - 006. - **Fields no code reads have nothing keeping them current.** Until catalogue entities are generated from them, only review does. - Validating the record is work the environment did not require. From bfcac8753c818bb2bcc3cfcbc1fc47021440fde8 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 10:27:37 +0200 Subject: [PATCH 25/54] Shortened comment --- src/Source/MtmSpatialMaps/HandicapParking.php | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index 72e0653..66a0c55 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -12,16 +12,7 @@ use App\Source\SourceInterface; /** - * Disabled parking bays in Aarhus Municipality, exported from SpatialMap. - * - * Where the feed is read from, the CRS it publishes and the model it is - * published as come from this key's manifest entry; this class owns only the - * field mapping. - * - * Published at site level rather than as a ParkingGroup subdivision: the feed - * describes locations with a count of reserved bays and nothing above them, - * and ParkingGroup requires a parent site this source does not contain. See - * ADR 006, and ADR 005 rule 2 for the principle behind it. + * Disabled parking bays in Aarhus Municipality * * @see config/sources.yaml * @see https://github.com/smart-data-models/dataModel.Parking/tree/master/OnStreetParking From f51f6111809ba0b03205370ab5ef0b50edfbd4b4 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 10:41:28 +0200 Subject: [PATCH 26/54] Updated changelog --- CHANGELOG.md | 35 ++++------------------------------- 1 file changed, 4 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5758a47..b089b68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,36 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added - -- `app:import` command with `--dry-run` and `--limit`, exposed as `task import`. -- `SourceInterface`: extension point for further data sets, discovered through - `#[AutoconfigureTag('app.source')]`. -- `Wgs84Transformer`: reprojects coordinates from any registered CRS to WGS84, - for a single position or a whole GeoJSON geometry of any type. -- `FeedReader`: reads a feed from a filesystem path or an http(s) URL and decodes - it, without interpreting its shape. -- `NgsiEntity`: builds normalized NGSI-LD entities. -- `NgsiLdBroker`: idempotent batch upsert to an NGSI-LD context broker. -- `task broker:entities` for reading entities back out of the broker. -- Architecture Decision Records under `docs/adr`. -- Added test suite -- `config/sources.yaml`: one record per data set, holding the feed URL, its CRS - and the model it is published as alongside the metadata no code can state — - owner, contact, licence, update frequency, and the source fields deliberately - left unpublished with the reason for each. Field names follow DCAT-AP. -- `SourceCatalog` and `SourceDescriptor`: read and validate the manifest, - failing loudly on a missing or malformed record rather than defaulting. - -### Changed - -- Sources read their feed URL, CRS and model from `config/sources.yaml` instead - of holding them in an environment variable and a class constant. - -### Removed - -- `ENTER_MTM_SPATIALMAPS_HANDICAP_PARKING_SOURCE`, and with it the pattern of - one environment variable per data set. Where a feed is read from is a fact - about the data set, not about the environment. +* [#3](https://github.com/itk-dev/enter/pull/3) + * Import command that reads a geospatial feed, reprojects it to WGS84 and upserts it to an NGSI-LD broker. + * A committed record per data set — feed, CRS, model, DCAT-AP metadata. + * An extension point for adding data sets, a test suite, and architecture decision records. [Unreleased]: https://github.com/itk-dev/enter From e556c01f5e858053a354e5105e3a7419de28fe9c Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 10:45:08 +0200 Subject: [PATCH 27/54] Updated README --- README.md | 166 +----------------------------------------------------- 1 file changed, 2 insertions(+), 164 deletions(-) diff --git a/README.md b/README.md index 511ead3..6762905 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Run `task` to see what cool task are available. Running `ddev` can help with oth ## Adapter -Takes an Aarhus open-data set, converts it to [NGSI-LD], and upserts it into the +Takes an open-data set, converts it to [NGSI-LD], and upserts it into the context broker. ``` text @@ -26,18 +26,6 @@ source feed (JSON) → context broker ``` -| Class | Responsibility | -| ------------------------------------------- | ------------------------------------------------- | -| `App\Source\SourceInterface` | Contract for one input data set | -| `App\Source\SourceCatalog` | Reads the source manifest | -| `App\Source\SourceDescriptor` | One manifest entry: what a data set is and where | -| `App\Source\FeedReader` | Feed URL → decoded JSON | -| `App\Source\MtmSpatialMaps\HandicapParking` | Disabled parking bays → `OnStreetParking` | -| `App\Geo\Wgs84Transformer` | Any registered CRS → WGS84, any GeoJSON geometry | -| `App\Ngsi\NgsiEntity` | Builds normalized NGSI-LD entities | -| `App\Broker\NgsiLdBroker` | Batch upsert to the broker | -| `App\Command\ImportCommand` | `app:import` | - ``` shell task import # list the available sources task import -- mtm_spatialmaps-handicap-parking # import one @@ -48,17 +36,7 @@ task broker:entities -- OnStreetParking 10 # read back what land ### Source manifest Every data set is recorded in [config/sources.yaml](config/sources.yaml), keyed -by the identifier `app:import` takes as its argument. The feed URL, the -coordinate reference system it publishes and the Smart Data Model it is -published as are read from there, so each exists in one place only. The rest of -an entry is what no code can state: owner, contact, licence, update frequency, -and the source fields deliberately left unpublished with the reason for each. -The field mapping stays in the source class, which is the only place that knows -the feed's shape. - -Field names follow [DCAT-AP], the metadata profile European data portals -harvest, so registering a data set is a translation of its entry rather than a -new survey. See [ADR 007](docs/adr/007-source-manifest.md). +by the identifier `app:import` takes as its argument. See [ADR 007](docs/adr/007-source-manifest.md). Adding a data set means adding one `SourceInterface` implementation and one manifest entry. The class is discovered through @@ -77,143 +55,3 @@ A [Scorpio Broker](https://scorpio.readthedocs.io/) is part of the development s ``` shell ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/types | jq" ``` - -Load some example data: - -``` shell name=import-toilet -ddev console app:broker:entity:delete toilet -ddev console app:broker:import:geojson toilet 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter' -ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --get --data-urlencode type=toilet" | jq '.[]|with_entries(select([.key] | inside(["id", "type", "location"])))' -``` - -``` shell name=import-handicapparkering -ddev console app:broker:entity:delete handicapparkering -ddev console app:broker:import:geojson handicapparkering 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap' -ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --get --data-urlencode type=handicapparkering" | jq '.[]|with_entries(select([.key] | inside(["id", "type", "location"])))' -``` - -``` shell name=import-hundeskov -ddev console app:broker:entity:delete hundeskov -ddev console app:broker:import:geojson hundeskov 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=hundeskove_friluftsliv_aarhus' -ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --get --data-urlencode type=hundeskov" | jq '.[]|with_entries(select([.key] | inside(["id", "type", "location", "geometry"])))' -``` - -### Broker API request examples - - - -``` shell name=scorpio-entity-create substitutions="{«entity-type»: Room, «entity-id»: 'house2:smartrooms:room1'}" -ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --header 'content-type: application/json' --data @-" <<'JSON' -{ - "type": "«entity-type»", - "id": "«entity-id»" -} -JSON - - -# EPSG:4326?! -ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities/«entity-id»/attrs --header 'content-type: application/json' --data @-" <<'JSON' -{ - "location": { - "type": "geo:json", - "value": { - "type": "Point", - "coordinates": [ - 10.15711687080293, 56.126271111641266 - ] - } - } -} -JSON - - -# Get the entities -ddev exec "curl --silent --header 'accept: application/ld+json' http://scorpio.local:9090/ngsi-ld/v1/entities?type=«entity-type» | jq" - -# Get the entities as GeoJSON -ddev exec "curl --silent --header 'accept: application/geo+json' http://scorpio.local:9090/ngsi-ld/v1/entities?type=«entity-type» | jq" -``` - -> [!CAUTION] -> Excuse me what?! -> -> ``` shell -> ddev exec "curl --silent --header 'accept: application/geo+json' 'http://scorpio.local:9090/ngsi-ld/v1/entities?georel=near;maxDistance%3D%3D2000&geometry=Point&coordinates=%5B8,40%5D'" -> ddev exec "curl --silent --header 'accept: application/geo+json' 'http://scorpio.local:9090/ngsi-ld/v1/entities?georel=near;maxDistance==2000&geometry=Point&coordinates=%5B8,40%5D'" -> ddev exec "curl --silent --header 'accept: application/geo+json' 'http://scorpio.local:9090/ngsi-ld/v1/entities?georel=near;maxDistance==2000&geometry=Point&coordinates=[10,56]'" -> > ``` - -## GeoJSON - -* "[Position](https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1)": `(longitude, latitude)` - * "[Coordinate Reference System](https://datatracker.ietf.org/doc/html/rfc7946#section-4)" - * But … - - → : - - ```json - { - "type": "FeatureCollection", - "crs": { - "type": "name", - "properties": { - "name": "EPSG:25832" - } - }, - "bbox": [563262.0903961, 6209995.05346621, 579587.800694027, 6231954.36230685], - "features": [ - { - "type": "Feature", - "geometry": { - "type": "MultiPoint", - "coordinates": [ - [576933.018139537, 6218035.22691216] - ] - }, - "properties": { - … - ``` - -* - ---- - -* -* -* -* - -Must `location` be a `Point` in ngsi-ld? - -``` shell -ddev exec --service scorpio-db "psql ngb ngb" -``` - -``` sql -2026-08-30 10:26:19.789 UTC [41] LOG: execute 0000000: WITH D0 AS (SELECT ID, ENTITY, TRUE as PARENT FROM ENTITY WHERE ST_DWithin( location::geography, ST_SetSRID(ST_GeomFromGeoJSON('{"type": "Point", "coordinates": [10.236808047161853,56.101226991176155] }'), 4326)::geography, 2000.0) ORDER BY createdAt limit $1 offset $2) SELECT ID, ENTITY, PARENT FROM D0 -``` - - - -``` shell -ddev exec --service scorpio-db "psql ngb ngb" <<< "SELECT id , e_types, location FROM entity;" -``` - -``` shell name=hmm -ddev console app:import:geojson toilet 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter' \ - && ddev exec --service scorpio-db "psql ngb ngb" <<< "SELECT id , e_types, ST_AsText(location) AS location FROM entity WHERE id = 'toilet:0000';" \ - && ddev exec --service scorpio-db "psql ngb ngb" <<< "SELECT temporalentity_id, ST_AsText(location) AS location, ST_asText(geovalue) AS geovalue, createdat FROM temporalentityattrinstance WHERE temporalentity_id = 'toilet:0000' ORDER BY createdat DESC LIMIT 10;" -``` - - - -* [Public toilets in Aarhus Municipality](https://www.opendata.dk/city-of-aarhus/toiletter-i-aarhus-kommune) - * [Offentlige - toiletter](http://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter): - -* [Parking in Aarhus Municipality – Zones, Permits and - Spaces](https://www.opendata.dk/city-of-aarhus/parkering-i-aarhus-kommune) - * [Handicapparkering](https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap): - - -* From 8e0627b21acdae7542b27c59d54307b8faa59e2e Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 10:46:16 +0200 Subject: [PATCH 28/54] Removed comment from Taskfile --- Taskfile.yml | 5 ----- 1 file changed, 5 deletions(-) diff --git a/Taskfile.yml b/Taskfile.yml index 6ac1694..b5cfede 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -40,11 +40,6 @@ tasks: desc: 'Import a source into the broker, e.g. task import -- mtm_spatialmaps-handicap-parking' cmd: ddev console app:import {{.CLI_ARGS}} - # The broker itself needs no start task: Scorpio is a DDEV custom service - # (.ddev/docker-compose.scorpio.yaml), so it comes up with the rest of the - # site. Read requests need the domain context in a Link header, which the - # script supplies; the env values come from the dotenv block above, and are - # passed in explicitly because the container shell does not read .env. broker:entities: desc: 'List broker entities of a type, e.g. task broker:entities -- OnStreetParking 10' cmd: >- From c29945e4e383b5466c0d153d961608a0d7a0c3d9 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 10:48:43 +0200 Subject: [PATCH 29/54] Fixed coding standards --- README.md | 1 - src/Source/MtmSpatialMaps/HandicapParking.php | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 6762905..3957294 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,6 @@ with no further wiring. Design decisions are recorded in [docs/adr](docs/adr/README.md). [NGSI-LD]: https://www.etsi.org/committee/cim -[DCAT-AP]: https://semiceu.github.io/DCAT-AP/ ## Broker diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index 66a0c55..bf7dead 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -12,7 +12,7 @@ use App\Source\SourceInterface; /** - * Disabled parking bays in Aarhus Municipality + * Disabled parking bays in Aarhus Municipality. * * @see config/sources.yaml * @see https://github.com/smart-data-models/dataModel.Parking/tree/master/OnStreetParking From 01cfeb5b227914bf6837440bd8ffae96299376b3 Mon Sep 17 00:00:00 2001 From: Jeppe Julius Krogh <106669866+jeppekroghitk@users.noreply.github.com> Date: Mon, 7 Sep 2026 12:55:31 +0200 Subject: [PATCH 30/54] Update src/Geo/Wgs84Transformer.php Co-authored-by: Mikkel Ricky --- src/Geo/Wgs84Transformer.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Geo/Wgs84Transformer.php b/src/Geo/Wgs84Transformer.php index 4894555..3d46433 100644 --- a/src/Geo/Wgs84Transformer.php +++ b/src/Geo/Wgs84Transformer.php @@ -107,7 +107,7 @@ public function point(string $srid, float $x, float $y): array * * @return array{type: string, coordinates: mixed} */ - public function geometry(string $srid, array $geometry): array + public function transformGeometry(string $srid, array $geometry): array { $type = $geometry['type'] ?? null; From b94942983324a5f1942423f6f0fa2583a6922ac4 Mon Sep 17 00:00:00 2001 From: Jeppe Julius Krogh <106669866+jeppekroghitk@users.noreply.github.com> Date: Mon, 7 Sep 2026 12:57:01 +0200 Subject: [PATCH 31/54] Update src/Geo/Wgs84Transformer.php Co-authored-by: Mikkel Ricky --- src/Geo/Wgs84Transformer.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Geo/Wgs84Transformer.php b/src/Geo/Wgs84Transformer.php index 3d46433..84b7593 100644 --- a/src/Geo/Wgs84Transformer.php +++ b/src/Geo/Wgs84Transformer.php @@ -129,7 +129,7 @@ public function transformGeometry(string $srid, array $geometry): array * * @return array */ - private function transformCoordinates(string $srid, mixed $coordinates): array + private function transformCoordinates(string $srid, array $coordinates): array { if (!\is_array($coordinates) || [] === $coordinates) { throw new \InvalidArgumentException('GeoJSON coordinates must be a non-empty array.'); From 684f13e31053478fc2ccaa433f102408bac3df02 Mon Sep 17 00:00:00 2001 From: Jeppe Julius Krogh <106669866+jeppekroghitk@users.noreply.github.com> Date: Mon, 7 Sep 2026 13:23:26 +0200 Subject: [PATCH 32/54] Update src/Ngsi/NgsiEntity.php Co-authored-by: Mikkel Ricky --- src/Ngsi/NgsiEntity.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Ngsi/NgsiEntity.php b/src/Ngsi/NgsiEntity.php index 6f38391..dbc2f79 100644 --- a/src/Ngsi/NgsiEntity.php +++ b/src/Ngsi/NgsiEntity.php @@ -30,7 +30,7 @@ public function id(): string * because the source data uses "" for "not filled in" and a broker would * otherwise store the emptiness as a fact. */ - public function property(string $name, mixed $value, ?string $observedAt = null): self + public function setProperty(string $name, mixed $value, ?string $observedAt = null): self { if (null === $value || '' === $value || [] === $value) { return $this; From 43ef3ac7821fe542185cd9156100a0b8cdfdbd80 Mon Sep 17 00:00:00 2001 From: Jeppe Julius Krogh <106669866+jeppekroghitk@users.noreply.github.com> Date: Mon, 7 Sep 2026 13:23:56 +0200 Subject: [PATCH 33/54] Update src/Source/SourceInterface.php Co-authored-by: Mikkel Ricky --- src/Source/SourceInterface.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Source/SourceInterface.php b/src/Source/SourceInterface.php index 978c5ce..b372d28 100644 --- a/src/Source/SourceInterface.php +++ b/src/Source/SourceInterface.php @@ -19,7 +19,7 @@ interface SourceInterface { /** - * Identifier used to select this source on the command line. + * Unique identifier for this source. */ public function key(): string; From 21d75592a1122f46a1647720e99579871f74707e Mon Sep 17 00:00:00 2001 From: Jeppe Julius Krogh <106669866+jeppekroghitk@users.noreply.github.com> Date: Mon, 7 Sep 2026 13:26:46 +0200 Subject: [PATCH 34/54] Update src/Source/FeedReader.php Co-authored-by: Mikkel Ricky --- src/Source/FeedReader.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Source/FeedReader.php b/src/Source/FeedReader.php index 6a9c280..d2b6400 100644 --- a/src/Source/FeedReader.php +++ b/src/Source/FeedReader.php @@ -9,7 +9,7 @@ /** * Fetches a feed over HTTP and decodes it. */ -final readonly class FeedReader +final readonly class DataSourceReader { public function __construct( private HttpClientInterface $client, From 93897e33ae0f1ac374a5325a52eb2e5190f06b2c Mon Sep 17 00:00:00 2001 From: Jeppe Julius Krogh <106669866+jeppekroghitk@users.noreply.github.com> Date: Mon, 7 Sep 2026 13:28:32 +0200 Subject: [PATCH 35/54] Update src/Source/FeedReader.php Co-authored-by: Mikkel Ricky --- src/Source/FeedReader.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Source/FeedReader.php b/src/Source/FeedReader.php index d2b6400..c84ce03 100644 --- a/src/Source/FeedReader.php +++ b/src/Source/FeedReader.php @@ -23,7 +23,7 @@ public function __construct( * be fetched, does not contain valid JSON, or * does not decode to an array */ - public function read(string $location): array + public function read(string $url): array { if (!str_starts_with($location, 'http://') && !str_starts_with($location, 'https://')) { throw new \RuntimeException(\sprintf('Feed location must be an http(s) URL, got "%s".', $location)); From 2a56d2ca3cb2122e0c3d18f20eba7db4a7a6bea4 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 14:34:56 +0200 Subject: [PATCH 36/54] Renamed FeedReader to DataSourceReader --- src/Source/{FeedReader.php => DataSourceReader.php} | 0 src/Source/MtmSpatialMaps/HandicapParking.php | 4 ++-- .../{FeedReaderTest.php => DataSourceReaderTest.php} | 8 ++++---- tests/Source/MtmSpatialMaps/HandicapParkingTest.php | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) rename src/Source/{FeedReader.php => DataSourceReader.php} (100%) rename tests/Source/{FeedReaderTest.php => DataSourceReaderTest.php} (94%) diff --git a/src/Source/FeedReader.php b/src/Source/DataSourceReader.php similarity index 100% rename from src/Source/FeedReader.php rename to src/Source/DataSourceReader.php diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index bf7dead..441a452 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -6,7 +6,7 @@ use App\Geo\Wgs84Transformer; use App\Ngsi\NgsiEntity; -use App\Source\FeedReader; +use App\Source\DataSourceReader; use App\Source\SourceCatalog; use App\Source\SourceDescriptor; use App\Source\SourceInterface; @@ -22,7 +22,7 @@ private const string KEY = 'mtm_spatialmaps-handicap-parking'; public function __construct( - private FeedReader $reader, + private DataSourceReader $reader, private Wgs84Transformer $transformer, private SourceCatalog $catalog, ) { diff --git a/tests/Source/FeedReaderTest.php b/tests/Source/DataSourceReaderTest.php similarity index 94% rename from tests/Source/FeedReaderTest.php rename to tests/Source/DataSourceReaderTest.php index 8d925b8..9c866ca 100644 --- a/tests/Source/FeedReaderTest.php +++ b/tests/Source/DataSourceReaderTest.php @@ -4,12 +4,12 @@ namespace App\Tests\Source; -use App\Source\FeedReader; +use App\Source\DataSourceReader; use PHPUnit\Framework\TestCase; use Symfony\Component\HttpClient\MockHttpClient; use Symfony\Component\HttpClient\Response\MockResponse; -class FeedReaderTest extends TestCase +class DataSourceReaderTest extends TestCase { /** * Content-agnostic on purpose: a minimal FeatureCollection with meaningless @@ -18,9 +18,9 @@ class FeedReaderTest extends TestCase */ private const string FEATURE_COLLECTION = '{"type":"FeatureCollection","features":[{"example":1},{"example":2}]}'; - private function reader(?MockHttpClient $client = null): FeedReader + private function reader(?MockHttpClient $client = null): DataSourceReader { - return new FeedReader($client ?? new MockHttpClient()); + return new DataSourceReader($client ?? new MockHttpClient()); } public function testItReadsFromAnHttpUrl(): void diff --git a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php index e0065a7..4a67b14 100644 --- a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php +++ b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php @@ -6,7 +6,7 @@ use App\Geo\Wgs84Transformer; use App\Ngsi\NgsiEntity; -use App\Source\FeedReader; +use App\Source\DataSourceReader; use App\Source\MtmSpatialMaps\HandicapParking; use App\Source\SourceCatalog; use App\Source\SourceDescriptor; @@ -40,7 +40,7 @@ protected function setUp(): void return new MockResponse(json_encode($this->feed(), \JSON_THROW_ON_ERROR)); }); - $source = new HandicapParking(new FeedReader($client), new Wgs84Transformer(), $catalog); + $source = new HandicapParking(new DataSourceReader($client), new Wgs84Transformer(), $catalog); $this->entities = array_map( static fn (NgsiEntity $entity): array => $entity->toArray(['https://example.com/context.jsonld']), From e4452fd003532f507c34816b785a529f461da2ef Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 14:47:03 +0200 Subject: [PATCH 37/54] Validated the source manifest as Symfony config Replaces the hand-written field checks with a config tree, and reads the whole manifest during container warm-up, so a malformed entry fails the build instead of the one import that happens to select it. --- CHANGELOG.md | 2 + composer.json | 1 + composer.lock | 2 +- src/Source/SourceCatalog.php | 136 +++++++-------------- src/Source/SourceCatalogWarmer.php | 38 ++++++ src/Source/SourceManifestConfiguration.php | 115 +++++++++++++++++ tests/Source/SourceCatalogTest.php | 91 +++++++++++++- tests/Source/SourceCatalogWarmerTest.php | 86 +++++++++++++ 8 files changed, 376 insertions(+), 95 deletions(-) create mode 100644 src/Source/SourceCatalogWarmer.php create mode 100644 src/Source/SourceManifestConfiguration.php create mode 100644 tests/Source/SourceCatalogWarmerTest.php diff --git a/CHANGELOG.md b/CHANGELOG.md index b089b68..e6e1c78 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,5 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 * Import command that reads a geospatial feed, reprojects it to WGS84 and upserts it to an NGSI-LD broker. * A committed record per data set — feed, CRS, model, DCAT-AP metadata. * An extension point for adding data sets, a test suite, and architecture decision records. + * Source manifest validated against a Symfony config tree and read during container warm-up, so a malformed + entry fails the build rather than the one import that selects it. [Unreleased]: https://github.com/itk-dev/enter diff --git a/composer.json b/composer.json index b3bcf60..fad8d55 100644 --- a/composer.json +++ b/composer.json @@ -13,6 +13,7 @@ "proj4php/proj4php": "^2.0", "symfony/asset": "~8.1.0", "symfony/asset-mapper": "~8.1.0", + "symfony/config": "~8.1.0", "symfony/console": "~8.1.6", "symfony/dotenv": "~8.1.6", "symfony/flex": "^2", diff --git a/composer.lock b/composer.lock index 2b3d45f..2d8ddb4 100644 --- a/composer.lock +++ b/composer.lock @@ -4,7 +4,7 @@ "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies", "This file is @generated automatically" ], - "content-hash": "89753fcad5de669970a5e88087a1a523", + "content-hash": "dfaceee7643a3761619ee1e8ae943e6a", "packages": [ { "name": "composer/semver", diff --git a/src/Source/SourceCatalog.php b/src/Source/SourceCatalog.php index cdcd9f9..2b385df 100644 --- a/src/Source/SourceCatalog.php +++ b/src/Source/SourceCatalog.php @@ -4,6 +4,8 @@ namespace App\Source; +use Symfony\Component\Config\Definition\Exception\InvalidConfigurationException; +use Symfony\Component\Config\Definition\Processor; use Symfony\Component\DependencyInjection\Attribute\Autowire; use Symfony\Component\Yaml\Exception\ParseException; use Symfony\Component\Yaml\Yaml; @@ -11,10 +13,13 @@ /** * The manifest of data sets this application publishes. * - * Parsed on first use rather than during container warm-up, so a malformed - * entry fails the import that needs it instead of every cache clear. + * The record's shape is a Symfony config tree rather than hand-written checks, + * and SourceCatalogWarmer reads the whole manifest during container warm-up, so + * a malformed entry fails the build instead of waiting for the one import that + * happens to select it. * * @see config/sources.yaml + * @see SourceManifestConfiguration * @see docs/adr/007-source-manifest.md */ final class SourceCatalog @@ -57,113 +62,60 @@ public function all(): array */ private function load(): array { - if (!is_file($this->manifest)) { - throw new \RuntimeException(\sprintf('Source manifest "%s" does not exist.', $this->manifest)); - } - - try { - $parsed = Yaml::parseFile($this->manifest); - } catch (ParseException $exception) { - throw new \RuntimeException(\sprintf('Source manifest "%s" is not valid YAML: %s', $this->manifest, $exception->getMessage()), previous: $exception); - } - - // Without this the wrong shape yields an empty catalogue, which reads - // as "no data sets are registered" rather than as a broken file. - $sources = \is_array($parsed) ? $parsed['sources'] ?? null : null; - if (!\is_array($sources)) { - throw new \RuntimeException(\sprintf('Source manifest "%s" must contain a "sources" mapping at the top level.', $this->manifest)); - } - $descriptors = []; - foreach ($sources as $key => $entry) { - $key = (string) $key; - if (!\is_array($entry)) { - throw new \RuntimeException(\sprintf('Entry "%s" in %s must be a mapping, got %s.', $key, $this->manifest, get_debug_type($entry))); - } - - $descriptors[$key] = $this->descriptor($key, $entry); + foreach ($this->validated() as $key => $entry) { + $descriptors[$key] = new SourceDescriptor( + key: $key, + title: $entry['title'], + accessUrl: $entry['access_url'], + crs: $entry['crs'], + model: $entry['model'], + description: $entry['description'], + publisher: $entry['publisher'], + contact: $entry['contact'], + landingPage: $entry['landing_page'], + mediaType: $entry['media_type'], + updateFrequency: $entry['update_frequency'], + licence: $entry['licence'], + omittedFields: $entry['omitted_fields'], + ); } return $descriptors; } /** - * @param array $entry - */ - private function descriptor(string $key, array $entry): SourceDescriptor - { - return new SourceDescriptor( - key: $key, - title: $this->required($key, $entry, 'title'), - accessUrl: $this->required($key, $entry, 'access_url'), - crs: $this->required($key, $entry, 'crs'), - model: $this->required($key, $entry, 'model'), - description: $this->optional($key, $entry, 'description'), - publisher: $this->optional($key, $entry, 'publisher'), - contact: $this->optional($key, $entry, 'contact'), - landingPage: $this->optional($key, $entry, 'landing_page'), - mediaType: $this->optional($key, $entry, 'media_type'), - updateFrequency: $this->optional($key, $entry, 'update_frequency'), - licence: $this->optional($key, $entry, 'licence'), - omittedFields: $this->omittedFields($key, $entry), - ); - } - - /** - * @param array $entry - */ - private function required(string $key, array $entry, string $field): string - { - return $this->optional($key, $entry, $field) - ?? throw new \RuntimeException(\sprintf('Entry "%s" in %s is missing the required field "%s"; an import cannot run without it.', $key, $this->manifest, $field)); - } - - /** - * An empty value means "not filled in", the same as an absent key, so both - * become null rather than an empty string. - * - * @param array $entry + * @return array}> */ - private function optional(string $key, array $entry, string $field): ?string + private function validated(): array { - $value = $entry[$field] ?? null; - - if (null === $value || '' === $value) { - return null; + if (!is_file($this->manifest)) { + throw new \RuntimeException(\sprintf('Source manifest "%s" does not exist.', $this->manifest)); } - if (!is_scalar($value)) { - throw new \RuntimeException(\sprintf('Field "%s" of entry "%s" in %s must be a single value, got %s.', $field, $key, $this->manifest, get_debug_type($value))); + try { + $parsed = Yaml::parseFile($this->manifest); + } catch (ParseException $exception) { + throw new \RuntimeException(\sprintf('Source manifest "%s" is not valid YAML: %s', $this->manifest, $exception->getMessage()), previous: $exception); } - return trim((string) $value); - } - - /** - * @param array $entry - * - * @return array - */ - private function omittedFields(string $key, array $entry): array - { - $omitted = $entry['omitted_fields'] ?? []; - - if (!\is_array($omitted)) { - throw new \RuntimeException(\sprintf('Field "omitted_fields" of entry "%s" in %s must map each field to the reason it is not published.', $key, $this->manifest)); + // The tree is rooted at the entries themselves, so it never sees the + // key holding them. Without this the wrong shape yields an empty + // catalogue, which reads as "no data sets are registered" rather than + // as a broken file. + $sources = \is_array($parsed) ? $parsed['sources'] ?? null : null; + if (!\is_array($sources)) { + throw new \RuntimeException(\sprintf('Source manifest "%s" must contain a "sources" mapping at the top level.', $this->manifest)); } - $reasons = []; - foreach ($omitted as $field => $reason) { - // The reason is the half of the record that cannot be recovered - // from the code, so a bare list of names is not accepted. - if (!\is_string($reason) || '' === trim($reason)) { - throw new \RuntimeException(\sprintf('Omitted field "%s" of entry "%s" in %s needs a reason.', $field, $key, $this->manifest)); - } - - $reasons[(string) $field] = trim($reason); + try { + /** @var array}> $processed */ + $processed = new Processor()->processConfiguration(new SourceManifestConfiguration(), [$sources]); + } catch (InvalidConfigurationException $exception) { + throw new \RuntimeException(\sprintf('Source manifest "%s" is invalid: %s', $this->manifest, $exception->getMessage()), previous: $exception); } - return $reasons; + return $processed; } } diff --git a/src/Source/SourceCatalogWarmer.php b/src/Source/SourceCatalogWarmer.php new file mode 100644 index 0000000..c262f5d --- /dev/null +++ b/src/Source/SourceCatalogWarmer.php @@ -0,0 +1,38 @@ +catalog->all(); + + return []; + } +} diff --git a/src/Source/SourceManifestConfiguration.php b/src/Source/SourceManifestConfiguration.php new file mode 100644 index 0000000..05383d6 --- /dev/null +++ b/src/Source/SourceManifestConfiguration.php @@ -0,0 +1,115 @@ +getRootNode() + // The import selects a data set by its key exactly as written in + // the manifest. Key normalization rewrites a key that contains + // dashes and no underscore, which breaks that lookup silently. + ->normalizeKeys(false) + ->requiresAtLeastOneElement() + ->arrayPrototype() + ->beforeNormalization() + ->always(self::blankToNull()) + ->end() + ->children() + ->scalarNode('title') + ->isRequired() + ->cannotBeEmpty() + ->info('What the data set is called where it is published.') + ->end() + ->scalarNode('access_url') + ->isRequired() + ->cannotBeEmpty() + ->info('Where the feed is read from; an import cannot run without it.') + ->end() + ->scalarNode('crs') + ->isRequired() + ->cannotBeEmpty() + ->info('The CRS the feed publishes coordinates in, e.g. "EPSG:25832". A wrong value yields well-formed coordinates in the wrong place.') + ->end() + ->scalarNode('model') + ->isRequired() + ->cannotBeEmpty() + ->info('Smart Data Model the data set is published as.') + ->end() + ->scalarNode('description')->defaultNull()->end() + ->scalarNode('publisher')->defaultNull()->end() + ->scalarNode('contact')->defaultNull()->end() + ->scalarNode('landing_page')->defaultNull()->end() + ->scalarNode('media_type')->defaultNull()->end() + ->scalarNode('update_frequency')->defaultNull()->end() + ->scalarNode('licence') + ->defaultNull() + ->info('Empty means the terms are unsettled; DCAT-AP requires one before the data set can be registered.') + ->end() + ->arrayNode('omitted_fields') + ->defaultValue([]) + ->normalizeKeys(false) + ->scalarPrototype() + ->beforeNormalization() + ->always(self::blankToNullValue()) + ->end() + ->cannotBeEmpty() + ->info('The reason a field is withheld cannot be recovered from the code, so a bare list of names is not accepted.') + ->end() + ->end() + ->end() + ->end(); + + return $tree; + } + + /** + * An empty value means "not filled in", the same as an absent key, so both + * become null rather than an empty string. + * + * Normalization runs before the type check, so an entry that is not a + * mapping has to pass through untouched for the tree to report it as one. + */ + private static function blankToNull(): \Closure + { + return static function (mixed $entry): mixed { + if (!\is_array($entry)) { + return $entry; + } + + return array_map(self::blankToNullValue(), $entry); + }; + } + + private static function blankToNullValue(): \Closure + { + return static function (mixed $value): mixed { + if (!\is_string($value)) { + return $value; + } + + $value = trim($value); + + return '' === $value ? null : $value; + }; + } +} diff --git a/tests/Source/SourceCatalogTest.php b/tests/Source/SourceCatalogTest.php index e5f1dbd..61a0f57 100644 --- a/tests/Source/SourceCatalogTest.php +++ b/tests/Source/SourceCatalogTest.php @@ -84,7 +84,7 @@ public function testItRejectsAnEntryMissingAFieldTheImportNeeds(): void YAML)); $this->expectException(\RuntimeException::class); - $this->expectExceptionMessage('missing the required field "crs"'); + $this->expectExceptionMessage('The child config "crs" under "sources.a-source" must be configured'); $catalog->all(); } @@ -103,11 +103,98 @@ public function testItRejectsAnOmittedFieldWithoutAReason(): void YAML)); $this->expectException(\RuntimeException::class); - $this->expectExceptionMessage('needs a reason'); + $this->expectExceptionMessage('"sources.a-source.omitted_fields.some_field" cannot contain an empty value'); $catalog->all(); } + /** + * The import selects a data set by the key written in the manifest, and the + * config tree rewrites a key that has dashes and no underscore unless told + * otherwise. A rewritten key stops matching without saying so. + */ + public function testItKeepsADashedSourceKeyIntact(): void + { + $catalog = new SourceCatalog($this->manifest(<<<'YAML' + sources: + handicap-parking: + title: A source + access_url: https://example.com/feed.json + crs: EPSG:25832 + model: Example + YAML)); + + $this->assertSame(['handicap-parking'], array_keys($catalog->all())); + $this->assertSame('handicap-parking', $catalog->get('handicap-parking')->key); + } + + /** + * A misspelled optional field was dropped in silence before, which loses a + * fact the record exists to carry. + */ + public function testItRejectsAFieldTheManifestDoesNotDefine(): void + { + $catalog = new SourceCatalog($this->manifest(<<<'YAML' + sources: + a-source: + title: A source + access_url: https://example.com/feed.json + crs: EPSG:25832 + model: Example + license: CC-BY-4.0 + YAML)); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('Unrecognized option "license" under "sources.a-source"'); + + $catalog->all(); + } + + public function testItRejectsAnEntryThatIsNotAMapping(): void + { + $catalog = new SourceCatalog($this->manifest("sources:\n a-source: just a string\n")); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('Invalid type for path "sources.a-source"'); + + $catalog->all(); + } + + public function testItRejectsAManifestThatRegistersNothing(): void + { + $catalog = new SourceCatalog($this->manifest("sources: {}\n")); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('should have at least 1 element'); + + $catalog->all(); + } + + /** + * An empty value means "not filled in", the same as an absent key, so an + * unanswered question reads the same either way. + */ + public function testItReadsABlankOptionalFieldAsUnknown(): void + { + $catalog = new SourceCatalog($this->manifest(<<<'YAML' + sources: + a-source: + title: A source + access_url: https://example.com/feed.json + crs: EPSG:25832 + model: Example + publisher: ' Aarhus Kommune ' + contact: '' + licence: ~ + YAML)); + + $descriptor = $catalog->get('a-source'); + + $this->assertSame('Aarhus Kommune', $descriptor->publisher); + $this->assertNull($descriptor->contact); + $this->assertNull($descriptor->licence); + } + public function testItReportsAManifestThatIsNotThere(): void { $catalog = new SourceCatalog('/no/such/sources.yaml'); diff --git a/tests/Source/SourceCatalogWarmerTest.php b/tests/Source/SourceCatalogWarmerTest.php new file mode 100644 index 0000000..02294d7 --- /dev/null +++ b/tests/Source/SourceCatalogWarmerTest.php @@ -0,0 +1,86 @@ + */ + private array $written = []; + + protected function tearDown(): void + { + foreach ($this->written as $path) { + if (is_file($path)) { + unlink($path); + } + } + + $this->written = []; + } + + /** + * An optional warmer can be skipped, and a manifest that is only read when + * an import selects it is exactly what warming exists to avoid. + */ + public function testItIsNotOptional(): void + { + $this->assertFalse($this->warmer(\dirname(__DIR__, 2).'/config/sources.yaml')->isOptional()); + } + + public function testItWarmsTheShippedManifestWithoutPreloadingAnything(): void + { + $warmer = $this->warmer(\dirname(__DIR__, 2).'/config/sources.yaml'); + + $this->assertSame([], $warmer->warmUp(sys_get_temp_dir(), sys_get_temp_dir())); + } + + /** + * The reason for warming at all: an entry no import selects still fails the + * build rather than waiting to be discovered. + */ + public function testItFailsOnAnEntryNoImportWouldReach(): void + { + $warmer = $this->warmer($this->manifest(<<<'YAML' + sources: + a-source: + title: A source + access_url: https://example.com/feed.json + crs: EPSG:25832 + model: Example + unreached-source: + title: Another source + access_url: https://example.com/other.json + model: Example + YAML)); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('The child config "crs" under "sources.unreached-source" must be configured'); + + $warmer->warmUp(sys_get_temp_dir(), sys_get_temp_dir()); + } + + private function warmer(string $manifest): SourceCatalogWarmer + { + return new SourceCatalogWarmer(new SourceCatalog($manifest)); + } + + private function manifest(string $yaml): string + { + $path = tempnam(sys_get_temp_dir(), 'sources-'); + + if (false === $path) { + $this->fail('Could not create a temporary manifest.'); + } + + file_put_contents($path, $yaml); + $this->written[] = $path; + + return $path; + } +} From 02cc6cf683bcb7de4ae9e1db665ef56c385c63a2 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 14:52:07 +0200 Subject: [PATCH 38/54] Fixed error in pull request template --- .github/PULL_REQUEST_TEMPLATE.md | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index e39b258..333ba2f 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,4 +1,3 @@ - Date: Mon, 7 Sep 2026 15:42:52 +0200 Subject: [PATCH 39/54] Corrected adr 007 --- docs/adr/007-source-manifest.md | 128 ++++++++++---------------------- 1 file changed, 41 insertions(+), 87 deletions(-) diff --git a/docs/adr/007-source-manifest.md b/docs/adr/007-source-manifest.md index d644c8f..7491f71 100644 --- a/docs/adr/007-source-manifest.md +++ b/docs/adr/007-source-manifest.md @@ -10,109 +10,63 @@ ## Context -Each published data set carries facts the import needs — where the feed is read -from, the coordinate reference system its coordinates are in, and the model it -is published as — and facts only people need: who owns the data, on what terms -it may be republished, how often it changes, and which of its fields are -deliberately not published, with the reason for each. +This application publishes data sets it does not own. Each needs a +description — where it comes from, how to read it, who owns it, on what terms +it may be republished. -The first group must be readable by code. The second is what a public data -portal requires at registration, and what a data owner asks for when -establishing what happened to their data. Both grow with the number of data -sets. - -Recording the two groups separately produces a machine-readable value and a -written description of the same value, which can then disagree. Recording only -the first leaves the rest unwritten, and the questions it answers are then -answered from memory. - -This ADR serves to decide where a data set's own facts are recorded. +This ADR serves to decide where the data sets and their descriptions live. ### Drivers -- **Functional:** the code reads the facts it needs from the same record a - person reads; a data set is registerable on a public portal without a fresh - survey; an incomplete record fails the import that needs it rather than - producing an incomplete publication. -- **Non-functional:** adding a data set requires no deployment change; the - record is reviewable as a diff; no fact exists in two places. +- **Functional:** each data set is listed with what is needed to read and + republish it. +- **Non-functional:** adding a data set needs no deployment change, and its + description is reviewable as a diff. ### Options Considered -1. **One environment variable per data set.** Follows the convention that - configuration belongs in the environment, and lets a value differ per - environment. But variable names grow with the catalogue, so each new data - set becomes a deployment change; the environment carries strings only, - leaving metadata beyond an address nowhere to live; and values are invisible - in review, so a wrong one is found by running the import. -2. **Every fact in the class that maps the data set.** Nothing can diverge, - there being one copy, and the language enforces its presence. But metadata - is then readable only by opening code, extracting it for portal registration - requires writing an extractor, and correcting a licence or a contact becomes - a code change reviewed as one. -3. **A committed manifest the code reads.** One record per data set, keyed by - the identifier the import selects it with, holding the facts the code needs - beside those it does not, in a shape a catalogue profile can be generated - from. Validating the record becomes work of our own, and the values are - identical in every environment. +1. **One environment variable per data set.** Variable names grow with the + catalogue, and the environment holds only strings. +2. **Every fact in the class that maps the data set.** Nothing can diverge, but + the description is readable only by opening code, and correcting a licence + becomes a code change. +3. **A committed manifest the code reads.** One record per data set, holding + the description beside the values the import needs; its shape has to be + declared. 4. **An external catalogue or registry service.** The eventual home of - published metadata, with search and harvesting already built. But it has to - be running for an import to work, it is a second system to operate, and it - must be populated before anything can be published from it — from records - that would have to live somewhere else in the meantime. + published metadata, but a second system to operate, populated before + anything can be published from it. ## Decision -Record each data set in a **committed manifest**, keyed by the identifier the -import selects it with, and read from it every fact the code needs. - -Four rules follow: - -1. **Record only what the code cannot state.** How a feed's fields map onto the - model, and every quirk of its shape, stay in the class that performs the - mapping. Restating them in the manifest recreates the divergence the - manifest exists to prevent. -2. **A fact both the code and a reader need is read from the manifest.** It is - not also written in code, in a comment, or in the README. -3. **An incomplete or malformed record is an error.** Fields an import cannot - run without are required, and their absence raises rather than defaulting. A - fact that is unknown is recorded as unknown, so the gap stays visible. -4. **Name the fields after the catalogue profile the data will be registered - under** — DCAT-AP — so that publication is a translation rather than a - redesign. - -Rationale: - -- What has to be prevented is a value diverging from its description, so the - two belong to the same record. -- Portal registration and answering a data owner need the same fields, and - neither can be derived from mapping code. -- A record is reviewed alongside the class that consumes it, so a reviewer sees - the address, the reference system and the model together with the mapping - that assumes them. -- Where a feed is read from is a fact about the data set, not about the machine - running the import, so the environment is the wrong place for it. +Keep one **committed manifest** listing every data set this application +publishes, keyed by the identifier the import selects a data set by, and read +from it every fact the code needs. + +1. **Record only what the code cannot state.** Field mappings and feed quirks + stay in the class that maps the data set. +2. **A fact the manifest records is not restated elsewhere**, in code, a + comment, or documentation. +3. **An incomplete record is an error.** Required fields raise rather than + default, an unknown fact is recorded as unknown, and the shape is a schema + the framework validates when the application is built. +4. **Name the fields after DCAT-AP**, the profile the data is registered under, + so publication is a translation. ## Consequences ### Positive -- One record per data set, reviewable as a diff and versioned with the code. -- Adding a data set is one class and one record, with no deployment change. -- Registration on a public portal is a translation of records that exist. -- Licence, ownership and what was withheld have a single answer, and an - unanswered question shows as an empty value rather than as nothing at all. -- The same records can generate catalogue entities later without introducing a - second source of truth. +- Every data set is listed in one place, reviewable as a diff and versioned + with the code. +- Portal registration and owner questions translate records that already exist. +- A wrong record, or a field the manifest does not define, fails the build. ### Negative / Trade-offs -- **Values are identical in every environment.** Pointing a data set at a copy - for testing means editing a committed file. That is consistent with reading - feeds where they live, but it removes an escape hatch the environment offered. -- **A wrong reference system or model in the manifest is as damaging as a wrong - one in code, while looking less like code.** A wrong reference system yields - coordinates that are well-formed and in the wrong place. -- **Fields no code reads have nothing keeping them current.** Until catalogue - entities are generated from them, only review does. -- Validating the record is work the environment did not require. +- Values are identical in every environment, so pointing a data set at a test + copy means editing a committed file. +- A wrong reference system looks less like code than it is, and yields + coordinates that are well-formed and misplaced. +- A malformed record blocks every build, not just the import that reads it. +- Fields no code reads have only review keeping them current. From 538ae480723af9d2184a460fb49ed6bd97004c82 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 15:48:02 +0200 Subject: [PATCH 40/54] ADR 007 edits --- docs/adr/007-source-manifest.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/docs/adr/007-source-manifest.md b/docs/adr/007-source-manifest.md index 7491f71..47fcea2 100644 --- a/docs/adr/007-source-manifest.md +++ b/docs/adr/007-source-manifest.md @@ -10,31 +10,28 @@ ## Context -This application publishes data sets it does not own. Each needs a -description — where it comes from, how to read it, who owns it, on what terms -it may be republished. - -This ADR serves to decide where the data sets and their descriptions live. +This application publishes data sets it does not own. This ADR serves +to decide where the data sets and their specifications live. ### Drivers - **Functional:** each data set is listed with what is needed to read and republish it. - **Non-functional:** adding a data set needs no deployment change, and its - description is reviewable as a diff. + specification is reviewable as a diff. ### Options Considered 1. **One environment variable per data set.** Variable names grow with the - catalogue, and the environment holds only strings. + collection, and the environment holds only strings. 2. **Every fact in the class that maps the data set.** Nothing can diverge, but - the description is readable only by opening code, and correcting a licence + the specification is readable only by opening code, and correcting a licence becomes a code change. 3. **A committed manifest the code reads.** One record per data set, holding - the description beside the values the import needs; its shape has to be + the specification beside the values the import needs; its shape has to be declared. 4. **An external catalogue or registry service.** The eventual home of - published metadata, but a second system to operate, populated before + published specifications, but a second system to operate, populated before anything can be published from it. ## Decision @@ -50,8 +47,8 @@ from it every fact the code needs. 3. **An incomplete record is an error.** Required fields raise rather than default, an unknown fact is recorded as unknown, and the shape is a schema the framework validates when the application is built. -4. **Name the fields after DCAT-AP**, the profile the data is registered under, - so publication is a translation. +4. **Structure each record after DCAT-AP**, the specification the data will be + published under, so publication is a translation rather than a redesign. ## Consequences @@ -59,7 +56,8 @@ from it every fact the code needs. - Every data set is listed in one place, reviewable as a diff and versioned with the code. -- Portal registration and owner questions translate records that already exist. +- Publishing a specification onward, and answering the data owner, translate + records that already exist. - A wrong record, or a field the manifest does not define, fails the build. ### Negative / Trade-offs From cda1ba2f26b6e94d51f6b9d35a26c6c5692962b0 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 15:48:51 +0200 Subject: [PATCH 41/54] Raname calls to renamed methods --- src/Geo/Wgs84Transformer.php | 4 +++- src/Source/DataSourceReader.php | 10 +++++----- src/Source/MtmSpatialMaps/HandicapParking.php | 12 ++++++------ tests/Command/ImportCommandTest.php | 2 +- tests/Geo/Wgs84TransformerTest.php | 14 +++++++------- 5 files changed, 22 insertions(+), 20 deletions(-) diff --git a/src/Geo/Wgs84Transformer.php b/src/Geo/Wgs84Transformer.php index 84b7593..9f57abe 100644 --- a/src/Geo/Wgs84Transformer.php +++ b/src/Geo/Wgs84Transformer.php @@ -127,11 +127,13 @@ public function transformGeometry(string $srid, array $geometry): array * of those for Polygon, and so on. Recursing until the first element is * numeric handles every depth without enumerating the types. * + * @param array $coordinates + * * @return array */ private function transformCoordinates(string $srid, array $coordinates): array { - if (!\is_array($coordinates) || [] === $coordinates) { + if ([] === $coordinates) { throw new \InvalidArgumentException('GeoJSON coordinates must be a non-empty array.'); } diff --git a/src/Source/DataSourceReader.php b/src/Source/DataSourceReader.php index c84ce03..c8f3751 100644 --- a/src/Source/DataSourceReader.php +++ b/src/Source/DataSourceReader.php @@ -25,23 +25,23 @@ public function __construct( */ public function read(string $url): array { - if (!str_starts_with($location, 'http://') && !str_starts_with($location, 'https://')) { - throw new \RuntimeException(\sprintf('Feed location must be an http(s) URL, got "%s".', $location)); + if (!str_starts_with($url, 'http://') && !str_starts_with($url, 'https://')) { + throw new \RuntimeException(\sprintf('Feed location must be an http(s) URL, got "%s".', $url)); } - $json = $this->fetch($location); + $json = $this->fetch($url); try { $decoded = json_decode($json, true, 512, \JSON_THROW_ON_ERROR); } catch (\JsonException $exception) { - throw new \RuntimeException(\sprintf('Invalid JSON in "%s": %s', $location, $exception->getMessage()), previous: $exception); + throw new \RuntimeException(\sprintf('Invalid JSON in "%s": %s', $url, $exception->getMessage()), previous: $exception); } // A JSON document may legally be a scalar. Every feed we consume is a // list or an object, and a scalar here means the location is wrong // rather than that the feed is empty. if (!\is_array($decoded)) { - throw new \RuntimeException(\sprintf('Expected a JSON array or object in "%s", got %s.', $location, get_debug_type($decoded))); + throw new \RuntimeException(\sprintf('Expected a JSON array or object in "%s", got %s.', $url, get_debug_type($decoded))); } return $decoded; diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index 441a452..27bcdcd 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -78,12 +78,12 @@ private function toEntity(array $feature, SourceDescriptor $source): ?NgsiEntity // models have separate category enums, so values are not interchangeable. // `onStreet` is dropped because the entity type already states it. return $entity - ->property('name', $this->address($row)) - ->property('description', trim((string) ($row['bemrk'] ?? ''))) - ->property('category', ['forDisabled']) - ->property('totalSpotNumber', (int) ($row['invalidepladser'] ?? 0)) - ->property('source', $source->accessUrl) - ->geoProperty('location', $this->transformer->geometry($source->crs, $geometry)); + ->setProperty('name', $this->address($row)) + ->setProperty('description', trim((string) ($row['bemrk'] ?? ''))) + ->setProperty('category', ['forDisabled']) + ->setProperty('totalSpotNumber', (int) ($row['invalidepladser'] ?? 0)) + ->setProperty('source', $source->accessUrl) + ->geoProperty('location', $this->transformer->transformGeometry($source->crs, $geometry)); } /** diff --git a/tests/Command/ImportCommandTest.php b/tests/Command/ImportCommandTest.php index ae858a3..a1d68ad 100644 --- a/tests/Command/ImportCommandTest.php +++ b/tests/Command/ImportCommandTest.php @@ -121,7 +121,7 @@ public function testItRejectsAnUnknownSource(): void public function testDryRunPrintsThePayloadAndSendsNothing(): void { $entity = new NgsiEntity('urn:ngsi-ld:Example:1', 'Example') - ->property('name', 'Example'); + ->setProperty('name', 'Example'); $tester = $this->tester([$this->source('one-entity', $entity)]); diff --git a/tests/Geo/Wgs84TransformerTest.php b/tests/Geo/Wgs84TransformerTest.php index aacf039..2f2192b 100644 --- a/tests/Geo/Wgs84TransformerTest.php +++ b/tests/Geo/Wgs84TransformerTest.php @@ -93,7 +93,7 @@ public function testItAcceptsAdditionalDefinitions(): void public function testItReprojectsAPointGeometry(): void { - $geometry = $this->transformer->geometry(self::UTM32, [ + $geometry = $this->transformer->transformGeometry(self::UTM32, [ 'type' => 'Point', 'coordinates' => [self::REFERENCE_EASTING, self::REFERENCE_NORTHING], ]); @@ -104,7 +104,7 @@ public function testItReprojectsAPointGeometry(): void public function testItReprojectsALineString(): void { - $geometry = $this->transformer->geometry(self::UTM32, [ + $geometry = $this->transformer->transformGeometry(self::UTM32, [ 'type' => 'LineString', 'coordinates' => [ [574108.2557507273, 6222343.6199512165], @@ -120,7 +120,7 @@ public function testItReprojectsALineString(): void public function testItReprojectsAPolygonPreservingNesting(): void { - $geometry = $this->transformer->geometry(self::UTM32, [ + $geometry = $this->transformer->transformGeometry(self::UTM32, [ 'type' => 'Polygon', 'coordinates' => [ [ @@ -147,7 +147,7 @@ public function testItReprojectsAMultiPolygon(): void [574108.0, 6222343.0], ]; - $geometry = $this->transformer->geometry(self::UTM32, [ + $geometry = $this->transformer->transformGeometry(self::UTM32, [ 'type' => 'MultiPolygon', 'coordinates' => [[$ring], [$ring]], ]); @@ -160,14 +160,14 @@ public function testItRejectsAGeometryWithoutCoordinates(): void { $this->expectException(\InvalidArgumentException::class); - $this->transformer->geometry(self::UTM32, ['type' => 'Point']); + $this->transformer->transformGeometry(self::UTM32, ['type' => 'Point']); } public function testItRejectsAGeometryCollection(): void { $this->expectException(\InvalidArgumentException::class); - $this->transformer->geometry(self::UTM32, [ + $this->transformer->transformGeometry(self::UTM32, [ 'type' => 'GeometryCollection', 'geometries' => [], ]); @@ -177,6 +177,6 @@ public function testItRejectsAPositionWithASingleOrdinate(): void { $this->expectException(\InvalidArgumentException::class); - $this->transformer->geometry(self::UTM32, ['type' => 'Point', 'coordinates' => [574108.0]]); + $this->transformer->transformGeometry(self::UTM32, ['type' => 'Point', 'coordinates' => [574108.0]]); } } From 365325a037c8bb87612f5ad49dd223b174abd94c Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Mon, 7 Sep 2026 15:49:31 +0200 Subject: [PATCH 42/54] Coding standards --- docs/adr/007-source-manifest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/007-source-manifest.md b/docs/adr/007-source-manifest.md index 47fcea2..8bb90c3 100644 --- a/docs/adr/007-source-manifest.md +++ b/docs/adr/007-source-manifest.md @@ -10,7 +10,7 @@ ## Context -This application publishes data sets it does not own. This ADR serves +This application publishes data sets it does not own. This ADR serves to decide where the data sets and their specifications live. ### Drivers From 0ed759862c94c51100aa17f1a57940557278b389 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 09:47:31 +0200 Subject: [PATCH 43/54] Grouped the manifest classes under App\Source\Manifest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Catalog, Descriptor, Schema and Validator now sit together, which drops the repeated Source prefix. The cache warmer became Validator: it caches nothing, and exists so a malformed record fails the build. Schema no longer implements ConfigurationInterface — that is a bundle extension contract, not a description of a committed data file — so the catalogue hands the built tree to Processor::process() instead. --- .../Catalog.php} | 27 ++++++------- .../Descriptor.php} | 4 +- .../Schema.php} | 13 ++++--- src/Source/Manifest/Validator.php | 34 +++++++++++++++++ src/Source/MtmSpatialMaps/HandicapParking.php | 8 ++-- src/Source/SourceCatalogWarmer.php | 38 ------------------- .../CatalogTest.php} | 32 ++++++++-------- .../ValidatorTest.php} | 34 +++++++++-------- .../MtmSpatialMaps/HandicapParkingTest.php | 8 ++-- 9 files changed, 100 insertions(+), 98 deletions(-) rename src/Source/{SourceCatalog.php => Manifest/Catalog.php} (84%) rename src/Source/{SourceDescriptor.php => Manifest/Descriptor.php} (95%) rename src/Source/{SourceManifestConfiguration.php => Manifest/Schema.php} (92%) create mode 100644 src/Source/Manifest/Validator.php delete mode 100644 src/Source/SourceCatalogWarmer.php rename tests/Source/{SourceCatalogTest.php => Manifest/CatalogTest.php} (85%) rename tests/Source/{SourceCatalogWarmerTest.php => Manifest/ValidatorTest.php} (58%) diff --git a/src/Source/SourceCatalog.php b/src/Source/Manifest/Catalog.php similarity index 84% rename from src/Source/SourceCatalog.php rename to src/Source/Manifest/Catalog.php index 2b385df..41baa06 100644 --- a/src/Source/SourceCatalog.php +++ b/src/Source/Manifest/Catalog.php @@ -2,7 +2,7 @@ declare(strict_types=1); -namespace App\Source; +namespace App\Source\Manifest; use Symfony\Component\Config\Definition\Exception\InvalidConfigurationException; use Symfony\Component\Config\Definition\Processor; @@ -13,18 +13,19 @@ /** * The manifest of data sets this application publishes. * - * The record's shape is a Symfony config tree rather than hand-written checks, - * and SourceCatalogWarmer reads the whole manifest during container warm-up, so - * a malformed entry fails the build instead of waiting for the one import that - * happens to select it. + * The record's shape is declared in Schema rather than checked by hand. Parsing + * is deferred to first use, and Validator reads every record when the + * application is built, so a malformed entry fails the build instead of waiting + * for the one import that happens to select it. * * @see config/sources.yaml - * @see SourceManifestConfiguration + * @see Schema + * @see Validator * @see docs/adr/007-source-manifest.md */ -final class SourceCatalog +final class Catalog { - /** @var array|null */ + /** @var array|null */ private ?array $descriptors = null; public function __construct( @@ -36,7 +37,7 @@ public function __construct( /** * @throws \RuntimeException when the manifest cannot be read, or carries no entry for the key */ - public function get(string $key): SourceDescriptor + public function get(string $key): Descriptor { $descriptors = $this->all(); @@ -48,7 +49,7 @@ public function get(string $key): SourceDescriptor } /** - * @return array keyed by source key + * @return array keyed by source key * * @throws \RuntimeException when the manifest cannot be read */ @@ -58,14 +59,14 @@ public function all(): array } /** - * @return array + * @return array */ private function load(): array { $descriptors = []; foreach ($this->validated() as $key => $entry) { - $descriptors[$key] = new SourceDescriptor( + $descriptors[$key] = new Descriptor( key: $key, title: $entry['title'], accessUrl: $entry['access_url'], @@ -111,7 +112,7 @@ private function validated(): array try { /** @var array}> $processed */ - $processed = new Processor()->processConfiguration(new SourceManifestConfiguration(), [$sources]); + $processed = new Processor()->process(Schema::tree(), [$sources]); } catch (InvalidConfigurationException $exception) { throw new \RuntimeException(\sprintf('Source manifest "%s" is invalid: %s', $this->manifest, $exception->getMessage()), previous: $exception); } diff --git a/src/Source/SourceDescriptor.php b/src/Source/Manifest/Descriptor.php similarity index 95% rename from src/Source/SourceDescriptor.php rename to src/Source/Manifest/Descriptor.php index 83b2b14..34a1e2b 100644 --- a/src/Source/SourceDescriptor.php +++ b/src/Source/Manifest/Descriptor.php @@ -2,7 +2,7 @@ declare(strict_types=1); -namespace App\Source; +namespace App\Source\Manifest; /** * One entry from the source manifest: what a data set is, where it is read @@ -15,7 +15,7 @@ * @see config/sources.yaml * @see docs/adr/007-source-manifest.md */ -final readonly class SourceDescriptor +final readonly class Descriptor { /** * @param string $key identifier the import selects this data set by diff --git a/src/Source/SourceManifestConfiguration.php b/src/Source/Manifest/Schema.php similarity index 92% rename from src/Source/SourceManifestConfiguration.php rename to src/Source/Manifest/Schema.php index 05383d6..6d20aa3 100644 --- a/src/Source/SourceManifestConfiguration.php +++ b/src/Source/Manifest/Schema.php @@ -2,10 +2,10 @@ declare(strict_types=1); -namespace App\Source; +namespace App\Source\Manifest; use Symfony\Component\Config\Definition\Builder\TreeBuilder; -use Symfony\Component\Config\Definition\ConfigurationInterface; +use Symfony\Component\Config\Definition\NodeInterface; /** * The shape of the source manifest, as a Symfony config tree. @@ -15,12 +15,15 @@ * suffixed to that error as a hint, which is the moment a field's rationale is * needed. * + * Deliberately not a ConfigurationInterface: that is a bundle extension's + * contract, and this schema describes a committed data file instead. + * * @see config/sources.yaml * @see docs/adr/007-source-manifest.md */ -final readonly class SourceManifestConfiguration implements ConfigurationInterface +final readonly class Schema { - public function getConfigTreeBuilder(): TreeBuilder + public static function tree(): NodeInterface { $tree = new TreeBuilder('sources'); @@ -79,7 +82,7 @@ public function getConfigTreeBuilder(): TreeBuilder ->end() ->end(); - return $tree; + return $tree->buildTree(); } /** diff --git a/src/Source/Manifest/Validator.php b/src/Source/Manifest/Validator.php new file mode 100644 index 0000000..01d54dd --- /dev/null +++ b/src/Source/Manifest/Validator.php @@ -0,0 +1,34 @@ + nothing is written, so nothing is preloaded + */ + public function warmUp(string $cacheDir, ?string $buildDir = null): array + { + $this->catalog->all(); + + return []; + } +} diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index 27bcdcd..b170ff0 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -7,8 +7,8 @@ use App\Geo\Wgs84Transformer; use App\Ngsi\NgsiEntity; use App\Source\DataSourceReader; -use App\Source\SourceCatalog; -use App\Source\SourceDescriptor; +use App\Source\Manifest\Catalog; +use App\Source\Manifest\Descriptor; use App\Source\SourceInterface; /** @@ -24,7 +24,7 @@ public function __construct( private DataSourceReader $reader, private Wgs84Transformer $transformer, - private SourceCatalog $catalog, + private Catalog $catalog, ) { } @@ -50,7 +50,7 @@ public function entities(): iterable /** * @param array $feature GeoJSON Feature */ - private function toEntity(array $feature, SourceDescriptor $source): ?NgsiEntity + private function toEntity(array $feature, Descriptor $source): ?NgsiEntity { // A Feature keeps its attributes under `properties` and its geometry // beside them, so neither is at the feature's top level. diff --git a/src/Source/SourceCatalogWarmer.php b/src/Source/SourceCatalogWarmer.php deleted file mode 100644 index c262f5d..0000000 --- a/src/Source/SourceCatalogWarmer.php +++ /dev/null @@ -1,38 +0,0 @@ -catalog->all(); - - return []; - } -} diff --git a/tests/Source/SourceCatalogTest.php b/tests/Source/Manifest/CatalogTest.php similarity index 85% rename from tests/Source/SourceCatalogTest.php rename to tests/Source/Manifest/CatalogTest.php index 61a0f57..bfa83b5 100644 --- a/tests/Source/SourceCatalogTest.php +++ b/tests/Source/Manifest/CatalogTest.php @@ -2,12 +2,12 @@ declare(strict_types=1); -namespace App\Tests\Source; +namespace App\Tests\Source\Manifest; -use App\Source\SourceCatalog; +use App\Source\Manifest\Catalog; use PHPUnit\Framework\TestCase; -class SourceCatalogTest extends TestCase +class CatalogTest extends TestCase { /** @var list */ private array $written = []; @@ -25,7 +25,7 @@ protected function tearDown(): void public function testTheShippedManifestIsUsable(): void { - $catalog = new SourceCatalog(\dirname(__DIR__, 2).'/config/sources.yaml'); + $catalog = new Catalog(\dirname(__DIR__, 3).'/config/sources.yaml'); $this->assertNotSame([], $catalog->all(), 'The manifest registers no data sets.'); } @@ -36,7 +36,7 @@ public function testTheShippedManifestIsUsable(): void */ public function testEveryShippedEntryCanBeImportedFrom(): void { - $catalog = new SourceCatalog(\dirname(__DIR__, 2).'/config/sources.yaml'); + $catalog = new Catalog(\dirname(__DIR__, 3).'/config/sources.yaml'); foreach ($catalog->all() as $key => $descriptor) { $this->assertSame($key, $descriptor->key); @@ -48,7 +48,7 @@ public function testEveryShippedEntryCanBeImportedFrom(): void public function testItNamesTheKnownEntriesWhenAskedForAnUnknownOne(): void { - $catalog = new SourceCatalog($this->manifest(<<<'YAML' + $catalog = new Catalog($this->manifest(<<<'YAML' sources: a-source: title: A source @@ -65,7 +65,7 @@ public function testItNamesTheKnownEntriesWhenAskedForAnUnknownOne(): void public function testItRejectsAManifestWithoutASourcesMapping(): void { - $catalog = new SourceCatalog($this->manifest("data_sets:\n a-source: {}\n")); + $catalog = new Catalog($this->manifest("data_sets:\n a-source: {}\n")); $this->expectException(\RuntimeException::class); $this->expectExceptionMessage('must contain a "sources" mapping'); @@ -75,7 +75,7 @@ public function testItRejectsAManifestWithoutASourcesMapping(): void public function testItRejectsAnEntryMissingAFieldTheImportNeeds(): void { - $catalog = new SourceCatalog($this->manifest(<<<'YAML' + $catalog = new Catalog($this->manifest(<<<'YAML' sources: a-source: title: A source @@ -91,7 +91,7 @@ public function testItRejectsAnEntryMissingAFieldTheImportNeeds(): void public function testItRejectsAnOmittedFieldWithoutAReason(): void { - $catalog = new SourceCatalog($this->manifest(<<<'YAML' + $catalog = new Catalog($this->manifest(<<<'YAML' sources: a-source: title: A source @@ -115,7 +115,7 @@ public function testItRejectsAnOmittedFieldWithoutAReason(): void */ public function testItKeepsADashedSourceKeyIntact(): void { - $catalog = new SourceCatalog($this->manifest(<<<'YAML' + $catalog = new Catalog($this->manifest(<<<'YAML' sources: handicap-parking: title: A source @@ -134,7 +134,7 @@ public function testItKeepsADashedSourceKeyIntact(): void */ public function testItRejectsAFieldTheManifestDoesNotDefine(): void { - $catalog = new SourceCatalog($this->manifest(<<<'YAML' + $catalog = new Catalog($this->manifest(<<<'YAML' sources: a-source: title: A source @@ -152,7 +152,7 @@ public function testItRejectsAFieldTheManifestDoesNotDefine(): void public function testItRejectsAnEntryThatIsNotAMapping(): void { - $catalog = new SourceCatalog($this->manifest("sources:\n a-source: just a string\n")); + $catalog = new Catalog($this->manifest("sources:\n a-source: just a string\n")); $this->expectException(\RuntimeException::class); $this->expectExceptionMessage('Invalid type for path "sources.a-source"'); @@ -162,7 +162,7 @@ public function testItRejectsAnEntryThatIsNotAMapping(): void public function testItRejectsAManifestThatRegistersNothing(): void { - $catalog = new SourceCatalog($this->manifest("sources: {}\n")); + $catalog = new Catalog($this->manifest("sources: {}\n")); $this->expectException(\RuntimeException::class); $this->expectExceptionMessage('should have at least 1 element'); @@ -176,7 +176,7 @@ public function testItRejectsAManifestThatRegistersNothing(): void */ public function testItReadsABlankOptionalFieldAsUnknown(): void { - $catalog = new SourceCatalog($this->manifest(<<<'YAML' + $catalog = new Catalog($this->manifest(<<<'YAML' sources: a-source: title: A source @@ -197,7 +197,7 @@ public function testItReadsABlankOptionalFieldAsUnknown(): void public function testItReportsAManifestThatIsNotThere(): void { - $catalog = new SourceCatalog('/no/such/sources.yaml'); + $catalog = new Catalog('/no/such/sources.yaml'); $this->expectException(\RuntimeException::class); $this->expectExceptionMessage('does not exist'); @@ -207,7 +207,7 @@ public function testItReportsAManifestThatIsNotThere(): void public function testItReportsUnparsableYaml(): void { - $catalog = new SourceCatalog($this->manifest("sources:\n - [unbalanced\n")); + $catalog = new Catalog($this->manifest("sources:\n - [unbalanced\n")); $this->expectException(\RuntimeException::class); $this->expectExceptionMessage('is not valid YAML'); diff --git a/tests/Source/SourceCatalogWarmerTest.php b/tests/Source/Manifest/ValidatorTest.php similarity index 58% rename from tests/Source/SourceCatalogWarmerTest.php rename to tests/Source/Manifest/ValidatorTest.php index 02294d7..ada913d 100644 --- a/tests/Source/SourceCatalogWarmerTest.php +++ b/tests/Source/Manifest/ValidatorTest.php @@ -2,13 +2,13 @@ declare(strict_types=1); -namespace App\Tests\Source; +namespace App\Tests\Source\Manifest; -use App\Source\SourceCatalog; -use App\Source\SourceCatalogWarmer; +use App\Source\Manifest\Catalog; +use App\Source\Manifest\Validator; use PHPUnit\Framework\TestCase; -class SourceCatalogWarmerTest extends TestCase +class ValidatorTest extends TestCase { /** @var list */ private array $written = []; @@ -25,28 +25,30 @@ protected function tearDown(): void } /** - * An optional warmer can be skipped, and a manifest that is only read when - * an import selects it is exactly what warming exists to avoid. + * A check that can be skipped is not a check. */ public function testItIsNotOptional(): void { - $this->assertFalse($this->warmer(\dirname(__DIR__, 2).'/config/sources.yaml')->isOptional()); + $this->assertFalse($this->validator(\dirname(__DIR__, 3).'/config/sources.yaml')->isOptional()); } - public function testItWarmsTheShippedManifestWithoutPreloadingAnything(): void + /** + * It validates rather than caches, so it leaves nothing behind to preload. + */ + public function testItAcceptsTheShippedManifestAndWritesNothing(): void { - $warmer = $this->warmer(\dirname(__DIR__, 2).'/config/sources.yaml'); + $validator = $this->validator(\dirname(__DIR__, 3).'/config/sources.yaml'); - $this->assertSame([], $warmer->warmUp(sys_get_temp_dir(), sys_get_temp_dir())); + $this->assertSame([], $validator->warmUp(sys_get_temp_dir(), sys_get_temp_dir())); } /** - * The reason for warming at all: an entry no import selects still fails the - * build rather than waiting to be discovered. + * The reason for checking at build time: an entry no import selects still + * fails the build rather than waiting to be discovered. */ public function testItFailsOnAnEntryNoImportWouldReach(): void { - $warmer = $this->warmer($this->manifest(<<<'YAML' + $validator = $this->validator($this->manifest(<<<'YAML' sources: a-source: title: A source @@ -62,12 +64,12 @@ public function testItFailsOnAnEntryNoImportWouldReach(): void $this->expectException(\RuntimeException::class); $this->expectExceptionMessage('The child config "crs" under "sources.unreached-source" must be configured'); - $warmer->warmUp(sys_get_temp_dir(), sys_get_temp_dir()); + $validator->warmUp(sys_get_temp_dir(), sys_get_temp_dir()); } - private function warmer(string $manifest): SourceCatalogWarmer + private function validator(string $manifest): Validator { - return new SourceCatalogWarmer(new SourceCatalog($manifest)); + return new Validator(new Catalog($manifest)); } private function manifest(string $yaml): string diff --git a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php index 4a67b14..6517824 100644 --- a/tests/Source/MtmSpatialMaps/HandicapParkingTest.php +++ b/tests/Source/MtmSpatialMaps/HandicapParkingTest.php @@ -7,9 +7,9 @@ use App\Geo\Wgs84Transformer; use App\Ngsi\NgsiEntity; use App\Source\DataSourceReader; +use App\Source\Manifest\Catalog; +use App\Source\Manifest\Descriptor; use App\Source\MtmSpatialMaps\HandicapParking; -use App\Source\SourceCatalog; -use App\Source\SourceDescriptor; use PHPUnit\Framework\TestCase; use Symfony\Component\HttpClient\MockHttpClient; use Symfony\Component\HttpClient\Response\MockResponse; @@ -22,7 +22,7 @@ class HandicapParkingTest extends TestCase { private const string KEY = 'mtm_spatialmaps-handicap-parking'; - private SourceDescriptor $source; + private Descriptor $source; private string $requestedUrl; @@ -31,7 +31,7 @@ class HandicapParkingTest extends TestCase protected function setUp(): void { - $catalog = new SourceCatalog(\dirname(__DIR__, 3).'/config/sources.yaml'); + $catalog = new Catalog(\dirname(__DIR__, 3).'/config/sources.yaml'); $this->source = $catalog->get(self::KEY); $client = new MockHttpClient(function (string $method, string $url): MockResponse { From 78c3538063736dbf0f98042d9224ea9c2cddc5c4 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 09:52:41 +0200 Subject: [PATCH 44/54] Cleaned up comments --- src/Source/Manifest/Catalog.php | 9 --------- src/Source/Manifest/Descriptor.php | 3 --- src/Source/Manifest/Schema.php | 8 -------- 3 files changed, 20 deletions(-) diff --git a/src/Source/Manifest/Catalog.php b/src/Source/Manifest/Catalog.php index 41baa06..722e589 100644 --- a/src/Source/Manifest/Catalog.php +++ b/src/Source/Manifest/Catalog.php @@ -13,11 +13,6 @@ /** * The manifest of data sets this application publishes. * - * The record's shape is declared in Schema rather than checked by hand. Parsing - * is deferred to first use, and Validator reads every record when the - * application is built, so a malformed entry fails the build instead of waiting - * for the one import that happens to select it. - * * @see config/sources.yaml * @see Schema * @see Validator @@ -101,10 +96,6 @@ private function validated(): array throw new \RuntimeException(\sprintf('Source manifest "%s" is not valid YAML: %s', $this->manifest, $exception->getMessage()), previous: $exception); } - // The tree is rooted at the entries themselves, so it never sees the - // key holding them. Without this the wrong shape yields an empty - // catalogue, which reads as "no data sets are registered" rather than - // as a broken file. $sources = \is_array($parsed) ? $parsed['sources'] ?? null : null; if (!\is_array($sources)) { throw new \RuntimeException(\sprintf('Source manifest "%s" must contain a "sources" mapping at the top level.', $this->manifest)); diff --git a/src/Source/Manifest/Descriptor.php b/src/Source/Manifest/Descriptor.php index 34a1e2b..13c66ad 100644 --- a/src/Source/Manifest/Descriptor.php +++ b/src/Source/Manifest/Descriptor.php @@ -5,9 +5,6 @@ namespace App\Source\Manifest; /** - * One entry from the source manifest: what a data set is, where it is read - * from, and on what terms. - * * Everything about a feed except its field mapping, which belongs to the * source class. The names follow DCAT-AP so that publishing the catalogue is a * rename rather than a second survey. diff --git a/src/Source/Manifest/Schema.php b/src/Source/Manifest/Schema.php index 6d20aa3..c4a977e 100644 --- a/src/Source/Manifest/Schema.php +++ b/src/Source/Manifest/Schema.php @@ -10,14 +10,6 @@ /** * The shape of the source manifest, as a Symfony config tree. * - * The tree is rooted at the entries rather than at the file, so a validation - * error names the path a maintainer sees in the manifest. Each `info()` is - * suffixed to that error as a hint, which is the moment a field's rationale is - * needed. - * - * Deliberately not a ConfigurationInterface: that is a bundle extension's - * contract, and this schema describes a committed data file instead. - * * @see config/sources.yaml * @see docs/adr/007-source-manifest.md */ From 05fa2386746f81b2887fe2910e05887c241f5d46 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 10:00:32 +0200 Subject: [PATCH 45/54] Minor correction in comment --- src/Source/Manifest/Descriptor.php | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/Source/Manifest/Descriptor.php b/src/Source/Manifest/Descriptor.php index 13c66ad..9f0e36b 100644 --- a/src/Source/Manifest/Descriptor.php +++ b/src/Source/Manifest/Descriptor.php @@ -6,8 +6,7 @@ /** * Everything about a feed except its field mapping, which belongs to the - * source class. The names follow DCAT-AP so that publishing the catalogue is a - * rename rather than a second survey. + * source class. The names follow DCAT-AP. * * @see config/sources.yaml * @see docs/adr/007-source-manifest.md From 603147dc81fa15be689805b3af22798271889808 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 11:23:18 +0200 Subject: [PATCH 46/54] Extracted the import logic into a DataSourceImporter service --- src/Import/DataSourceImporter.php | 132 ++++++++++++ src/Import/Exception/EmptySourceException.php | 20 ++ .../Exception/UnknownSourceException.php | 28 +++ .../Exception/UpsertFailedException.php | 19 ++ src/Import/ImportResult.php | 23 ++ tests/Import/DataSourceImporterTest.php | 201 ++++++++++++++++++ tests/Source/FakeSource.php | 61 ++++++ 7 files changed, 484 insertions(+) create mode 100644 src/Import/DataSourceImporter.php create mode 100644 src/Import/Exception/EmptySourceException.php create mode 100644 src/Import/Exception/UnknownSourceException.php create mode 100644 src/Import/Exception/UpsertFailedException.php create mode 100644 src/Import/ImportResult.php create mode 100644 tests/Import/DataSourceImporterTest.php create mode 100644 tests/Source/FakeSource.php diff --git a/src/Import/DataSourceImporter.php b/src/Import/DataSourceImporter.php new file mode 100644 index 0000000..6ae8f8c --- /dev/null +++ b/src/Import/DataSourceImporter.php @@ -0,0 +1,132 @@ + $sources + */ + public function __construct( + #[AutowireIterator('app.source')] + private iterable $sources, + private NgsiLdBroker $broker, + #[Autowire(env: 'ENTER_NGSI_CONTEXT_URLS')] + private string $contextUrls, + ) { + } + + /** + * @return list every registered source key, in registration order + */ + public function keys(): array + { + return array_keys($this->registry()); + } + + /** + * Converts a source to NGSI-LD without sending anything. + * + * @return non-empty-list> + * + * @throws UnknownSourceException when no source is registered under the key + * @throws EmptySourceException when the source yields no entities + */ + public function payload(string $key, ?int $limit = null): array + { + $registry = $this->registry(); + + if (!isset($registry[$key])) { + throw new UnknownSourceException($key, array_keys($registry)); + } + + // A limit below 1 is a mistyped option rather than a request to import + // nothing, and importing nothing is the one outcome this class refuses + // to report as a success. + $limit = null === $limit ? null : max(1, $limit); + $contexts = $this->contexts(); + + $payload = []; + foreach ($registry[$key]->entities() as $entity) { + $payload[] = $entity->toArray($contexts); + + // A source reads a whole feed lazily, so the limit stops the + // conversion instead of trimming its result. + if (null !== $limit && \count($payload) >= $limit) { + break; + } + } + + // A source that yields nothing is almost always misconfigured rather + // than genuinely empty, and it fails silently by construction: a + // record skipped for a missing field looks exactly like a feed with no + // records. Fail loudly so it cannot be mistaken for a successful run. + if ([] === $payload) { + throw new EmptySourceException($key); + } + + return $payload; + } + + /** + * @throws UnknownSourceException when no source is registered under the key + * @throws EmptySourceException when the source yields no entities + * @throws UpsertFailedException when the broker cannot be written to + */ + public function import(string $key, ?int $limit = null): ImportResult + { + $payload = $this->payload($key, $limit); + + try { + $status = $this->broker->upsert($payload); + } catch (\Throwable $exception) { + // A broker that is down or rejects the batch is an operational + // condition, not a bug in the conversion. Giving it a type of its + // own lets a caller report it plainly while the exceptions a + // broken feed raises — the ones worth a stack trace — pass through. + throw new UpsertFailedException($exception); + } + + return new ImportResult(\count($payload), $status, $this->broker->brokerUrl()); + } + + /** + * @return array keyed by source key + */ + private function registry(): array + { + $registry = []; + + foreach ($this->sources as $source) { + $registry[$source->key()] = $source; + } + + return $registry; + } + + /** + * @return list + */ + private function contexts(): array + { + return array_values(array_filter(array_map(trim(...), explode(',', $this->contextUrls)))); + } +} diff --git a/src/Import/Exception/EmptySourceException.php b/src/Import/Exception/EmptySourceException.php new file mode 100644 index 0000000..5a87617 --- /dev/null +++ b/src/Import/Exception/EmptySourceException.php @@ -0,0 +1,20 @@ + $known every key that is registered + */ + public function __construct( + public readonly string $key, + public readonly array $known, + ) { + parent::__construct(\sprintf( + 'Unknown source "%s". Available: %s.', + $key, + [] === $known ? 'none' : implode(', ', $known) + )); + } +} diff --git a/src/Import/Exception/UpsertFailedException.php b/src/Import/Exception/UpsertFailedException.php new file mode 100644 index 0000000..1e18c24 --- /dev/null +++ b/src/Import/Exception/UpsertFailedException.php @@ -0,0 +1,19 @@ +getMessage(), previous: $previous); + } +} diff --git a/src/Import/ImportResult.php b/src/Import/ImportResult.php new file mode 100644 index 0000000..f891f65 --- /dev/null +++ b/src/Import/ImportResult.php @@ -0,0 +1,23 @@ + $sources + */ + private function importer( + iterable $sources, + ?MockHttpClient $client = null, + string $contextUrls = self::CONTEXT_URLS, + ): DataSourceImporter { + return new DataSourceImporter( + $sources, + new NgsiLdBroker($client ?? new MockHttpClient(), self::BROKER_URL), + $contextUrls, + ); + } + + public function testItListsTheRegisteredSourceKeys(): void + { + $importer = $this->importer([FakeSource::withEntities('a-source'), FakeSource::withEntities('b-source')]); + + $this->assertSame(['a-source', 'b-source'], $importer->keys()); + } + + public function testItListsNothingWhenNoSourceIsRegistered(): void + { + $this->assertSame([], $this->importer([])->keys()); + } + + public function testItRejectsAnUnknownSourceAndNamesTheKnownOnes(): void + { + $importer = $this->importer([FakeSource::withEntities('some-source')]); + + $this->expectException(UnknownSourceException::class); + $this->expectExceptionMessage('Unknown source "nope". Available: some-source.'); + + $importer->payload('nope'); + } + + /** + * The important one: a source yielding nothing used to be reported as a + * completed import, which is indistinguishable from a working one. + */ + public function testItFailsWhenASourceProducesNothing(): void + { + $importer = $this->importer([new FakeSource('empty-source')]); + + $this->expectException(EmptySourceException::class); + $this->expectExceptionMessage('Source "empty-source" produced no entities.'); + + $importer->payload('empty-source'); + } + + /** + * The guard belongs to the conversion rather than to the dry run, so an + * import that intends to send is held to it too. + */ + public function testItSendsNothingWhenASourceProducesNothing(): void + { + $client = new MockHttpClient(); + + try { + $this->importer([new FakeSource('empty-source')], $client)->import('empty-source'); + $this->fail('An empty source was reported as a completed import.'); + } catch (EmptySourceException) { + $this->assertSame(0, $client->getRequestsCount(), 'An empty payload was sent to the broker.'); + } + } + + public function testEveryEntityCarriesTheConfiguredContexts(): void + { + $importer = $this->importer([FakeSource::withEntities('one-entity', 'urn:ngsi-ld:Example:1')]); + + $payload = $importer->payload('one-entity'); + + $this->assertSame( + ['https://example.com/domain.jsonld', 'https://example.com/core.jsonld'], + $payload[0]['@context'] + ); + } + + /** + * The contexts arrive as one comma-separated environment variable, so they + * are written by hand and carry whatever spacing that produces. + */ + public function testItIgnoresSpacingAndEmptyEntriesInTheConfiguredContexts(): void + { + $importer = $this->importer( + [FakeSource::withEntities('one-entity', 'urn:ngsi-ld:Example:1')], + contextUrls: ' https://example.com/domain.jsonld , ,', + ); + + $payload = $importer->payload('one-entity'); + + $this->assertSame(['https://example.com/domain.jsonld'], $payload[0]['@context']); + } + + public function testALimitCapsThePayload(): void + { + $importer = $this->importer([FakeSource::withEntities('many', 'urn:1', 'urn:2', 'urn:3', 'urn:4', 'urn:5')]); + + $this->assertCount(2, $importer->payload('many', 2)); + } + + /** + * A real source reads a whole feed, so a limit that converts everything + * and then trims would do all the work it was given to avoid. + */ + public function testALimitStopsPullingFromTheSource(): void + { + $source = FakeSource::withEntities('many', 'urn:1', 'urn:2', 'urn:3', 'urn:4', 'urn:5'); + + $this->importer([$source])->payload('many', 2); + + $this->assertSame(2, $source->produced()); + } + + /** + * --limit 0 is a mistyped option. Honouring it would produce an empty + * payload, which is the one outcome an import refuses to call a success. + */ + public function testALimitBelowOneStillImportsOneEntity(): void + { + $importer = $this->importer([FakeSource::withEntities('many', 'urn:1', 'urn:2')]); + + $this->assertCount(1, $importer->payload('many', 0)); + } + + public function testBuildingThePayloadSendsNothing(): void + { + $client = new MockHttpClient(); + + $this->importer([FakeSource::withEntities('one-entity', 'urn:1')], $client)->payload('one-entity'); + + $this->assertSame(0, $client->getRequestsCount()); + } + + public function testItUpsertsThePayloadAndReportsWhatTheBrokerDid(): void + { + $client = new MockHttpClient(new MockResponse('', ['http_code' => 204])); + + $result = $this->importer([FakeSource::withEntities('many', 'urn:1', 'urn:2')], $client)->import('many'); + + $this->assertSame(2, $result->count); + $this->assertSame(204, $result->status); + $this->assertSame(self::BROKER_URL, $result->brokerUrl); + $this->assertSame(1, $client->getRequestsCount(), 'The entities were not sent as one batch.'); + } + + public function testItReportsWhatTheBrokerSaidWhenTheUpsertIsRejected(): void + { + $client = new MockHttpClient(new MockResponse('{"title":"Bad Request"}', ['http_code' => 400])); + + try { + $this->importer([FakeSource::withEntities('one-entity', 'urn:1')], $client)->import('one-entity'); + $this->fail('A rejected upsert was reported as a completed import.'); + } catch (UpsertFailedException $exception) { + $this->assertStringContainsString('HTTP 400', $exception->getMessage()); + $this->assertStringContainsString('Bad Request', $exception->getMessage()); + $this->assertNotNull($exception->getPrevious(), 'The broker\'s own exception was discarded.'); + } + } + + /** + * A broker that cannot be reached fails in the HTTP client rather than in + * the broker's status check, and the two are the same thing to a caller. + */ + public function testItFailsTheSameWayWhenTheBrokerCannotBeReached(): void + { + $client = new MockHttpClient(static function (): never { + throw new TransportException('Connection refused'); + }); + + $this->expectException(UpsertFailedException::class); + $this->expectExceptionMessage('Connection refused'); + + $this->importer([FakeSource::withEntities('one-entity', 'urn:1')], $client)->import('one-entity'); + } +} diff --git a/tests/Source/FakeSource.php b/tests/Source/FakeSource.php new file mode 100644 index 0000000..052ab33 --- /dev/null +++ b/tests/Source/FakeSource.php @@ -0,0 +1,61 @@ + $entities + */ + public function __construct( + private readonly string $key, + private readonly array $entities = [], + ) { + } + + /** + * @param string ...$ids entity ids, one entity each + */ + public static function withEntities(string $key, string ...$ids): self + { + return new self($key, array_map( + static fn (string $id): NgsiEntity => new NgsiEntity($id, 'Example'), + array_values($ids) + )); + } + + public function key(): string + { + return $this->key; + } + + public function entities(): iterable + { + foreach ($this->entities as $entity) { + ++$this->produced; + + yield $entity; + } + } + + /** + * How many entities were actually pulled from this source. + */ + public function produced(): int + { + return $this->produced; + } +} From 07aec4f9ddb4db0c3752e9249059bda65857dae0 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 11:24:05 +0200 Subject: [PATCH 47/54] Reduced ImportCommand to console plumbing --- src/Command/ImportCommand.php | 97 ++++++++------------ src/Source/SourceInterface.php | 4 +- tests/Command/ImportCommandTest.php | 131 +++++++++++++--------------- 3 files changed, 97 insertions(+), 135 deletions(-) diff --git a/src/Command/ImportCommand.php b/src/Command/ImportCommand.php index b0c6a46..e1c4fa9 100644 --- a/src/Command/ImportCommand.php +++ b/src/Command/ImportCommand.php @@ -4,8 +4,10 @@ namespace App\Command; -use App\Broker\NgsiLdBroker; -use App\Source\SourceInterface; +use App\Import\DataSourceImporter; +use App\Import\Exception\EmptySourceException; +use App\Import\Exception\UnknownSourceException; +use App\Import\Exception\UpsertFailedException; use Symfony\Component\Console\Attribute\AsCommand; use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Input\InputArgument; @@ -13,9 +15,11 @@ use Symfony\Component\Console\Input\InputOption; use Symfony\Component\Console\Output\OutputInterface; use Symfony\Component\Console\Style\SymfonyStyle; -use Symfony\Component\DependencyInjection\Attribute\Autowire; -use Symfony\Component\DependencyInjection\Attribute\AutowireIterator; +/** + * The console front for DataSourceImporter: selects a source, and turns what + * the import raises into an exit code and something readable. + */ #[AsCommand( name: 'app:import', description: 'Convert a data source to NGSI-LD and upsert it into the context broker.', @@ -24,15 +28,8 @@ final class ImportCommand extends Command { private const JSON_FLAGS = \JSON_PRETTY_PRINT | \JSON_UNESCAPED_SLASHES | \JSON_UNESCAPED_UNICODE; - /** - * @param iterable $sources - */ public function __construct( - #[AutowireIterator('app.source')] - private readonly iterable $sources, - private readonly NgsiLdBroker $broker, - #[Autowire(env: 'ENTER_NGSI_CONTEXT_URLS')] - private readonly string $contextUrls, + private readonly DataSourceImporter $importer, ) { parent::__construct(); } @@ -49,12 +46,9 @@ protected function execute(InputInterface $input, OutputInterface $output): int { $io = new SymfonyStyle($input, $output); - $sources = []; - foreach ($this->sources as $source) { - $sources[$source->key()] = $source; - } + $keys = $this->importer->keys(); - if ([] === $sources) { + if ([] === $keys) { $io->error('No data sources are registered.'); $io->listing([ 'A source must implement App\Source\SourceInterface.', @@ -64,50 +58,46 @@ protected function execute(InputInterface $input, OutputInterface $output): int return Command::FAILURE; } - $key = $input->getArgument('source'); + $argument = $input->getArgument('source'); - if (null === $key) { + if (null === $argument) { // Options only make sense together with a source. Listing the // sources and exiting successfully would look like an import ran. if ($input->getOption('dry-run') || null !== $input->getOption('limit')) { $io->error(\sprintf( 'No source given. Available: %s.', - implode(', ', array_keys($sources)) + implode(', ', $keys) )); return Command::INVALID; } $io->section('Available sources'); - $io->listing(array_keys($sources)); + $io->listing($keys); return Command::SUCCESS; } - if (!isset($sources[$key])) { - $io->error(\sprintf('Unknown source "%s". Available: %s.', $key, implode(', ', array_keys($sources)))); + $key = (string) $argument; + $limit = null !== $input->getOption('limit') ? (int) $input->getOption('limit') : null; - return Command::INVALID; - } - - $limit = null !== $input->getOption('limit') ? max(1, (int) $input->getOption('limit')) : null; - $contexts = $this->contexts(); + try { + if ($input->getOption('dry-run')) { + $payload = $this->importer->payload($key, $limit); - $payload = []; - foreach ($sources[$key]->entities() as $entity) { - $payload[] = $entity->toArray($contexts); + $output->writeln(json_encode($payload, self::JSON_FLAGS)); + $io->note(\sprintf('Dry run: %d entities were not sent.', \count($payload))); - if (null !== $limit && \count($payload) >= $limit) { - break; + return Command::SUCCESS; } - } - // A source that yields nothing is almost always misconfigured rather - // than genuinely empty, and it fails silently by construction: a - // record skipped for a missing field looks exactly like a feed with no - // records. Fail loudly so it cannot be mistaken for a successful run. - if ([] === $payload) { - $io->error(\sprintf('Source "%s" produced no entities.', $key)); + $result = $this->importer->import($key, $limit); + } catch (UnknownSourceException $exception) { + $io->error($exception->getMessage()); + + return Command::INVALID; + } catch (EmptySourceException $exception) { + $io->error($exception->getMessage()); $io->text( 'The source ran to completion without raising an exception, so every record was ' .'discarded by the source\'s own guards rather than failing. Verbosity flags will ' @@ -120,18 +110,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int ]); return Command::FAILURE; - } - - if ($input->getOption('dry-run')) { - $output->writeln(json_encode($payload, self::JSON_FLAGS)); - $io->note(\sprintf('Dry run: %d entities were not sent.', \count($payload))); - - return Command::SUCCESS; - } - - try { - $status = $this->broker->upsert($payload); - } catch (\Throwable $exception) { + } catch (UpsertFailedException $exception) { $io->error($exception->getMessage()); return Command::FAILURE; @@ -139,19 +118,11 @@ protected function execute(InputInterface $input, OutputInterface $output): int $io->success(\sprintf( 'Upserted %d entities into %s (HTTP %d).', - \count($payload), - $this->broker->brokerUrl(), - $status + $result->count, + $result->brokerUrl, + $result->status )); return Command::SUCCESS; } - - /** - * @return list - */ - private function contexts(): array - { - return array_values(array_filter(array_map(trim(...), explode(',', $this->contextUrls)))); - } } diff --git a/src/Source/SourceInterface.php b/src/Source/SourceInterface.php index b372d28..eaea6a7 100644 --- a/src/Source/SourceInterface.php +++ b/src/Source/SourceInterface.php @@ -12,8 +12,8 @@ * * Implementations own everything specific to their feed: where it comes from, * its field names, its quirks, and which Smart Data Model it maps onto. The - * broker and the import command stay unaware of all of it, so adding the next - * ENTER data set means adding one class and nothing else. + * broker and the import stay unaware of all of it, so adding the next ENTER + * data set means adding one class and nothing else. */ #[AutoconfigureTag('app.source')] interface SourceInterface diff --git a/tests/Command/ImportCommandTest.php b/tests/Command/ImportCommandTest.php index a1d68ad..c337ed1 100644 --- a/tests/Command/ImportCommandTest.php +++ b/tests/Command/ImportCommandTest.php @@ -6,72 +6,42 @@ use App\Broker\NgsiLdBroker; use App\Command\ImportCommand; -use App\Ngsi\NgsiEntity; +use App\Import\DataSourceImporter; use App\Source\SourceInterface; +use App\Tests\Source\FakeSource; use PHPUnit\Framework\TestCase; use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Tester\CommandTester; use Symfony\Component\HttpClient\MockHttpClient; +use Symfony\Component\HttpClient\Response\MockResponse; +/** + * The console surface only: the affordances around an import, and which exit + * code each outcome maps to. What an import decides belongs to + * DataSourceImporter and is covered by DataSourceImporterTest. + */ class ImportCommandTest extends TestCase { /** * @param iterable $sources */ - private function tester(iterable $sources): CommandTester + private function tester(iterable $sources, ?MockHttpClient $client = null): CommandTester { - return new CommandTester(new ImportCommand( + return new CommandTester(new ImportCommand(new DataSourceImporter( $sources, - new NgsiLdBroker(new MockHttpClient(), 'http://broker.invalid'), + new NgsiLdBroker($client ?? new MockHttpClient(), 'http://broker.invalid'), 'https://example.com/context.jsonld', - )); + ))); } - private function source(string $key, NgsiEntity ...$entities): SourceInterface + public function testItListsTheSourcesWhenCalledBare(): void { - return new readonly class($key, $entities) implements SourceInterface { - /** @param list $entities */ - public function __construct( - private string $key, - private array $entities, - ) { - } - - public function key(): string - { - return $this->key; - } - - public function entities(): iterable - { - yield from $this->entities; - } - }; - } - - /** - * The important one: a source yielding nothing used to exit successfully - * with a warning, which is indistinguishable from a working import. - */ - public function testItFailsWhenASourceProducesNothing(): void - { - $tester = $this->tester([$this->source('empty-source')]); - - $status = $tester->execute(['source' => 'empty-source', '--dry-run' => true]); - - $this->assertSame(Command::FAILURE, $status); - $this->assertStringContainsString('produced no entities', $tester->getDisplay()); - } - - public function testItSuggestsCausesWhenASourceProducesNothing(): void - { - $tester = $this->tester([$this->source('empty-source')]); - $tester->execute(['source' => 'empty-source', '--dry-run' => true]); + $tester = $this->tester([FakeSource::withEntities('some-source')]); - $display = $tester->getDisplay(); + $status = $tester->execute([]); - $this->assertStringContainsString('path or URL', $display); - $this->assertStringContainsString('envelope, nesting, field names', $display); + $this->assertSame(Command::SUCCESS, $status); + $this->assertStringContainsString('some-source', $tester->getDisplay()); } public function testItFailsWhenNoSourcesAreRegistered(): void @@ -90,7 +60,7 @@ public function testItFailsWhenNoSourcesAreRegistered(): void */ public function testItRejectsOptionsWithoutASource(): void { - $tester = $this->tester([$this->source('some-source')]); + $tester = $this->tester([FakeSource::withEntities('some-source')]); $status = $tester->execute(['--dry-run' => true]); @@ -98,32 +68,37 @@ public function testItRejectsOptionsWithoutASource(): void $this->assertStringContainsString('No source given', $tester->getDisplay()); } - public function testItStillListsSourcesWhenCalledBare(): void + public function testAnUnknownSourceIsTheCallersMistake(): void { - $tester = $this->tester([$this->source('some-source')]); + $tester = $this->tester([FakeSource::withEntities('some-source')]); - $status = $tester->execute([]); + $status = $tester->execute(['source' => 'nope']); - $this->assertSame(Command::SUCCESS, $status); - $this->assertStringContainsString('some-source', $tester->getDisplay()); + $this->assertSame(Command::INVALID, $status); + $this->assertStringContainsString('Unknown source "nope"', $tester->getDisplay()); } - public function testItRejectsAnUnknownSource(): void + /** + * The failure carries no exception to show, so the command has to supply + * the places worth looking itself. + */ + public function testAnEmptySourceFailsAndSuggestsCauses(): void { - $tester = $this->tester([$this->source('some-source')]); + $tester = $this->tester([new FakeSource('empty-source')]); - $status = $tester->execute(['source' => 'nope']); + $status = $tester->execute(['source' => 'empty-source']); + $display = $tester->getDisplay(); - $this->assertSame(Command::INVALID, $status); - $this->assertStringContainsString('Unknown source "nope"', $tester->getDisplay()); + $this->assertSame(Command::FAILURE, $status); + $this->assertStringContainsString('produced no entities', $display); + $this->assertStringContainsString('path or URL', $display); + $this->assertStringContainsString('envelope, nesting, field names', $display); } public function testDryRunPrintsThePayloadAndSendsNothing(): void { - $entity = new NgsiEntity('urn:ngsi-ld:Example:1', 'Example') - ->setProperty('name', 'Example'); - - $tester = $this->tester([$this->source('one-entity', $entity)]); + $client = new MockHttpClient(); + $tester = $this->tester([FakeSource::withEntities('one-entity', 'urn:ngsi-ld:Example:1')], $client); $status = $tester->execute(['source' => 'one-entity', '--dry-run' => true]); $display = $tester->getDisplay(); @@ -131,18 +106,34 @@ public function testDryRunPrintsThePayloadAndSendsNothing(): void $this->assertSame(Command::SUCCESS, $status); $this->assertStringContainsString('"urn:ngsi-ld:Example:1"', $display); $this->assertStringContainsString('1 entities were not sent', $display); + $this->assertSame(0, $client->getRequestsCount()); } - public function testLimitCapsThePayload(): void + public function testItReportsWhatWasUpserted(): void { - $entities = []; - foreach (range(1, 5) as $i) { - $entities[] = new NgsiEntity(\sprintf('urn:ngsi-ld:Example:%d', $i), 'Example'); - } + $client = new MockHttpClient(new MockResponse('', ['http_code' => 204])); + $tester = $this->tester([FakeSource::withEntities('one-entity', 'urn:ngsi-ld:Example:1')], $client); - $tester = $this->tester([$this->source('many', ...$entities)]); - $tester->execute(['source' => 'many', '--dry-run' => true, '--limit' => 2]); + $status = $tester->execute(['source' => 'one-entity']); + $display = $tester->getDisplay(); + + $this->assertSame(Command::SUCCESS, $status); + $this->assertStringContainsString('Upserted 1 entities', $display); + $this->assertStringContainsString('HTTP 204', $display); + } + + /** + * The broker being down is an operational condition rather than a bug, so + * it is reported as a message instead of an uncaught exception. + */ + public function testABrokerFailureIsReportedAsAnError(): void + { + $client = new MockHttpClient(new MockResponse('', ['http_code' => 500])); + $tester = $this->tester([FakeSource::withEntities('one-entity', 'urn:ngsi-ld:Example:1')], $client); - $this->assertStringContainsString('2 entities were not sent', $tester->getDisplay()); + $status = $tester->execute(['source' => 'one-entity']); + + $this->assertSame(Command::FAILURE, $status); + $this->assertStringContainsString('HTTP 500', $tester->getDisplay()); } } From 64aa10f06153385d43d39f205c3b7bbb811616c8 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 12:57:54 +0200 Subject: [PATCH 48/54] Move context from comma seperated list in .env to each dataset via config/sources.yaml --- .env | 2 +- config/sources.yaml | 1 + src/Import/DataSourceImporter.php | 47 +++++++-------- src/Source/Manifest/Catalog.php | 5 +- src/Source/Manifest/Descriptor.php | 1 + src/Source/Manifest/Schema.php | 5 ++ tests/Command/ImportCommandTest.php | 12 +++- tests/Import/DataSourceImporterTest.php | 72 ++++++++++++++++++++--- tests/Source/Manifest/CatalogTest.php | 7 +++ tests/Source/Manifest/ValidatorTest.php | 2 + tests/Source/Manifest/WritesManifests.php | 61 +++++++++++++++++++ 11 files changed, 181 insertions(+), 34 deletions(-) create mode 100644 tests/Source/Manifest/WritesManifests.php diff --git a/.env b/.env index a9d7b27..b68e4c9 100644 --- a/.env +++ b/.env @@ -36,7 +36,7 @@ APP_BROKER_BASE_URI=http://scorpio.local:9090/ # JSON-LD contexts attached to every entity, outermost last so the ETSI core # context resolves the NGSI-LD terms and the domain context resolves the # Smart Data Models ones. -ENTER_NGSI_CONTEXT_URLS='https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld,https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld' +ENTER_NGSI_CONTEXT_URLS='https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld' # A single domain context, for the Link header that read requests need. Only # used by `task broker:entities`, and it must be the context defining the type diff --git a/config/sources.yaml b/config/sources.yaml index b011401..bebb760 100644 --- a/config/sources.yaml +++ b/config/sources.yaml @@ -17,6 +17,7 @@ sources: media_type: application/geo+json crs: 'EPSG:25832' model: OnStreetParking + context_url: 'https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld' update_frequency: continuous # The portal states no licence for this data set. DCAT-AP requires diff --git a/src/Import/DataSourceImporter.php b/src/Import/DataSourceImporter.php index 6ae8f8c..3030870 100644 --- a/src/Import/DataSourceImporter.php +++ b/src/Import/DataSourceImporter.php @@ -8,17 +8,13 @@ use App\Import\Exception\EmptySourceException; use App\Import\Exception\UnknownSourceException; use App\Import\Exception\UpsertFailedException; +use App\Source\Manifest\Catalog; use App\Source\SourceInterface; use Symfony\Component\DependencyInjection\Attribute\Autowire; use Symfony\Component\DependencyInjection\Attribute\AutowireIterator; /** * Converts a registered source to NGSI-LD and upserts it into the broker. - * - * Owns what an import decides: which sources exist, which contexts their - * entities carry, how many to take, and what counts as a failed run. That - * leaves the command around it with argument parsing and exit codes, and lets - * the decisions be exercised without a console. */ final readonly class DataSourceImporter { @@ -28,6 +24,7 @@ public function __construct( #[AutowireIterator('app.source')] private iterable $sources, + private Catalog $catalog, private NgsiLdBroker $broker, #[Autowire(env: 'ENTER_NGSI_CONTEXT_URLS')] private string $contextUrls, @@ -43,7 +40,7 @@ public function keys(): array } /** - * Converts a source to NGSI-LD without sending anything. + * Converts a source to NGSI-LD. * * @return non-empty-list> * @@ -52,33 +49,30 @@ public function keys(): array */ public function payload(string $key, ?int $limit = null): array { + // Collect all dataset keys. $registry = $this->registry(); + // Check if requested dataset key exists. if (!isset($registry[$key])) { throw new UnknownSourceException($key, array_keys($registry)); } - // A limit below 1 is a mistyped option rather than a request to import - // nothing, and importing nothing is the one outcome this class refuses - // to report as a success. + // Define minimum limit, in case of limit defined as less than 1. $limit = null === $limit ? null : max(1, $limit); - $contexts = $this->contexts(); + + // Load context for given dataset. + $contexts = $this->contexts($key); $payload = []; foreach ($registry[$key]->entities() as $entity) { $payload[] = $entity->toArray($contexts); - // A source reads a whole feed lazily, so the limit stops the - // conversion instead of trimming its result. + // Break upon limit. if (null !== $limit && \count($payload) >= $limit) { break; } } - // A source that yields nothing is almost always misconfigured rather - // than genuinely empty, and it fails silently by construction: a - // record skipped for a missing field looks exactly like a feed with no - // records. Fail loudly so it cannot be mistaken for a successful run. if ([] === $payload) { throw new EmptySourceException($key); } @@ -93,22 +87,24 @@ public function payload(string $key, ?int $limit = null): array */ public function import(string $key, ?int $limit = null): ImportResult { + // Get payload from dataset. $payload = $this->payload($key, $limit); + // Try to upsert broker with payload. try { $status = $this->broker->upsert($payload); } catch (\Throwable $exception) { - // A broker that is down or rejects the batch is an operational - // condition, not a bug in the conversion. Giving it a type of its - // own lets a caller report it plainly while the exceptions a - // broken feed raises — the ones worth a stack trace — pass through. throw new UpsertFailedException($exception); } + // Return result. return new ImportResult(\count($payload), $status, $this->broker->brokerUrl()); } /** + * Get list of registered datasets. + * + * @see config/sources.yaml * @return array keyed by source key */ private function registry(): array @@ -123,10 +119,15 @@ private function registry(): array } /** - * @return list + * Return an array of contexts. Each dataset holds its own context + * @see config/sources.yaml + * @return array */ - private function contexts(): array + private function contexts(string $key): array { - return array_values(array_filter(array_map(trim(...), explode(',', $this->contextUrls)))); + return [ + $this->catalog->get($key)->contextUrl, + ...array_values(array_filter(array_map(trim(...), explode(',', $this->contextUrls)))), + ]; } } diff --git a/src/Source/Manifest/Catalog.php b/src/Source/Manifest/Catalog.php index 722e589..71ad47f 100644 --- a/src/Source/Manifest/Catalog.php +++ b/src/Source/Manifest/Catalog.php @@ -67,6 +67,7 @@ private function load(): array accessUrl: $entry['access_url'], crs: $entry['crs'], model: $entry['model'], + contextUrl: $entry['context_url'], description: $entry['description'], publisher: $entry['publisher'], contact: $entry['contact'], @@ -82,7 +83,7 @@ private function load(): array } /** - * @return array}> + * @return array}> */ private function validated(): array { @@ -102,7 +103,7 @@ private function validated(): array } try { - /** @var array}> $processed */ + /** @var array}> $processed */ $processed = new Processor()->process(Schema::tree(), [$sources]); } catch (InvalidConfigurationException $exception) { throw new \RuntimeException(\sprintf('Source manifest "%s" is invalid: %s', $this->manifest, $exception->getMessage()), previous: $exception); diff --git a/src/Source/Manifest/Descriptor.php b/src/Source/Manifest/Descriptor.php index 9f0e36b..46d0631 100644 --- a/src/Source/Manifest/Descriptor.php +++ b/src/Source/Manifest/Descriptor.php @@ -26,6 +26,7 @@ public function __construct( public string $accessUrl, public string $crs, public string $model, + public string $contextUrl, public ?string $description = null, public ?string $publisher = null, public ?string $contact = null, diff --git a/src/Source/Manifest/Schema.php b/src/Source/Manifest/Schema.php index c4a977e..bbaf41c 100644 --- a/src/Source/Manifest/Schema.php +++ b/src/Source/Manifest/Schema.php @@ -50,6 +50,11 @@ public static function tree(): NodeInterface ->cannotBeEmpty() ->info('Smart Data Model the data set is published as.') ->end() + ->scalarNode('context_url') + ->isRequired() + ->cannotBeEmpty() + ->info('JSON-LD context defining the model\'s terms. Without the right one a consumer reads the entity\'s attributes as undefined strings.') + ->end() ->scalarNode('description')->defaultNull()->end() ->scalarNode('publisher')->defaultNull()->end() ->scalarNode('contact')->defaultNull()->end() diff --git a/tests/Command/ImportCommandTest.php b/tests/Command/ImportCommandTest.php index c337ed1..6d4c7ca 100644 --- a/tests/Command/ImportCommandTest.php +++ b/tests/Command/ImportCommandTest.php @@ -7,8 +7,10 @@ use App\Broker\NgsiLdBroker; use App\Command\ImportCommand; use App\Import\DataSourceImporter; +use App\Source\Manifest\Catalog; use App\Source\SourceInterface; use App\Tests\Source\FakeSource; +use App\Tests\Source\Manifest\WritesManifests; use PHPUnit\Framework\TestCase; use Symfony\Component\Console\Command\Command; use Symfony\Component\Console\Tester\CommandTester; @@ -22,15 +24,23 @@ */ class ImportCommandTest extends TestCase { + use WritesManifests; + /** * @param iterable $sources */ private function tester(iterable $sources, ?MockHttpClient $client = null): CommandTester { + $keys = []; + foreach ($sources as $source) { + $keys[] = $source->key(); + } + return new CommandTester(new ImportCommand(new DataSourceImporter( $sources, + new Catalog($this->manifestFor($keys)), new NgsiLdBroker($client ?? new MockHttpClient(), 'http://broker.invalid'), - 'https://example.com/context.jsonld', + 'https://example.com/core.jsonld', ))); } diff --git a/tests/Import/DataSourceImporterTest.php b/tests/Import/DataSourceImporterTest.php index a1c666a..3e838a7 100644 --- a/tests/Import/DataSourceImporterTest.php +++ b/tests/Import/DataSourceImporterTest.php @@ -9,8 +9,10 @@ use App\Import\Exception\EmptySourceException; use App\Import\Exception\UnknownSourceException; use App\Import\Exception\UpsertFailedException; +use App\Source\Manifest\Catalog; use App\Source\SourceInterface; use App\Tests\Source\FakeSource; +use App\Tests\Source\Manifest\WritesManifests; use PHPUnit\Framework\TestCase; use Symfony\Component\HttpClient\Exception\TransportException; use Symfony\Component\HttpClient\MockHttpClient; @@ -18,20 +20,27 @@ class DataSourceImporterTest extends TestCase { + use WritesManifests; private const string BROKER_URL = 'http://broker.invalid'; - private const string CONTEXT_URLS = 'https://example.com/domain.jsonld,https://example.com/core.jsonld'; + private const string CORE_CONTEXT = 'https://example.com/core.jsonld'; /** - * @param iterable $sources + * @param iterable $sources each is registered in the manifest under its own key */ private function importer( iterable $sources, ?MockHttpClient $client = null, - string $contextUrls = self::CONTEXT_URLS, + string $contextUrls = self::CORE_CONTEXT, ): DataSourceImporter { + $keys = []; + foreach ($sources as $source) { + $keys[] = $source->key(); + } + return new DataSourceImporter( $sources, + new Catalog($this->manifestFor($keys)), new NgsiLdBroker($client ?? new MockHttpClient(), self::BROKER_URL), $contextUrls, ); @@ -89,18 +98,64 @@ public function testItSendsNothingWhenASourceProducesNothing(): void } } - public function testEveryEntityCarriesTheConfiguredContexts(): void + /** + * Later entries win term conflicts, so the data set's own context comes + * first and the core context last, where it stays authoritative over the + * NGSI-LD terms a domain context may also define. + */ + public function testEveryEntityCarriesItsDataSetContextThenTheCoreContext(): void { $importer = $this->importer([FakeSource::withEntities('one-entity', 'urn:ngsi-ld:Example:1')]); $payload = $importer->payload('one-entity'); $this->assertSame( - ['https://example.com/domain.jsonld', 'https://example.com/core.jsonld'], + [self::dataSetContext('one-entity'), self::CORE_CONTEXT], $payload[0]['@context'] ); } + /** + * Two data sets published under different models must not advertise each + * other's vocabulary, which a single app-wide list cannot avoid. + */ + public function testEachDataSetCarriesOnlyItsOwnContext(): void + { + $importer = $this->importer([ + FakeSource::withEntities('first-source', 'urn:ngsi-ld:Example:1'), + FakeSource::withEntities('second-source', 'urn:ngsi-ld:Example:2'), + ]); + + $this->assertSame( + [self::dataSetContext('first-source'), self::CORE_CONTEXT], + $importer->payload('first-source')[0]['@context'] + ); + $this->assertSame( + [self::dataSetContext('second-source'), self::CORE_CONTEXT], + $importer->payload('second-source')[0]['@context'] + ); + } + + /** + * A source registered in the container but not in the manifest has no + * context to publish under, so it fails rather than emitting entities a + * consumer cannot resolve. + */ + public function testItFailsWhenASourceHasNoManifestEntry(): void + { + $importer = new DataSourceImporter( + [FakeSource::withEntities('unregistered', 'urn:ngsi-ld:Example:1')], + new Catalog($this->manifestFor(['other-source'])), + new NgsiLdBroker(new MockHttpClient(), self::BROKER_URL), + self::CORE_CONTEXT, + ); + + $this->expectException(\RuntimeException::class); + $this->expectExceptionMessage('No entry for source "unregistered"'); + + $importer->payload('unregistered'); + } + /** * The contexts arrive as one comma-separated environment variable, so they * are written by hand and carry whatever spacing that produces. @@ -109,12 +164,15 @@ public function testItIgnoresSpacingAndEmptyEntriesInTheConfiguredContexts(): vo { $importer = $this->importer( [FakeSource::withEntities('one-entity', 'urn:ngsi-ld:Example:1')], - contextUrls: ' https://example.com/domain.jsonld , ,', + contextUrls: ' '.self::CORE_CONTEXT.' , ,', ); $payload = $importer->payload('one-entity'); - $this->assertSame(['https://example.com/domain.jsonld'], $payload[0]['@context']); + $this->assertSame( + [self::dataSetContext('one-entity'), self::CORE_CONTEXT], + $payload[0]['@context'] + ); } public function testALimitCapsThePayload(): void diff --git a/tests/Source/Manifest/CatalogTest.php b/tests/Source/Manifest/CatalogTest.php index bfa83b5..2707d49 100644 --- a/tests/Source/Manifest/CatalogTest.php +++ b/tests/Source/Manifest/CatalogTest.php @@ -43,6 +43,7 @@ public function testEveryShippedEntryCanBeImportedFrom(): void $this->assertMatchesRegularExpression('#^https?://#', $descriptor->accessUrl, $key); $this->assertMatchesRegularExpression('/^EPSG:\d+$/', $descriptor->crs, $key); $this->assertNotSame('', $descriptor->model, $key); + $this->assertMatchesRegularExpression('#^https?://#', $descriptor->contextUrl, $key); } } @@ -55,6 +56,7 @@ public function testItNamesTheKnownEntriesWhenAskedForAnUnknownOne(): void access_url: https://example.com/feed.json crs: EPSG:25832 model: Example + context_url: https://example.com/context.jsonld YAML)); $this->expectException(\RuntimeException::class); @@ -81,6 +83,7 @@ public function testItRejectsAnEntryMissingAFieldTheImportNeeds(): void title: A source access_url: https://example.com/feed.json model: Example + context_url: https://example.com/context.jsonld YAML)); $this->expectException(\RuntimeException::class); @@ -98,6 +101,7 @@ public function testItRejectsAnOmittedFieldWithoutAReason(): void access_url: https://example.com/feed.json crs: EPSG:25832 model: Example + context_url: https://example.com/context.jsonld omitted_fields: some_field: ~ YAML)); @@ -122,6 +126,7 @@ public function testItKeepsADashedSourceKeyIntact(): void access_url: https://example.com/feed.json crs: EPSG:25832 model: Example + context_url: https://example.com/context.jsonld YAML)); $this->assertSame(['handicap-parking'], array_keys($catalog->all())); @@ -141,6 +146,7 @@ public function testItRejectsAFieldTheManifestDoesNotDefine(): void access_url: https://example.com/feed.json crs: EPSG:25832 model: Example + context_url: https://example.com/context.jsonld license: CC-BY-4.0 YAML)); @@ -183,6 +189,7 @@ public function testItReadsABlankOptionalFieldAsUnknown(): void access_url: https://example.com/feed.json crs: EPSG:25832 model: Example + context_url: https://example.com/context.jsonld publisher: ' Aarhus Kommune ' contact: '' licence: ~ diff --git a/tests/Source/Manifest/ValidatorTest.php b/tests/Source/Manifest/ValidatorTest.php index ada913d..53230bb 100644 --- a/tests/Source/Manifest/ValidatorTest.php +++ b/tests/Source/Manifest/ValidatorTest.php @@ -55,10 +55,12 @@ public function testItFailsOnAnEntryNoImportWouldReach(): void access_url: https://example.com/feed.json crs: EPSG:25832 model: Example + context_url: https://example.com/context.jsonld unreached-source: title: Another source access_url: https://example.com/other.json model: Example + context_url: https://example.com/context.jsonld YAML)); $this->expectException(\RuntimeException::class); diff --git a/tests/Source/Manifest/WritesManifests.php b/tests/Source/Manifest/WritesManifests.php new file mode 100644 index 0000000..121509f --- /dev/null +++ b/tests/Source/Manifest/WritesManifests.php @@ -0,0 +1,61 @@ + */ + private array $manifests = []; + + protected function tearDown(): void + { + foreach ($this->manifests as $path) { + if (is_file($path)) { + unlink($path); + } + } + + $this->manifests = []; + } + + protected static function dataSetContext(string $key): string + { + return \sprintf('https://example.com/%s.jsonld', $key); + } + + /** + * @param list $keys + * + * @return string path to the manifest + */ + protected function manifestFor(array $keys): string + { + $entries = array_map(static fn (string $key): string => \sprintf( + " %s:\n title: %s\n access_url: https://example.com/%s.json\n crs: EPSG:25832\n model: Example\n context_url: %s", + $key, + $key, + $key, + self::dataSetContext($key), + ), $keys); + + $path = tempnam(sys_get_temp_dir(), 'sources-'); + + if (false === $path) { + $this->fail('Could not create a temporary manifest.'); + } + + file_put_contents($path, "sources:\n".implode("\n", $entries)."\n"); + $this->manifests[] = $path; + + return $path; + } +} From 87e540f1914158fbf9913a7005c9b6503827480a Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 13:06:53 +0200 Subject: [PATCH 49/54] Minor corrections --- src/Geo/Wgs84Transformer.php | 13 ++++--------- src/Import/Exception/EmptySourceException.php | 4 ---- src/Import/Exception/UnknownSourceException.php | 3 --- src/Import/Exception/UpsertFailedException.php | 3 --- 4 files changed, 4 insertions(+), 19 deletions(-) diff --git a/src/Geo/Wgs84Transformer.php b/src/Geo/Wgs84Transformer.php index 9f57abe..19489ba 100644 --- a/src/Geo/Wgs84Transformer.php +++ b/src/Geo/Wgs84Transformer.php @@ -20,10 +20,6 @@ final class Wgs84Transformer /** * PROJ definitions for coordinate systems this application reads. * - * The `towgs84=0,0,0,0,0,0,0` term treats ETRS89 as equivalent to WGS84. - * The two datums have diverged by roughly a metre since 1989; see ADR 004 - * for why that is accepted here. - * * @var array */ private const array DEFINITIONS = [ @@ -58,10 +54,9 @@ public function __construct(array $definitions = []) } /** - * Coordinates are not rounded. Downstream use is unknown and may include - * planning work, so the transformed value is published as computed. + * Coordinates are not rounded. * - * @param string $srid source CRS, e.g. "EPSG:25832" + * @param string $srid source CRS * * @return array{float, float} GeoJSON coordinate order: [longitude, latitude] */ @@ -80,7 +75,7 @@ public function toWgs84(string $srid, float $x, float $y): array } /** - * @param string $srid source CRS, e.g. "EPSG:25832" + * @param string $srid source CRS * * @return array{type: string, coordinates: array{float, float}} GeoJSON Point */ @@ -102,7 +97,7 @@ public function point(string $srid, float $x, float $y): array * A third ordinate (elevation) is dropped: the inputs are two-dimensional, * and vertical datums are a separate concern this class does not model. * - * @param string $srid source CRS, e.g. "EPSG:25832" + * @param string $srid source CRS * @param array $geometry GeoJSON geometry object * * @return array{type: string, coordinates: mixed} diff --git a/src/Import/Exception/EmptySourceException.php b/src/Import/Exception/EmptySourceException.php index 5a87617..d6bcc3f 100644 --- a/src/Import/Exception/EmptySourceException.php +++ b/src/Import/Exception/EmptySourceException.php @@ -6,10 +6,6 @@ /** * A source ran to completion and yielded no entities. - * - * Carries no diagnosis: the source discarded every record through its own - * guards without raising anything, so there is nothing here to report beyond - * which source it was. */ final class EmptySourceException extends \RuntimeException { diff --git a/src/Import/Exception/UnknownSourceException.php b/src/Import/Exception/UnknownSourceException.php index b607c65..c799ba3 100644 --- a/src/Import/Exception/UnknownSourceException.php +++ b/src/Import/Exception/UnknownSourceException.php @@ -6,9 +6,6 @@ /** * No source is registered under the requested key. - * - * Separate from EmptySourceException so a caller can tell a key it can fix - * from a source that ran and had nothing to show. */ final class UnknownSourceException extends \InvalidArgumentException { diff --git a/src/Import/Exception/UpsertFailedException.php b/src/Import/Exception/UpsertFailedException.php index 1e18c24..de80c2a 100644 --- a/src/Import/Exception/UpsertFailedException.php +++ b/src/Import/Exception/UpsertFailedException.php @@ -6,9 +6,6 @@ /** * The payload was built, but the broker could not be written to. - * - * Reusing the underlying message keeps whatever the broker said, and the - * original exception stays available as the cause. */ final class UpsertFailedException extends \RuntimeException { From c05adfedb323e8ee32468cf3e7c37c2ab39106c9 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 14:41:32 +0200 Subject: [PATCH 50/54] Removed task broker:entities for now --- .env | 5 ----- README.md | 1 - Taskfile.yml | 8 -------- task/scripts/broker-entities | 34 ---------------------------------- 4 files changed, 48 deletions(-) delete mode 100755 task/scripts/broker-entities diff --git a/.env b/.env index b68e4c9..ee8aa28 100644 --- a/.env +++ b/.env @@ -38,11 +38,6 @@ APP_BROKER_BASE_URI=http://scorpio.local:9090/ # Smart Data Models ones. ENTER_NGSI_CONTEXT_URLS='https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld' -# A single domain context, for the Link header that read requests need. Only -# used by `task broker:entities`, and it must be the context defining the type -# being read — Parking while that is the only model published. -ENTER_NGSI_DOMAIN_CONTEXT=https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld - # Where each feed is read from is not configured here. It belongs to the data # set rather than to the environment, and lives in config/sources.yaml — see # docs/adr/007-source-manifest.md. diff --git a/README.md b/README.md index 3957294..c80ab34 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,6 @@ source feed (JSON) task import # list the available sources task import -- mtm_spatialmaps-handicap-parking # import one task import -- mtm_spatialmaps-handicap-parking --dry-run --limit 5 # print the payload instead -task broker:entities -- OnStreetParking 10 # read back what landed ``` ### Source manifest diff --git a/Taskfile.yml b/Taskfile.yml index b5cfede..84fbd88 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -40,14 +40,6 @@ tasks: desc: 'Import a source into the broker, e.g. task import -- mtm_spatialmaps-handicap-parking' cmd: ddev console app:import {{.CLI_ARGS}} - broker:entities: - desc: 'List broker entities of a type, e.g. task broker:entities -- OnStreetParking 10' - cmd: >- - ddev exec sh -c - "APP_BROKER_BASE_URI='$APP_BROKER_BASE_URI' - ENTER_NGSI_DOMAIN_CONTEXT='$ENTER_NGSI_DOMAIN_CONTEXT' - sh task/scripts/broker-entities {{.CLI_ARGS}}" - coding-standards:apply: desc: 'Apply coding standards' cmds: diff --git a/task/scripts/broker-entities b/task/scripts/broker-entities deleted file mode 100755 index 280bb6c..0000000 --- a/task/scripts/broker-entities +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/sh -# List entities of a given type from the NGSI-LD broker. -# -# Read requests need a domain context supplied in a Link header; without it the -# broker cannot expand a short type name into the full term the entity was -# written under. The context comes from ENTER_NGSI_DOMAIN_CONTEXT, which must -# be the one defining the type being asked for. -# -# Usage: broker-entities [limit] - -set -eu - -if [ $# -lt 1 ]; then - echo "Usage: broker-entities [limit]" >&2 - echo " NGSI-LD entity type, e.g. OnStreetParking" >&2 - echo " [limit] maximum entities to return (default 100)" >&2 - exit 64 -fi - -TYPE="$1" -LIMIT="${2:-100}" - -RESPONSE=$(curl -sS \ - -H "Link: <${ENTER_NGSI_DOMAIN_CONTEXT}>; rel=\"http://www.w3.org/ns/json-ld#context\"; type=\"application/ld+json\"" \ - "${APP_BROKER_BASE_URI%/}/ngsi-ld/v1/entities?type=${TYPE}&limit=${LIMIT}") - -printf '%s\n' "$RESPONSE" - -# A misspelled type, or one whose context is not the one configured, returns an -# empty list with HTTP 200 — indistinguishable from an empty broker or a failed -# import. Say what was asked for so the difference is visible. -if [ "$(printf '%s' "$RESPONSE" | tr -d '[:space:]')" = "[]" ]; then - printf '\nNo entities of type "%s". A type that does not match, or one defined in a context other than the configured ENTER_NGSI_DOMAIN_CONTEXT, returns an empty list rather than an error — check both before concluding the import failed.\n' "$TYPE" >&2 -fi From a97f6020b72a719e0c6fc559a9f4a6768e390f76 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 14:42:35 +0200 Subject: [PATCH 51/54] Removed comment from .env --- .env | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.env b/.env index ee8aa28..819e914 100644 --- a/.env +++ b/.env @@ -37,8 +37,4 @@ APP_BROKER_BASE_URI=http://scorpio.local:9090/ # context resolves the NGSI-LD terms and the domain context resolves the # Smart Data Models ones. ENTER_NGSI_CONTEXT_URLS='https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld' - -# Where each feed is read from is not configured here. It belongs to the data -# set rather than to the environment, and lives in config/sources.yaml — see -# docs/adr/007-source-manifest.md. ###< app ### From 346cbc6fec91d661478c46de6fd748ccf6a62903 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 14:43:01 +0200 Subject: [PATCH 52/54] Cut some comments --- src/Source/MtmSpatialMaps/HandicapParking.php | 8 ++------ src/Source/SourceInterface.php | 9 ++++----- 2 files changed, 6 insertions(+), 11 deletions(-) diff --git a/src/Source/MtmSpatialMaps/HandicapParking.php b/src/Source/MtmSpatialMaps/HandicapParking.php index b170ff0..1612cf0 100644 --- a/src/Source/MtmSpatialMaps/HandicapParking.php +++ b/src/Source/MtmSpatialMaps/HandicapParking.php @@ -15,7 +15,6 @@ * Disabled parking bays in Aarhus Municipality. * * @see config/sources.yaml - * @see https://github.com/smart-data-models/dataModel.Parking/tree/master/OnStreetParking */ final readonly class HandicapParking implements SourceInterface { @@ -48,12 +47,12 @@ public function entities(): iterable } /** + * Maps one feed record onto an NgsiEntity. + * * @param array $feature GeoJSON Feature */ private function toEntity(array $feature, Descriptor $source): ?NgsiEntity { - // A Feature keeps its attributes under `properties` and its geometry - // beside them, so neither is at the feature's top level. $row = $feature['properties'] ?? null; $geometry = $feature['geometry'] ?? null; @@ -74,9 +73,6 @@ private function toEntity(array $feature, Descriptor $source): ?NgsiEntity $source->model ); - // `forDisabled` rather than ParkingGroup's `onlyDisabled`: the two - // models have separate category enums, so values are not interchangeable. - // `onStreet` is dropped because the entity type already states it. return $entity ->setProperty('name', $this->address($row)) ->setProperty('description', trim((string) ($row['bemrk'] ?? ''))) diff --git a/src/Source/SourceInterface.php b/src/Source/SourceInterface.php index eaea6a7..f985048 100644 --- a/src/Source/SourceInterface.php +++ b/src/Source/SourceInterface.php @@ -8,12 +8,11 @@ use Symfony\Component\DependencyInjection\Attribute\AutoconfigureTag; /** - * One input data set, converted to NGSI-LD entities. + * Converts one input data set into NGSI-LD entities. * - * Implementations own everything specific to their feed: where it comes from, - * its field names, its quirks, and which Smart Data Model it maps onto. The - * broker and the import stay unaware of all of it, so adding the next ENTER - * data set means adding one class and nothing else. + * An implementation owns its feed's origin, field names, quirks and target + * Smart Data Model. The broker and the import know none of that, so a new + * ENTER data set costs exactly one class. */ #[AutoconfigureTag('app.source')] interface SourceInterface From 3353d0869c965837e8126ab4421e415688a2dea0 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 14:43:28 +0200 Subject: [PATCH 53/54] Coding standards --- src/Import/DataSourceImporter.php | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/Import/DataSourceImporter.php b/src/Import/DataSourceImporter.php index 3030870..f2a7fa7 100644 --- a/src/Import/DataSourceImporter.php +++ b/src/Import/DataSourceImporter.php @@ -105,6 +105,7 @@ public function import(string $key, ?int $limit = null): ImportResult * Get list of registered datasets. * * @see config/sources.yaml + * * @return array keyed by source key */ private function registry(): array @@ -119,8 +120,10 @@ private function registry(): array } /** - * Return an array of contexts. Each dataset holds its own context + * Return an array of contexts. Each dataset holds its own context. + * * @see config/sources.yaml + * * @return array */ private function contexts(string $key): array From 388d88a463c5f3125fcb44cc77f5d2d1345f7553 Mon Sep 17 00:00:00 2001 From: Jeppe Krogh Date: Tue, 8 Sep 2026 14:57:57 +0200 Subject: [PATCH 54/54] ADR shortening --- docs/adr/001-architecture-symfony-docker.md | 75 ++++------ docs/adr/002-publish-to-a-context-broker.md | 120 +++++++-------- docs/adr/003-ngsi-ld-representation.md | 47 +++--- docs/adr/004-coordinate-reference-system.md | 139 ++++++++---------- .../005-smart-data-models-as-vocabulary.md | 134 +++++++---------- .../006-onstreetparking-over-parkinggroup.md | 87 +++++------ 6 files changed, 249 insertions(+), 353 deletions(-) diff --git a/docs/adr/001-architecture-symfony-docker.md b/docs/adr/001-architecture-symfony-docker.md index c0f8708..1431f7b 100644 --- a/docs/adr/001-architecture-symfony-docker.md +++ b/docs/adr/001-architecture-symfony-docker.md @@ -10,61 +10,49 @@ ## Context -The adapter reads open data sets, converts them to a standard smart-city -representation, and publishes them to a context broker. It needs a runtime, an -HTTP client, a console for running imports, and a local development environment -including a broker to import into. It has no web UI and no domain data of its -own. - -ITK Dev maintains a fleet of PHP services with an established Docker-based -development convention, expressed as versioned project templates with shared CI -and coding-standards configuration. A new application either adopts that or -diverges from it. - -This ADR serves to decide the runtime, framework and development -environment the application is built on. +This application reads open data sets, converts them to a standard smart-city +representation, and publishes them to a context broker. The organisation +maintains its PHP services on a versioned Docker template carrying shared CI +and coding-standards configuration. This ADR serves to decide the runtime, +framework and development environment the application is built on. ### Drivers -- **Functional:** scheduled console commands; outbound HTTP; a local broker. - No database and no HTTP surface of its own. -- **Non-functional:** minimal onboarding cost; shared tooling rather than - reimplemented tooling; reproducible across developers and CI; long-term - vendor support. +- **Functional:** scheduled console commands, outbound HTTP, and a local broker + to import into. No database and no HTTP surface of its own. +- **Non-functional:** shared tooling rather than reimplemented tooling, minimal + onboarding, reproducible across developers and CI, long-term vendor support. ### Options Considered -1. **PHP 8.4 / Symfony 8 on the ITK Dev `symfony-8` template.** Matches the - organisation's existing stack, so CI, coding standards and task runner come - for free; the console component suits scheduled imports. Provisions a web - server, database and mail catcher this application never uses, and its PHP - version runs ahead of developer hosts, making containers mandatory. -2. **Minimal framework project without the template, run on the host.** No - unused services, no container requirement for the application — but shared - CI and coding-standards config would be reimplemented by hand, and a local - broker needs containers anyway, so the dependency is moved rather than - removed. -3. **A second entry point in an existing internal application.** One - deployment to operate, but couples a batch importer's release cycle to a - user-facing application and inherits dependencies it has no use for. +1. **PHP 8.4 / Symfony 8 on the maintained template.** CI, coding standards and + task runner come for free, and its console suits scheduled imports; it + provisions services this application never uses, and its PHP runs ahead of + developer hosts. +2. **A minimal project on the host, without the template.** No unused services + and no container requirement, but shared configuration is rebuilt by hand + and a local broker needs containers anyway, moving the requirement rather + than removing it. +3. **A second entry point in an existing internal application.** One deployment + to operate, but couples a batch importer to a user-facing release cycle and + inherits dependencies it has no use for. 4. **A different language ecosystem on a bespoke setup.** Richer geospatial - libraries in some ecosystems, but no internal expertise and no shared - tooling. The transformations needed are available as mature libraries in the - established stack too. + libraries, but no internal expertise and no shared tooling; the needed + transformations exist as mature libraries in the established stack. ## Decision -**PHP 8.4 + Symfony 8** on the ITK Dev `symfony-8` template, as its **own -deployable service**, with a containerised broker overlay for local development. +**PHP 8.4 + Symfony 8** on the ITK Dev Docker template, as its **own deployable +service**, with a containerised broker overlay for local development. - Standardising costs less over the application's lifetime than trimming unused - services. A second toolchain must be learned and patched; idle containers + services: a second toolchain must be learned and patched; idle containers cost only disk. -- A batch importer and a user-facing application have different lifecycles and - failure modes, so they stay separate services. +- A batch importer's lifecycle and failure modes differ from a user-facing + application's, so it stays its own service. - No domain persistence is needed — the broker is the system of record — so the - template's database service is left unused rather than removed, keeping - template updates a clean diff. + template's database is left unused rather than removed, keeping template + updates a clean diff. - Local development includes a real broker, so imports are verified end to end rather than only as serialised output. @@ -78,9 +66,8 @@ deployable service**, with a containerised broker overlay for local development. ### Negative / Trade-offs -- **Containers are mandatory.** The template's PHP runs ahead of developer - hosts, so dependency management, console commands and tests cannot run - natively. Most likely source of first-run confusion. +- Containers are mandatory; dependency management, console commands and tests + cannot run natively. - A web server, database and mail catcher are provisioned and never used. - Broker images are not published for every CPU architecture, so local start-up may be slow under emulation. diff --git a/docs/adr/002-publish-to-a-context-broker.md b/docs/adr/002-publish-to-a-context-broker.md index 7372479..a1225c6 100644 --- a/docs/adr/002-publish-to-a-context-broker.md +++ b/docs/adr/002-publish-to-a-context-broker.md @@ -10,69 +10,54 @@ ## Context -The adapter makes data available to consumers the organisation does -not control and cannot brief. The data already exists in operational systems, -with heterogeneous formats, coordinate systems and access methods. What has to -be decided is the mechanism by which it is published. The source systems remain -authoritative; the published copy is not a system of record. - -This ADR serves to decide the mechanism by which data is published to -consumers. +This application republishes data held in operational systems, in heterogeneous +formats and coordinate systems, to consumers the organisation neither controls +nor can brief. The source systems stay authoritative; the published copy is not +a system of record. This ADR serves to decide the mechanism by which data is +published to consumers. ### Drivers -- **Functional:** query by location and attribute, not bulk download only; - several data sets reaching one consumer-facing surface; change notification; - adding a data set without changing consumer integrations. -- **Non-functional:** interpretable by consumers we have never spoken to; - operational cost proportionate to the data and the number of consumers; - existing client tooling rather than clients we supply; an interface that - outlives individual data sets. +- **Functional:** query by location and attribute rather than bulk download + only, several data sets on one consumer-facing surface, change notification, + and adding a data set without changing consumer integrations. +- **Non-functional:** interpretable by consumers never spoken to, served through + client tooling that already exists, operational cost proportionate to the data + and its consumers, and an interface that outlives individual data sets. ### Options Considered -1. **An NGSI-LD context broker.** Provides geospatial and attribute queries, - pagination, subscriptions and a temporal interface without implementing - them; payloads carry a vocabulary reference, so they are self-describing; - many producers converge on one consumer surface. Substantial operational - weight — typically a database and message bus alongside the broker — and the - strictness of a particular implementation is inherited. -2. **Static file export** on a web server or object store. Near-zero - operational cost, trivially cacheable, readable by anything. No query, so - consumers download everything and filter client-side; no change - notification; conventions must be documented in prose because nothing in the - file declares its own meaning. -3. **A bespoke REST API** over our own datastore. Exact fit, full control of - the query surface and semantics. Every capability is ours to build and - maintain — geo-queries, filtering, pagination, notifications, documentation, - clients, versioning — and consumers must learn an interface that exists - nowhere else. -4. **Direct database access or a read replica.** No API layer, powerful ad-hoc - querying. Exposes internal schema as a public contract, requires per-consumer +1. **An NGSI-LD context broker.** Geospatial and attribute queries, pagination, + subscriptions and a temporal interface without implementing them, and + payloads carrying a vocabulary reference; substantial operational weight, and + the strictness of a particular implementation is inherited. +2. **A bespoke REST API** over a datastore of our own. Exact fit and full + control of the query surface and its semantics; every capability from + geo-queries to notifications, clients and versioning is ours to build, and + consumers must learn an interface that exists nowhere else. +3. **Direct database access or a read replica.** No API layer, powerful ad-hoc + querying; exposes internal schema as a public contract, needs per-consumer credentials and network access, and is unusable by browser-based consumers. ## Decision Publish to an **NGSI-LD context broker**. -- Consumers of geographic data need "everything within this area" and - "everything of this kind" more often than the whole data set. A broker - provides that as a standard interface rather than a per-data-set feature. -- Publishing structure without a vocabulary reference requires every consumer - to have our documentation. A broker payload carries the reference. -- For a single small data set a static export would be cheaper and better. Once - several heterogeneous data sets must be published, the fixed operational cost - is paid once while the per-data-set cost approaches zero, and consumers - integrate once rather than once per source. -- New consumers require no change to the adapter, and new data sets require no - change to consumers. -- Existing viewers, dashboards and connectors speak this interface; a bespoke - API would mean supplying clients indefinitely. - -The broker's value is interoperability and query, not storage. If no consumer -reads the data through its interface, a static export would have been the better -decision. Revisit once data sets have been published long enough for consumers -to appear. +1. Consumers of geographic data ask for an area or a kind more often than for a + whole data set, and a broker offers that as a standard interface rather than + a per-data-set feature. +2. Structure published without a vocabulary reference obliges every consumer to + hold separate documentation; a broker payload carries the reference. +3. A single small data set would be served better by a static export. Across + several heterogeneous ones the fixed operational cost is paid once, the + per-data-set cost approaches zero, and consumers integrate once rather than + once per source. +4. New consumers require no change here, new data sets none from consumers, and + existing viewers, dashboards and connectors already speak this interface. + +The value taken is interoperability and query, not storage. Revisit once +consumers have had time to appear: if none read the data through the interface, +a static export was the better decision. ## Consequences @@ -80,25 +65,22 @@ to appear. - Geospatial and attribute queries, pagination, subscriptions and a temporal interface, none of which are implemented here. -- Payloads reference a shared vocabulary, so they need no bespoke - documentation. +- Payloads reference a shared vocabulary, so they need no bespoke documentation. - Additional data sets reach every existing consumer with no integration work. -- The adapter has no database and no read surface of its own. +- This application keeps no database and no read surface of its own. ### Negative / Trade-offs -- **Operational weight out of proportion to a small data set.** A broker - deployment is several services to run, patch, monitor and back up. For a - static data set a file on a web server would serve the same need. -- **Broker implementations impose constraints beyond the standard.** Those - encountered include accepting only one spelling of a UTC timestamp while - rejecting an equivalent one, requiring the vocabulary reference on read - requests — with omission returning an empty success rather than an error — - and collapsing single-element lists to scalars. -- **Vocabulary documents may be fetched over the network during writes**, so - third-party availability becomes part of the import path. -- **No delete semantics.** Upsert creates and updates but never removes, so - records that disappear upstream persist until reconciliation is built. -- JSON-LD is a learning curve for maintainers and consumers. -- Fitting data to a shared vocabulary costs effort that publishing as-is - would not. +- Operational weight out of proportion to a small data set: several services to + run, patch, monitor and back up. +- Broker implementations impose constraints beyond the standard. Those met + include one accepted spelling of a UTC timestamp while an equivalent is + rejected, the vocabulary reference required on reads where omission returns an + empty success rather than an error, and single-element lists collapsed to + scalars. +- Vocabulary documents may be fetched over the network during writes, making + third-party availability part of the import path. +- Upsert creates and updates but never removes, so records that disappear + upstream persist until reconciliation is built. +- JSON-LD is a learning curve for maintainers and consumers, and fitting data to + a shared vocabulary costs effort that publishing as-is would not. diff --git a/docs/adr/003-ngsi-ld-representation.md b/docs/adr/003-ngsi-ld-representation.md index 8adcb7e..ffbb16f 100644 --- a/docs/adr/003-ngsi-ld-representation.md +++ b/docs/adr/003-ngsi-ld-representation.md @@ -10,57 +10,52 @@ ## Context -ADR 002 chooses an NGSI-LD context broker. That leaves two representation -choices open: how attributes are shaped, and which write operation is used. +ADR 002 chooses an NGSI-LD context broker, which leaves the representation +open. -NGSI v2, the older FIWARE API generation, was not considered viable: it has no -`@context`, so a shared vocabulary cannot be expressed, and the -organisation operates no v2 broker. - -This ADR serves to decide how attributes are shaped and which write -operation is used. +This ADR serves to decide how attributes are shaped and which write operation +is used. ### Drivers -- **Functional:** the broker must accept the payload on write; attribute-level - metadata must be expressible; repeated imports must not duplicate entities. -- **Non-functional:** idempotency; payload size; readability for consumers. +- **Functional:** the broker must accept the payload on write, attribute + metadata must be expressible, and repeated imports must not duplicate + entities. +- **Non-functional:** idempotency, payload size, readability for consumers. ### Options Considered #### Attribute form -1. **Normalized** — every attribute an object naming its kind. Verbose, but - the form brokers accept on write and the only one carrying metadata such as - `observedAt`. -2. **Key-values** — flat `name: value`. Much smaller and easier to read, but - read-only and cannot carry attribute metadata. +1. **Normalized.** Every attribute an object naming its kind. Verbose, but the + form brokers accept on write and the only one carrying metadata. +2. **Key-values.** Flat `name: value`, smaller and easier to read, but + read-only and unable to carry attribute metadata. #### Write operation -1. **Batch upsert** — creates or updates. Idempotent when identifiers are - derived from source keys. -2. **Create** — fails with `409` for identifiers that already exist, so a +1. **Batch upsert.** Creates or updates, so a re-import refreshes in place. +2. **Create.** Fails with `409` for an identifier that already exists, so a re-import errors rather than refreshing. -3. **Batch replace** — silently drops attributes absent from the payload, - making partial payloads destructive. +3. **Batch replace.** Silently drops attributes absent from the payload, + making a partial payload destructive. ## Decision Publish **normalized** NGSI-LD with `Content-Type: application/ld+json`, via **batch upsert**. -- Normalized is the only form accepted on write, so key-values is not a real - option for a producer. -- Upsert makes imports idempotent: identifiers derive from each source's - primary key, so re-running updates in place. +- Normalized is the only form accepted on write, so key-values is not + available to a producer. +- Identifiers derive from each data set's primary key, so upsert makes a + re-run update in place rather than add. ## Consequences ### Positive - Re-imports produce no duplicates and need no prior state. -- Attribute metadata remains available if a source ever supplies it. +- Attribute metadata stays expressible where a data set supplies it. ### Negative / Trade-offs diff --git a/docs/adr/004-coordinate-reference-system.md b/docs/adr/004-coordinate-reference-system.md index 177d6b7..f77f88a 100644 --- a/docs/adr/004-coordinate-reference-system.md +++ b/docs/adr/004-coordinate-reference-system.md @@ -11,102 +11,83 @@ ## Context Every entity the adapter publishes carries a `location` GeoProperty, so the -coordinate reference system is a cross-cutting concern rather than a per-source -detail. +coordinate reference system is cross-cutting rather than a per-input detail. +Inputs arrive in whatever CRS their publisher uses — Danish municipal data is +commonly projected, typically EPSG:25832 (ETRS89 / UTM zone 32N) as eastings and +northings in metres — and no single input CRS can be assumed. A GeoJSON envelope +states a geometry type but not units, so projected coordinates arrive inside one +undetected. -Input data arrives in whatever CRS its publisher uses. Danish municipal data is -commonly projected — typically EPSG:25832 (ETRS89 / UTM zone 32N), as eastings -and northings in metres. Other inputs may already be geographic, or use a -different projection. The adapter cannot assume one input CRS. - -Projected coordinates are sometimes delivered inside a GeoJSON envelope, which -states a geometry type but not units. The data models specify only that -`location` is GeoJSON and make no reference to a coordinate system; the -constraint comes from GeoJSON itself. - -This ADR serves to decide which coordinate reference system is published, -and at what precision. +This ADR serves to decide which coordinate reference system is published, and at +what precision. ### Drivers -- **Functional:** consumers must be able to interpret `location` unbriefed; - geo-queries must return correct results; clients must render without - preprocessing; one rule must hold for every input. -- **Non-functional:** self-description; conformance; uniformity across inputs; - precision no worse than the input. +- **Functional:** `location` is interpretable unbriefed, geo-queries return + correct results, and clients render without preprocessing. +- **Non-functional:** conformance and self-description; one rule for every + input; precision no worse than the input. ### Options Considered -1. **Normalise everything to WGS84, reprojecting in the adapter.** Conforms to - RFC 7946; self-describing; geo-queries work; one rule however many input - CRSs accumulate. Requires a reprojection dependency, and each input must - declare its CRS. -2. **Pass each input's native CRS through unchanged.** No transformation, no - dependency — but produces invalid GeoJSON with nowhere to declare the CRS, - makes entities from different inputs mutually incomparable, and breaks - geo-queries because distances are read as degrees. Every failure is silent. -3. **Publish WGS84 and also retain original coordinates in an extra - attribute.** Avoids a round trip for consumers wanting native coordinates, - but the attribute cannot have a stable shape: each input brings its own CRS - and geometry type, and it would be absent for inputs already in WGS84. A - consumer cannot code against that, so it would go unused. -4. **Pass native CRSs through under RFC 7946's "prior arrangement" clause, - documenting each out of band.** Permitted by the RFC, but the clause - requires all parties to have agreed — incompatible with a broker whose - consumers are unknown by design. +1. **Normalise everything to WGS84, reprojecting in the adapter.** One rule + however many input CRSs accumulate. Needs a reprojection dependency, and each + input must declare its CRS. +2. **Pass each input's native CRS through unchanged.** No transformation and no + dependency, but the GeoJSON is invalid, entities from different inputs are + incomparable, and distances read as degrees. +3. **Publish WGS84 and also keep the original coordinates in an extra + attribute.** Saves consumers a round trip, but the attribute has no stable + shape — CRS and geometry type differ per input, and it is absent for inputs + already in WGS84 — so nothing can be coded against it. +4. **Pass native CRSs through under RFC 7946's "prior arrangement" clause.** + Permitted, but the clause requires all parties to have agreed, which a broker + whose consumers are unknown by design cannot satisfy. ## Decision -Every `location` the adapter emits is **EPSG:4326 (WGS84) -longitude/latitude**, at **full precision — coordinates are not rounded**. Each -input declares its own CRS; reprojection happens at the boundary between reading -an input and building an entity, and nowhere else. - -- **The specification is unambiguous.** RFC 7946 §4: "The coordinate reference - system for all GeoJSON coordinates is a geographic coordinate reference - system, using the World Geodetic System 1984 (WGS 84) datum, with longitude - and latitude units of decimal degrees." NGSI-LD GeoProperty values are - GeoJSON, so the requirement is inherited. -- **There is no way to declare otherwise.** RFC 7946 Appendix B.1: - "Specification of coordinate reference systems has been removed, i.e., the - 'crs' member of [GJ2008] is no longer used." Publishing projected coordinates - means publishing an undeclarable assumption. Inputs may still carry that - deprecated member; it can be read, but not passed on. -- Entities from different inputs are queried together, so a query spanning two - inputs published in different CRSs returns meaningless results. -- A broker given projected coordinates accepts them, answers geo-queries - incorrectly, and renders points in the wrong location. No error is raised at - any stage. -- **The conversion is lossless at full precision.** A projected-to-geographic - round trip returns the input exactly when no rounding is applied. Rounding - trades accuracy for a marginal reduction in payload size. -- `source` and `seeAlso` can reference the originating export, which states its - own CRS. A coordinate copied into an extra attribute states nothing. +Every `location` the adapter emits is **EPSG:4326 (WGS84) longitude/latitude**, +at **full precision — coordinates are not rounded**. Each input declares its own +CRS; reprojection happens at the boundary between reading an input and building +an entity, and nowhere else. + +1. **The specification leaves no choice.** RFC 7946 §4 fixes the CRS for all + GeoJSON coordinates as geographic, WGS 84 datum, longitude and latitude in + decimal degrees; NGSI-LD GeoProperty values are GeoJSON and inherit it. +2. **There is no way to declare otherwise.** RFC 7946 Appendix B.1 removed the + `crs` member of the 2008 specification, so projected coordinates publish an + undeclarable assumption. An input may still carry that deprecated member; it + can be read, not passed on. +3. Nothing catches the mistake: a broker accepts projected coordinates, answers + geo-queries incorrectly and places points wrongly, raising no error; and + entities from different inputs are queried together, so a query spanning two + CRSs returns meaningless results. +4. **A round trip is lossless at full precision.** Projected to geographic and + back returns the input exactly when nothing is rounded; rounding trades + accuracy for a marginal payload reduction. +5. `source` and `seeAlso` can reference the originating export, which states its + own CRS. ## Consequences ### Positive -- Payloads are valid GeoJSON and NGSI-LD; geo-queries work and are comparable - across inputs; any client renders them unmodified. -- One rule for every present and future input. -- Reprojection is isolated in one component with its own tests, verified - against independently known reference coordinates, so a regression fails - loudly instead of silently relocating data. +- Payloads are valid GeoJSON and NGSI-LD, comparable across inputs, and render + in any client unmodified, under one rule for every present and future input. +- Reprojection is isolated in one tested component, verified against + independently known reference coordinates, so a regression fails loudly rather + than silently relocating data. ### Negative / Trade-offs -- Adds a reprojection dependency. National grid definitions are not always - shipped and may need registering explicitly, making them load-bearing - project code. -- **Datum shifts are approximated.** ETRS89-based grids are treated as - equivalent to WGS84 via a null datum transformation. The two were coincident - in 1989 and have diverged by roughly 0.5–1 m since, at about 2.5 cm per year. - What is published is therefore ETRS89 labelled WGS84. This is standard - practice in web GIS, but it is the largest error in the pipeline — greater - than the source's own positional accuracy — so a consumer using a rigorous - transformation with an explicit epoch will land about a metre away. +- Adds a reprojection dependency, and national grid definitions are not always + shipped, so registering them becomes load-bearing project code. +- Datum shifts are approximated: ETRS89-based grids are treated as equivalent + to WGS84 via a null datum transformation, so what is published is ETRS89 + labelled WGS84. Coincident in 1989, the two have diverged by roughly 0.5–1 m + at about 2.5 cm per year — standard practice in web GIS, and the largest + error introduced, so a consumer transforming rigorously with an explicit + epoch lands about a metre away. - Consumers with natively projected stacks must convert. - Every new input must declare its CRS, and unsupported ones need adding. -- Only point geometries were implemented initially; line and area geometries - were added later. +- Each geometry type needs its own reprojection, not points alone. diff --git a/docs/adr/005-smart-data-models-as-vocabulary.md b/docs/adr/005-smart-data-models-as-vocabulary.md index 42f11ec..7cba550 100644 --- a/docs/adr/005-smart-data-models-as-vocabulary.md +++ b/docs/adr/005-smart-data-models-as-vocabulary.md @@ -10,117 +10,87 @@ ## Context -NGSI-LD defines how attributes are carried and how to reference a vocabulary. -It does not define entity types or attribute names. Without a vocabulary the -JSON-LD context resolves to nothing, entity types are local strings, and -consumers still need our documentation to interpret anything. - -The choice also determines the cost of onboarding each data set, since mapping -a source onto an existing model takes more effort than exposing its fields -verbatim. - -This ADR serves to decide which vocabulary supplies entity types and +NGSI-LD defines how attributes are carried and how a vocabulary is referenced, +not which entity types and attribute names exist. Without one the JSON-LD +context resolves to nothing and consumers need our documentation to interpret +anything. This ADR serves to decide which vocabulary supplies entity types and attribute names, and the rules for mapping sources onto it. ### Drivers -- **Functional:** types and attributes interpretable without our - documentation; expressible as a JSON-LD context a broker can resolve; - coverage across the domains in scope. -- **Non-functional:** a vocabulary consumers plausibly already know; governed - and maintained by someone else; mapping cost that does not dominate - onboarding. +- **Functional:** types and attributes interpretable without our documentation, + expressible as a JSON-LD context a broker can resolve, covering the domains in + scope. +- **Non-functional:** a vocabulary consumers plausibly already know, maintained + by someone else, at a mapping cost that does not dominate onboarding. ### Options Considered -1. **Smart Data Models.** Purpose-built for NGSI-LD and the reference - vocabulary of that ecosystem; publishes JSON-LD context documents per - domain; broad coverage; each model ships a JSON schema and examples, giving - an objective conformance target; open governance. Model depth varies; - many models assume real-time sensing, so static inventory leaves attributes - unset; required attributes occasionally presuppose a hierarchy the source - lacks; enum spellings sometimes disagree between a model's schema and its - examples; versioning is loose. -2. **A vocabulary of our own, with self-hosted context documents.** Exact fit, - no required attributes we cannot satisfy, full control of naming and - versioning. Nobody else speaks it, so consumers return to reading our - documentation; governance, documentation and versioning become ours - indefinitely; no existing tooling recognises the types. -3. **A general-purpose web vocabulary.** Widely recognised, stable governance, - adequate for names, addresses and descriptions. No NGSI-LD conventions for - geometry or relationships, and no domain-specific terms, so the domains in - scope would remain unmodelled. +1. **Smart Data Models.** The openly governed reference vocabulary of the + NGSI-LD ecosystem, with broad coverage, per-domain context documents, and a + schema and examples per model to conform against. Depth varies, many models + assume real-time sensing, required attributes can presuppose a hierarchy a + source lacks, and versioning is loose. +2. **A vocabulary of our own, with self-hosted context documents.** Exact fit + and full control of naming and versioning, but nobody else speaks it, + governance and documentation stay ours indefinitely, and no existing tooling + recognises the types. +3. **A general-purpose web vocabulary.** Widely recognised and stably governed, + adequate for names, addresses and descriptions, but with no NGSI-LD + conventions for geometry or relationships and no domain terms, leaving the + domains in scope unmodelled. ## Decision Adopt **Smart Data Models**, referencing the relevant domain context documents -alongside the NGSI-LD core context. - -Two rules follow, and they matter more than the choice itself: +alongside the NGSI-LD core context. Which model a given data set uses is +recorded in its own ADR; this one states policy. 1. **Use an existing model; do not invent a type.** An imperfect standard type - is more useful to a consumer than a perfect private one. + is more useful to a consumer than a precise private one. 2. **Never fabricate a value to satisfy a model.** An absent attribute is - honest; a fabricated one is indistinguishable from a measured one. Applied: - + honest; a fabricated one is indistinguishable from a measured one. - Attributes the source cannot fill are left unset, not approximated, defaulted or inferred. - Where a model *requires* an attribute the source cannot supply, choose a - different model rather than inventing the value. Required relationships - are the common case: inventing a related entity yields something - schema-valid and factually wrong that must then be maintained - indefinitely. A sibling model without the requirement is the better - choice even if its terms are less precise. - - In a hierarchy — a root site, subdivisions beneath it, individual units - beneath those, each lower level requiring a relationship upward — publish - at the highest level the source can populate. Static inventory typically - describes a location and a count of units without describing what the - location is part of, so the site level is usually correct. - -The concrete model chosen for a given data set is recorded in its own ADR; this -one states policy only. + sibling model without the requirement, even if its terms are less precise. + - In a hierarchy where each level requires a relationship upward, publish at + the highest level the source can populate — typically the top, for a data + set giving a location and a count of units within it. Rationale: -- The context must resolve to terms a consumer recognises, or publishing gains - nothing over a file. -- Smart Data Models is the vocabulary the surrounding ecosystem uses and ships - the context documents needed to reference it, so adoption is a URL rather - than a project. +- A context must resolve to terms a consumer recognises, or publishing gains + nothing over a file, and Smart Data Models ships those documents, so adoption + is a URL rather than a project. - Shipped schemas and examples make modelling disagreements checkable against a specification instead of settled by preference. -- Mandatory relationships propagate downward: choosing a subdivision level - schedules the need for a parent rather than avoiding it, because the - individual-unit level requires a site as well. -- The costs are asymmetric. Publishing at site level and later finding real - sites exist means a one-off migration. Publishing at subdivision level and - never acquiring real sites means maintaining an invented entity - indefinitely, with every consumer that follows the relationship receiving - something meaningless. +- Required relationships propagate downward, so publishing below the top level + defers the need for a parent rather than avoiding it, and the costs are + asymmetric: acquiring real parents later is a one-off migration, whereas never + acquiring them means maintaining an invented entity that every consumer + following the relationship receives as meaningless. ## Consequences ### Positive -- Types and attributes resolve to shared global identifiers. -- Consumers may already have code for the types published. -- Modelling decisions have an external reference point. -- Later data sets are likely already covered, so onboarding does not start with - vocabulary design. -- Published entities are self-contained, with nothing invented to keep in sync. -- Finer granularity can be added later beneath what is already published. +- Types and attributes resolve to shared global identifiers consumers may + already have code for. +- A later data set is likely covered already, so onboarding it does not start + with vocabulary design. +- Published entities are self-contained, with nothing invented to keep in sync, + and finer granularity can be added beneath them later. ### Negative / Trade-offs -- **Many attributes will always be empty.** Models built around real-time - sensing carry availability, occupancy and detection attributes that static - inventory cannot fill. -- **Model choice is embedded in entity identifiers.** Changing model later - means deleting and re-publishing rather than updating in place, so selection - deserves attention before a data set is first published. -- **Enum values must be read from the schema, not the examples.** Where the two - disagree the schema is authoritative, and equivalent-looking values differ - between sibling models, so they must not be copied across. -- **Loose versioning.** A model can change without an obvious signal. +- Models built around real-time sensing carry attributes static inventory + cannot fill, so many will always be empty. +- Model choice is embedded in entity identifiers, so changing model later means + deleting and re-publishing rather than updating in place. +- Enum values must be read from the schema rather than the examples: where the + two disagree the schema is authoritative, and equivalent-looking values differ + between sibling models, so they cannot be copied across. +- Versioning is loose, so a model can change without an obvious signal. - Mapping a source to a model takes longer than exposing its fields verbatim, and occasionally the fit is poor. diff --git a/docs/adr/006-onstreetparking-over-parkinggroup.md b/docs/adr/006-onstreetparking-over-parkinggroup.md index 8b04272..f72fe3f 100644 --- a/docs/adr/006-onstreetparking-over-parkinggroup.md +++ b/docs/adr/006-onstreetparking-over-parkinggroup.md @@ -10,54 +10,42 @@ ## Context -The parking domain is organised as a hierarchy: - -```text -OnStreetParking / OffStreetParking site — no parent, requires id, type, location - └── ParkingGroup subdivision — requires refParkingSite - └── ParkingSpot individual unit — requires refParkingSite, status, category -``` +The parking domain is a hierarchy. A site — `OnStreetParking` or +`OffStreetParking` — requires only `id`, `type` and `location`. Below it, a +`ParkingGroup` subdivision and a `ParkingSpot` unit must each reference a site, +and a spot also requires `status` and `category`. This decision applies where a data set provides a count of units per location with a point geometry, no reference to a containing site, and neither per-unit geometry nor occupancy. -Entity identifiers embed the type by convention, so the choice must be made -before first publication: changing it afterwards means deleting and -re-publishing. - This ADR serves to decide which model in the parking hierarchy is published under those conditions. ### Drivers -- **Functional:** every mandatory relationship must point at an entity that - exists; the restriction on who may park must be expressible unambiguously; - finer granularity addable later without restructuring what is published. -- **Non-functional:** nothing invented purely to satisfy a schema; a choice - that is cheap to reverse in preference to one that is not. +- **Functional:** every mandatory relationship points at an entity that exists; + the restriction on who may park is expressible unambiguously; finer + granularity is addable later without restructuring. +- **Non-functional:** nothing invented purely to satisfy a schema; a reversible + choice in preference to one that is not. ### Options Considered -1. **`ParkingGroup`, creating the missing parent site.** `category` offers - `onlyDisabled`, which by name states exclusivity, and the model's reference - example for disabled parking sits at this level. But `refParkingSite` is - mandatory and no value is available for it, so a parent must be invented; - one spanning the whole administrative area asserts a false containment, and - its own mandatory geometry would carry no meaning. - `ParkingSpot` also requires a site, so adding per-unit data later would force - the invented entity into existence after entities had been published against - it. +1. **`ParkingGroup`, creating the missing parent site.** `onlyDisabled` states + exclusivity by name, and the model's reference example for disabled parking + sits at this level — but points at a real street-address site. + `refParkingSite` is mandatory with no value available, so a parent must be + invented; one spanning the administrative area asserts a false containment + and its own mandatory geometry carries no meaning. 2. **`ParkingGroup`, omitting `refParkingSite`.** Nothing invented, smallest change — but knowingly non-conformant, and a schema validator flags every entity. -3. **`OnStreetParking`.** Requires only `id`, `type` and `location`, all of - which are available. It is the entity both `ParkingGroup` and - `ParkingSpot` are required to reference, so finer granularity can be - attached beneath it, and migration down to `ParkingGroup` stays possible if - real site data appears. `category` offers only `forDisabled`, which does not - state exclusivity as plainly. -4. **`ParkingSpot`.** Models an individual unit, but only a count per location +3. **`OnStreetParking`.** Everything it requires is available, and it is the + entity both lower levels must reference, so granularity can be added beneath + it and migration downward stays possible if real sites appear. `category` + offers only `forDisabled`, which states exclusivity less plainly. +4. **`ParkingSpot`.** Models the individual unit, but only a count per location is available, `status` is mandatory with no occupancy data, and a parent site is required. Rejected outright. @@ -66,39 +54,32 @@ under those conditions. Publish each record as an **`OnStreetParking`** entity, with `category: ["forDisabled"]` and no `refParkingSite`. -- It is the only option that invents nothing; everything the model requires is - available. -- Mandatory relationships propagate downward, so choosing `ParkingGroup` would - schedule the invented parent rather than avoid it — `ParkingSpot` requires a - site too. +- The only option that invents nothing: everything it requires is available. +- Mandatory references propagate downward, so `ParkingGroup` would schedule the + invented parent rather than avoid it — `ParkingSpot` requires a site too. - The costs are asymmetric. Site level, then finding real sites exist, is a - one-off migration. Subdivision level, then never acquiring real sites, means + one-off migration; subdivision level, then never acquiring real sites, means maintaining an invented entity indefinitely. -The model's reference example does use `ParkingGroup` for disabled parking, but -points at a real street-address site. It shows what to do when a site exists, -not when none does. - ## Consequences ### Positive -- No dangling relationship; every entity is self-contained. -- Nothing invented to create, document or keep in sync. +- No dangling relationship: every entity is self-contained, and nothing invented + has to be created, documented or kept in sync. - Conformant to the model's schema without exceptions. - `ParkingGroup` or `ParkingSpot` entities can be attached beneath these later without changing them. ### Negative / Trade-offs -- **`forDisabled` does not state exclusivity.** The schema documents `category` - only as "Street parking category" with an enum list and defines no individual - value. The two models' enums use inconsistent prefixes for what appear to be - the same concepts — `forDisabled` / `forResidents` against `onlyDisabled` / - `onlyResidents` — while both carry `onlyWithPermit`, so the spelling cannot - be relied on to carry exclusivity. -- Values must not be copied between the two models' `category` enums. -- Entity type is embedded in identifiers, so any later change means deleting - and re-publishing. +- `forDisabled` does not state exclusivity. The schema documents `category` only + as "Street parking category" with an enum list and defines no individual + value, and the two models prefix apparently identical concepts inconsistently + — `forDisabled` / `forResidents` against `onlyDisabled` / `onlyResidents` — + while both carry `onlyWithPermit`, so values must not be copied between the + enums. +- Entity type is embedded in identifiers, so any later change means deleting and + re-publishing. - Domain experts may find a single address described as a "site" counter-intuitive.