Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
ecaba9c
Added ADR folder along with ADRs existing so far
jeppekroghitk Sep 1, 2026
9aa1a0b
Updated README
jeppekroghitk Sep 1, 2026
9dadf62
Updated CHANGELOG
jeppekroghitk Sep 1, 2026
6de59da
Added pointers to .env
jeppekroghitk Sep 1, 2026
a56f14c
Added a few commands to Taskfile
jeppekroghitk Sep 1, 2026
ff7146d
Added wgs84 transformer class
jeppekroghitk Sep 1, 2026
0386a7d
Added a generic way to retrieve data from broker via terminal
jeppekroghitk Sep 1, 2026
492d105
Added feedReader
jeppekroghitk Sep 1, 2026
6a2ebea
Added import command
jeppekroghitk Sep 1, 2026
1fdbf47
Added source interface for picking up new source adapters
jeppekroghitk Sep 1, 2026
d518d75
Added mtm spatialmaps handicap parking source adapter
jeppekroghitk Sep 1, 2026
1cde88a
Added NgsiEntity for normalized NGSI-LD output
jeppekroghitk Sep 1, 2026
20798f2
Added NgsiLdBroker for idempotent batch upserts
jeppekroghitk Sep 1, 2026
61f6190
Added tests
jeppekroghitk Sep 1, 2026
db7575b
Renaming paths to new source location structure
jeppekroghitk Sep 1, 2026
8e4f7ca
Added pull request template
jeppekroghitk Sep 1, 2026
4f28de9
Coding standards
jeppekroghitk Sep 1, 2026
b04cb63
Converted all ADRs in readme to draft
jeppekroghitk Sep 1, 2026
22d5017
Added ADR 007: data set metadata in a committed source manifest
jeppekroghitk Sep 4, 2026
8eb2a37
Added source manifest and catalog reader
jeppekroghitk Sep 4, 2026
8e36832
Moved handicap parking feed config from .env to the source manifest
jeppekroghitk Sep 4, 2026
0bd7178
Updated README and changelog for the source manifest
jeppekroghitk Sep 4, 2026
1d1e303
Cleaned up sources.yaml
jeppekroghitk Sep 7, 2026
fd623fc
Cleaned up ADR 007
jeppekroghitk Sep 7, 2026
bfcac87
Shortened comment
jeppekroghitk Sep 7, 2026
f51f611
Updated changelog
jeppekroghitk Sep 7, 2026
e556c01
Updated README
jeppekroghitk Sep 7, 2026
8e0627b
Removed comment from Taskfile
jeppekroghitk Sep 7, 2026
c29945e
Fixed coding standards
jeppekroghitk Sep 7, 2026
01cfeb5
Update src/Geo/Wgs84Transformer.php
jeppekroghitk Sep 7, 2026
b949429
Update src/Geo/Wgs84Transformer.php
jeppekroghitk Sep 7, 2026
684f13e
Update src/Ngsi/NgsiEntity.php
jeppekroghitk Sep 7, 2026
43ef3ac
Update src/Source/SourceInterface.php
jeppekroghitk Sep 7, 2026
21d7559
Update src/Source/FeedReader.php
jeppekroghitk Sep 7, 2026
93897e3
Update src/Source/FeedReader.php
jeppekroghitk Sep 7, 2026
2a56d2c
Renamed FeedReader to DataSourceReader
jeppekroghitk Sep 7, 2026
e4452fd
Validated the source manifest as Symfony config
jeppekroghitk Sep 7, 2026
02cc6cf
Fixed error in pull request template
jeppekroghitk Sep 7, 2026
210c5ac
Corrected adr 007
jeppekroghitk Sep 7, 2026
538ae48
ADR 007 edits
jeppekroghitk Sep 7, 2026
cda1ba2
Raname calls to renamed methods
jeppekroghitk Sep 7, 2026
365325a
Coding standards
jeppekroghitk Sep 7, 2026
0ed7598
Grouped the manifest classes under App\Source\Manifest
jeppekroghitk Sep 8, 2026
78c3538
Cleaned up comments
jeppekroghitk Sep 8, 2026
05fa238
Minor correction in comment
jeppekroghitk Sep 8, 2026
603147d
Extracted the import logic into a DataSourceImporter service
jeppekroghitk Sep 8, 2026
07aec4f
Reduced ImportCommand to console plumbing
jeppekroghitk Sep 8, 2026
64aa10f
Move context from comma seperated list in .env to each dataset via co…
jeppekroghitk Sep 8, 2026
87e540f
Minor corrections
jeppekroghitk Sep 8, 2026
c05adfe
Removed task broker:entities for now
jeppekroghitk Sep 8, 2026
a97f602
Removed comment from .env
jeppekroghitk Sep 8, 2026
346cbc6
Cut some comments
jeppekroghitk Sep 8, 2026
3353d08
Coding standards
jeppekroghitk Sep 8, 2026
388d88a
ADR shortening
jeppekroghitk Sep 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,9 @@ DEFAULT_URI=http://localhost

###> app ###
APP_BROKER_BASE_URI=http://scorpio.local:9090/

# JSON-LD contexts attached to every entity, outermost last so the ETSI core
# context resolves the NGSI-LD terms and the domain context resolves the
# Smart Data Models ones.
ENTER_NGSI_CONTEXT_URLS='https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld'
###< app ###
18 changes: 18 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#### Link to ticket

Please add a link to the ticket being addressed by this change.

#### Description

Please include a short description of the suggested change and the reasoning behind the approach you have chosen.

#### Screenshot of the result

If your change affects the user interface you should include a screenshot of the result with the pull request.

#### Checklist

- [ ] My code is covered by test cases.
- [ ] My code passes our test (all our tests).
- [ ] My code passes our static analysis suite.
- [ ] My code passes our continuous integration process.
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

* [#3](https://github.com/itk-dev/enter/pull/3)
* Import command that reads a geospatial feed, reprojects it to WGS84 and upserts it to an NGSI-LD broker.
* A committed record per data set — feed, CRS, model, DCAT-AP metadata.
* An extension point for adding data sets, a test suite, and architecture decision records.
* Source manifest validated against a Symfony config tree and read during container warm-up, so a malformed
entry fails the build rather than the one import that selects it.

[Unreleased]: https://github.com/itk-dev/enter
158 changes: 26 additions & 132 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ We use [DDEV](https://ddev.com/) and [Task](https://taskfile.dev/) for developme

``` shell
task site:install
```

``` shell
task site:update
Expand All @@ -12,150 +13,43 @@ ddev launch

Run `task` to see what cool task are available. Running `ddev` can help with other stuff.

## Broker

A [Scorpio Broker](https://scorpio.readthedocs.io/) is part of the development setup.

``` shell
ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/types | jq"
```

Load some example data:
## Adapter

``` shell name=import-toilet
ddev console app:broker:entity:delete toilet
ddev console app:broker:import:geojson toilet 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter'
ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --get --data-urlencode type=toilet" | jq '.[]|with_entries(select([.key] | inside(["id", "type", "location"])))'
```
Takes an open-data set, converts it to [NGSI-LD], and upserts it into the
context broker.

``` shell name=import-handicapparkering
ddev console app:broker:entity:delete handicapparkering
ddev console app:broker:import:geojson handicapparkering 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap'
ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --get --data-urlencode type=handicapparkering" | jq '.[]|with_entries(select([.key] | inside(["id", "type", "location"])))'
``` text
source feed (JSON)
→ SourceInterface implementation maps fields, fixes quirks, picks the data model
→ NgsiEntity normalized NGSI-LD: Property / GeoProperty / Relationship
→ NgsiLdBroker POST /ngsi-ld/v1/entityOperations/upsert
→ context broker
```

``` shell name=import-hundeskov
ddev console app:broker:entity:delete hundeskov
ddev console app:broker:import:geojson hundeskov 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=hundeskove_friluftsliv_aarhus'
ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --get --data-urlencode type=hundeskov" | jq '.[]|with_entries(select([.key] | inside(["id", "type", "location", "geometry"])))'
``` shell
task import # list the available sources
task import -- mtm_spatialmaps-handicap-parking # import one
task import -- mtm_spatialmaps-handicap-parking --dry-run --limit 5 # print the payload instead
```
Comment on lines +29 to 33

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thinks we should tell how to run these commands with docker compose rather than with a task. We (currently) mostly use task for local development and for productions one should use docker compose (and we'll use itkdev-docker-compose-server, but that's our choice and caused by our hosting setup).


### Broker API request examples

<https://scorpio.readthedocs.io/en/latest/API_walkthrough.html#entity-creation>

``` shell name=scorpio-entity-create substitutions="{«entity-type»: Room, «entity-id»: 'house2:smartrooms:room1'}"
ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities --header 'content-type: application/json' --data @-" <<'JSON'
{
"type": "«entity-type»",
"id": "«entity-id»"
}
JSON

### Source manifest

# EPSG:4326?!
ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/entities/«entity-id»/attrs --header 'content-type: application/json' --data @-" <<'JSON'
{
"location": {
"type": "geo:json",
"value": {
"type": "Point",
"coordinates": [
10.15711687080293, 56.126271111641266
]
}
}
}
JSON
Every data set is recorded in [config/sources.yaml](config/sources.yaml), keyed
by the identifier `app:import` takes as its argument. See [ADR 007](docs/adr/007-source-manifest.md).

Adding a data set means adding one `SourceInterface` implementation and one
manifest entry. The class is discovered through
`#[AutoconfigureTag('app.source')]` and shows up as an `app:import` argument
with no further wiring.

# Get the entities
ddev exec "curl --silent --header 'accept: application/ld+json' http://scorpio.local:9090/ngsi-ld/v1/entities?type=«entity-type» | jq"
Design decisions are recorded in [docs/adr](docs/adr/README.md).

# Get the entities as GeoJSON
ddev exec "curl --silent --header 'accept: application/geo+json' http://scorpio.local:9090/ngsi-ld/v1/entities?type=«entity-type» | jq"
```

> [!CAUTION]
> Excuse me what?!
>
> ``` shell
> ddev exec "curl --silent --header 'accept: application/geo+json' 'http://scorpio.local:9090/ngsi-ld/v1/entities?georel=near;maxDistance%3D%3D2000&geometry=Point&coordinates=%5B8,40%5D'"
> ddev exec "curl --silent --header 'accept: application/geo+json' 'http://scorpio.local:9090/ngsi-ld/v1/entities?georel=near;maxDistance==2000&geometry=Point&coordinates=%5B8,40%5D'"
> ddev exec "curl --silent --header 'accept: application/geo+json' 'http://scorpio.local:9090/ngsi-ld/v1/entities?georel=near;maxDistance==2000&geometry=Point&coordinates=[10,56]'"
> > ```

## GeoJSON

* "[Position](https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1)": `(longitude, latitude)`
* "[Coordinate Reference System](https://datatracker.ietf.org/doc/html/rfc7946#section-4)"
* But … <https://gis.stackexchange.com/a/437311>

<https://www.opendata.dk/city-of-aarhus/toiletter-i-aarhus-kommune> → <https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter>:

```json
{
"type": "FeatureCollection",
"crs": {
"type": "name",
"properties": {
"name": "EPSG:25832"
}
},
"bbox": [563262.0903961, 6209995.05346621, 579587.800694027, 6231954.36230685],
"features": [
{
"type": "Feature",
"geometry": {
"type": "MultiPoint",
"coordinates": [
[576933.018139537, 6218035.22691216]
]
},
"properties": {
```

* <https://geojson.com/>

---

* <https://http.dev/tools>
* <https://httpbin.io/>
* <https://epsg.io/transform#s_srs=4326&t_srs=25832&x=NaN&y=NaN>
* <https://fiware-datamodels.readthedocs.io/en/stable/guidelines/index.html#modelling-location>

Must `location` be a `Point` in ngsi-ld?
[NGSI-LD]: https://www.etsi.org/committee/cim

``` shell
ddev exec --service scorpio-db "psql ngb ngb"
```

``` sql
2026-08-30 10:26:19.789 UTC [41] LOG: execute 0000000: WITH D0 AS (SELECT ID, ENTITY, TRUE as PARENT FROM ENTITY WHERE ST_DWithin( location::geography, ST_SetSRID(ST_GeomFromGeoJSON('{"type": "Point", "coordinates": [10.236808047161853,56.101226991176155] }'), 4326)::geography, 2000.0) ORDER BY createdAt limit $1 offset $2) SELECT ID, ENTITY, PARENT FROM D0
```
## Broker

<https://postgis.net/docs/ST_GeomFromGeoJSON.html>
A [Scorpio Broker](https://scorpio.readthedocs.io/) is part of the development setup.

``` shell
ddev exec --service scorpio-db "psql ngb ngb" <<< "SELECT id , e_types, location FROM entity;"
```

``` shell name=hmm
ddev console app:import:geojson toilet 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter' \
&& ddev exec --service scorpio-db "psql ngb ngb" <<< "SELECT id , e_types, ST_AsText(location) AS location FROM entity WHERE id = 'toilet:0000';" \
&& ddev exec --service scorpio-db "psql ngb ngb" <<< "SELECT temporalentity_id, ST_AsText(location) AS location, ST_asText(geovalue) AS geovalue, createdat FROM temporalentityattrinstance WHERE temporalentity_id = 'toilet:0000' ORDER BY createdat DESC LIMIT 10;"
ddev exec "curl --silent http://scorpio.local:9090/ngsi-ld/v1/types | jq"
```

<https://www.opendata.dk/search?q=res_format:GeoJSON%20organization:city-of-aarhus>

* [Public toilets in Aarhus Municipality](https://www.opendata.dk/city-of-aarhus/toiletter-i-aarhus-kommune)
* [Offentlige
toiletter](http://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter):
<http://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=andre_toiletter>
* [Parking in Aarhus Municipality – Zones, Permits and
Spaces](https://www.opendata.dk/city-of-aarhus/parkering-i-aarhus-kommune)
* [Handicapparkering](https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap):
<https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap>

* <https://enter.ddev.site:33001/data/ngsi-ld/v1/types>
4 changes: 4 additions & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ tasks:
test:integration: *test_task
test:application: *test_task

import:
desc: 'Import a source into the broker, e.g. task import -- mtm_spatialmaps-handicap-parking'
cmd: ddev console app:import {{.CLI_ARGS}}

coding-standards:apply:
desc: 'Apply coding standards'
cmds:
Expand Down
1 change: 1 addition & 0 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"proj4php/proj4php": "^2.0",
"symfony/asset": "~8.1.0",
"symfony/asset-mapper": "~8.1.0",
"symfony/config": "~8.1.0",
"symfony/console": "~8.1.6",
"symfony/dotenv": "~8.1.6",
"symfony/flex": "^2",
Expand Down
2 changes: 1 addition & 1 deletion composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

37 changes: 37 additions & 0 deletions config/sources.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# The data sets this application publishes, one entry per source key.
#
# Field names follow DCAT-AP, the metadata profile European data portals
# harvest, so registering a data set is a translation of its entry rather than
# a new survey. See docs/adr/007-source-manifest.md.

sources:
mtm_spatialmaps-handicap-parking:
title: 'Handicapparkering, Aarhus Kommune'
description: >-
Disabled parking bays in Aarhus Municipality, with the number of
reserved bays per location.
publisher: 'Aarhus Kommune'
contact: ppg@aarhus.dk
landing_page: 'https://www.opendata.dk/city-of-aarhus/parkering-i-aarhus-kommune'
access_url: 'https://webkort.aarhuskommune.dk/spatialmap?page=get_geojson_opendata&datasource=invap'
media_type: application/geo+json
crs: 'EPSG:25832'
model: OnStreetParking
context_url: 'https://raw.githubusercontent.com/smart-data-models/dataModel.Parking/master/context.jsonld'
update_frequency: continuous

# The portal states no licence for this data set. DCAT-AP requires
# one, so it has to be settled with the data owner before the
# catalogue can be registered anywhere.
licence: ~

# Fields the feed carries that are not published. Recorded here
# because the source class shows what is mapped but cannot show what
# was left out, or why.
omitted_fields:
ident: 'Single-letter code; its meaning is not documented and not confirmed by the data owner.'
oprettet_af: 'Directory username of the municipal employee who created the record.'
rettet_af: 'Directory username of the municipal employee who last edited the record.'
oprettet_dato: 'Describes the register record.'
rettet_dato: 'Describes the register record.'
mi_style: 'MapInfo rendering style, empty throughout the export.'
74 changes: 74 additions & 0 deletions docs/adr/001-architecture-symfony-docker.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we need ADRs we should write them ourselves. Much of the text makes no sense to me.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that they could be even shorter and more concise. I think that some of the considerations behind the decisions makes a stronger argument towards the decided architecture. People in the future may use this project as inspiration for working with brokers, so i think that the proper documentation earns its place.

Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# 001: Architecture — Symfony 8 on the ITK Dev Docker template

| Field | Value |
|--------------------|----------------------------------------|
| **Created By** | Jeppe Krogh |
| **Date** | 2026-08-24 |
| **Decision Maker** | ITK Dev team |
| **Stakeholders** | ITK Dev developers, future maintainers |
| **Status** | Draft |

## Context

This application reads open data sets, converts them to a standard smart-city
representation, and publishes them to a context broker. The organisation
maintains its PHP services on a versioned Docker template carrying shared CI
and coding-standards configuration. This ADR serves to decide the runtime,
framework and development environment the application is built on.

### Drivers

- **Functional:** scheduled console commands, outbound HTTP, and a local broker
to import into. No database and no HTTP surface of its own.
- **Non-functional:** shared tooling rather than reimplemented tooling, minimal
onboarding, reproducible across developers and CI, long-term vendor support.

### Options Considered

1. **PHP 8.4 / Symfony 8 on the maintained template.** CI, coding standards and
task runner come for free, and its console suits scheduled imports; it
provisions services this application never uses, and its PHP runs ahead of
developer hosts.
2. **A minimal project on the host, without the template.** No unused services
and no container requirement, but shared configuration is rebuilt by hand
and a local broker needs containers anyway, moving the requirement rather
than removing it.
3. **A second entry point in an existing internal application.** One deployment
to operate, but couples a batch importer to a user-facing release cycle and
inherits dependencies it has no use for.
4. **A different language ecosystem on a bespoke setup.** Richer geospatial
libraries, but no internal expertise and no shared tooling; the needed
transformations exist as mature libraries in the established stack.

## Decision

**PHP 8.4 + Symfony 8** on the ITK Dev Docker template, as its **own deployable
service**, with a containerised broker overlay for local development.

- Standardising costs less over the application's lifetime than trimming unused
services: a second toolchain must be learned and patched; idle containers
cost only disk.
- A batch importer's lifecycle and failure modes differ from a user-facing
application's, so it stays its own service.
- No domain persistence is needed — the broker is the system of record — so the
template's database is left unused rather than removed, keeping template
updates a clean diff.
- Local development includes a real broker, so imports are verified end to end
rather than only as serialised output.

## Consequences

### Positive

- Onboarding cost close to zero; CI and coding standards work from the first
commit.
- No schema, no migrations, no state to keep consistent with the broker.

### Negative / Trade-offs

- Containers are mandatory; dependency management, console commands and tests
cannot run natively.
- A web server, database and mail catcher are provisioned and never used.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove any unused services.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Removed this statement.

- Broker images are not published for every CPU architecture, so local start-up
may be slow under emulation.
- The application follows the template's choices; deviating later has a cost.
Loading