Dynamic Entities — Usage

Setup

• Python: 3.8+ recommended. Install dependencies:

pip install -r requirements.txt

• Environment: create a .env file or export the following environment variables used by obp_client.py:
  • OBP_USERNAME: OBP username
  • OBP_PASSWORD: OBP password
  • OBP_CONSUMER_KEY: consumer key for DirectLogin
  • OBP_HOSTNAME: (optional) OBP base URL; defaults in obp_client.py

Files

• parse_minimum_fields.py: Parse the minimal field matrix Excel (min_field_matrix.xlsx by default) and optionally create dynamic entities on OBP.
• main.py: High-level management script that deletes objects, deletes matching dynamic entities, then recreates entities defined in dynamic_entities.py.

parse_minimum_fields.py — Usage

This creates the entities from the minimal field matrix Excel file, exported from the Google Sheet template.

• Run locally (print parsed entities):

python3 parse_minimum_fields.py [path/to/min_field_matrix.xlsx]

• Create dynamic entities on OBP:

python3 parse_minimum_fields.py [path/to/min_field_matrix.xlsx] --create

• Update existing dynamic entities on OBP:

python3 parse_minimum_fields.py [path/to/min_field_matrix.xlsx] --update

--update looks up each existing dynamic entity by name and updates its definition in place (entities not found on OBP are skipped). Use --create for fresh entities and --update to modify ones that already exist.

• Options:
  • file (positional): Path to the Excel file. Defaults to min_field_matrix.xlsx.
  • --create: If set, the script will POST created entity definitions to the OBP management API.
  • --update: If set, update existing dynamic entities (matched by name) instead of creating new ones.
  • --token: DirectLogin token to use (overrides token from obp_client.py).
  • --host: OBP host/base URL to use (overrides OBP_HOSTNAME).
  • --yes: When used with --create, skip interactive confirmation prompt.

Note: --create only creates — it does not delete existing entities or objects first. To do a clean wipe-and-recreate, use main.py (which deletes objects and entity definitions before recreating), but note that main.py rebuilds from the hardcoded entities in dynamic_entities.py, not from a spreadsheet.

Notes about parsing behavior:

• Column A is used for field names and entity: rows start new entities.
• Column D is preserved as the value (type) in the parsed attribute dict.
• Column F is used as the description for attributes (and the entity-level description on entity: rows).
• Column G is used as the example value for attributes when present.
• Field names are sanitized: dots and other disallowed characters are replaced by underscore (_), repeated underscores are collapsed, and leading/trailing underscores are removed.
• Example strings from column G have surrounding single or double quotes stripped.

Re-create the entities (delete then create from the spreadsheet)

A clean wipe-and-recreate driven entirely by the spreadsheet (this is what main.py does not do — main.py is tied to the hardcoded list in dynamic_entities.py):

1. Parse and save the entity list to entities_output.txt:

python3 parse_minimum_fields.py min_field_matrix.xlsx --save   # non-interactive
# or run without --save and answer "y" at the prompt (default filename entities_output.txt)

2. Delete those entities and all their records on OBP with delete_ogcr_entities.py:

python3 delete_ogcr_entities.py            # reads entities_output.txt by default; prompts for confirmation
python3 delete_ogcr_entities.py --yes      # skip the confirmation prompt

3. Re-create the entities from the spreadsheet:

python3 parse_minimum_fields.py min_field_matrix.xlsx --create --yes

Notes:

• delete_ogcr_entities.py deletes exactly the entities listed in entities_output.txt (one per Entity: line) and leaves all other dynamic entities on the instance untouched. If an entity was renamed in the spreadsheet, the old name is not in the file and will be left on OBP as an orphan — delete it separately. To wipe every dynamic entity instead, use delete_all_dynamic_entities.py.
• Deletion runs in repeated passes so reference (foreign-key) constraints between entities don't block a clean delete, and it exits non-zero if anything it was asked to delete survives.
• Always regenerate entities_output.txt (step 1) after editing the spreadsheet, so the delete list matches what you are about to create.
• delete_ogcr_entities.py options: file (positional, default entities_output.txt), --yes (skip confirmation), --token (override the DirectLogin token).

Create dummy data

create_dummy_data.py creates one sample object per entity, driven by the same spreadsheet. Run it after the entities exist on OBP (see "Re-create the entities" above):

python3 create_dummy_data.py [path/to/min_field_matrix.xlsx] [--token TOKEN]

• file (positional): spreadsheet path. Defaults to min_field_matrix.xlsx.
• --token: DirectLogin token (overrides the token from obp_client.py).
• --no-log: Do not write the audit trail to the <prefix>ogcr_dynamicentities_log dynamic entity.

How it works:

• Values come from the spreadsheet — each field is populated from its column G example value, coerced to the field's declared type (string, integer, number, boolean, json, DATE_WITH_DAY).

• Foreign keys are made valid — any <entity>_id field is overwritten with the real id of the referenced object, so the dummy data is referentially consistent (e.g. activity.operator_id points at the created operator, and audit_report links to the operator, activity, scheme, body, plans and certificate).

• Entities that own an <entity>_id field get a canonical id taken from the spreadsheet example; the verification/report entities without one receive an OBP-generated UUID.

• The field compliance_certificate_id (which does not follow the <entity>_id convention) is mapped to certificate_of_compliance via an explicit alias in the script (FK_ALIASES).

• The run is audited in OBP — unless --no-log is given, the script ensures a system dynamic entity named after this application, <prefix>ogcr_dynamicentities_log (the optional OBP_ENTITY_PREFIX, e.g. ogcr_dynamicentities_log with no prefix or ogcr3_ogcr_dynamicentities_log with OBP_ENTITY_PREFIX=ogcr3), exists (defined in ogcr_log_entity.py) and writes one record to it per object: entity_created for each created object and entity_failed for each failed create (with the OBP error text). Each record carries entity_name, entity_id, status, message, a UTC timestamp, and a json references list describing every reference:<x> field on the entity and how it resolved — each item has field, target, resolution (resolved = a real created id was used; fallback = the spreadsheet example value was used because the target is a static OBP entity or one we don't create here) and the value posted. Logging is best-effort: if the log entity cannot be created or a record fails to POST, the data creation continues uninterrupted.

Notes:

• It creates one record per entity. To create more (e.g. several parcels under one activity), extend the payload loop in main().
• It is fully spreadsheet-driven — it does not use the hardcoded entities in dynamic_entities.py.

Public read access (EntityHasPublicAccess)

An entity can be opened for unauthenticated read — useful for open reference data such as the country list. Tick the EntityHasPublicAccess column (column Q) on the entity's Entity: <name> row in the spreadsheet; leave the field rows blank, because the flag is entity-level, not per field. TRUE, 1, Y or a checkmark all count; blank or FALSE means not public.

The parser finds the column by header text, not position, so it can be moved and minor spelling differences (hasPublicAccess, Entity Has Public Access) still match. If the column is missing entirely — an older export — every entity simply defaults to not public.

A ticked entity is created with "hasPublicAccess": true, which gives it an extra route:

Route	Who	What
GET /obp/dynamic-entity/public/<entity>	anyone, no login	read only, shared-pool rows only
GET /obp/dynamic-entity/<entity>	role holders	unchanged; 401 without authentication
any write to /public/	—	404; there is no public write route

The flag is only sent when it is switched on, so an OBP build that predates it is unaffected. See the Dynamic-Entity-Access-Model glossary entry on your OBP instance for the full access model (hasPersonalEntity, hasCommunityAccess, useRowLevelAccess, authMode and field-level roles); this flag is its "curated reference data" pattern — role holders maintain the data, everyone reads it.

Currently ticked: country, technologies_practices_processes.

Fixtures (controlled vocabularies)

Some entities are not examples but fixed lists of values the rest of the system selects from. These live in fixtures.py, not in the spreadsheet, and create_dummy_data.py writes the whole list instead of a single example row — so every run of recreate_ogcr_entities.sh ends with exactly those rows present.

Currently fixtured:

• technologies_practices_processes — the 28 technologies/practices/processes an activity can declare. Ids are UPPERCASE_WITH_UNDERSCORES and the label is derived from the id in proper case, with acronyms in fixtures.ACRONYMS left uppercase (GEOLOGICAL_CO2_STORAGE → Geological CO2 Storage, ..._BECCS → ... BECCS).
• country — all 249 ISO 3166-1 alpha-2 codes, in iso_3166_1_countries.py. The id is the two-letter code (DE) and the label is the ISO English short name (Germany). A handful read formally (Korea, Republic of, Taiwan, Province of China); each carries a # commonly: comment if you prefer the common name. The module header has the one-liner that regenerates it from the system iso-codes package.

How a fixture is written:

• The id goes in <entity>_id. OBP preserves a supplied <entity>_id, so these codes are the stable keys other records reference.
• The label goes in the entity's name field, resolved per entity by fixtures.resolve_name_field: name, else <entity>_name, else the sheet's only other *_name field (this is how technologies_practices_processes.practice_name is found). If the sheet has several *_name fields the choice is ambiguous, so ids are written without a label and a warning is logged.
• Any other field the sheet declares for that entity keeps its spreadsheet example and declared type.
• Rows already stored are skipped, so the script is safe to re-run against a populated instance.
• Stored rows that are not in the fixture list are logged as a warning and left in place (a full recreate wipes them anyway).

Topping up an existing instance:

python3 create_dummy_data.py --fixtures-only     # writes only the fixtured entities

--fixtures-only skips every non-fixtured entity, so it will not duplicate (or error on) the single example rows those already have. Combined with the skip-if-present behaviour, it is the way to add newly defined fixture values, or to retry rows that failed, without a full wipe-and-recreate.

Id length: OBP stores <entity>_id as varchar(36), so a fixture id longer than 36 characters is rejected with OBP-50015 ... value too long for type character varying(36). Either shorten the id — the display name is independent, so use the (id, name) form to keep the full wording — or widen the column on the OBP side.

To add a value, add it to the list in fixtures.py. To fixture another entity, add an entity_name: [rows] pair to FIXTURES, where a row is either a bare id (label derived) or an explicit (id, label) pair.

main.py — Usage

• Run the management workflow (delete objects, delete entity definitions, recreate entities):

python3 main.py

• main.py uses credentials and host configured in environment (via obp_client.py). It does not accept CLI args; set the environment first.

Tips & Validation

• The parser attempts to coerce example values to appropriate types (integer, number, boolean, array/object via JSON) before sending to OBP so the example field matches OBP validation expectations.
• If you run with --create and receive a 400 validation error, inspect the printed parsed entities to find which property's example is mismatched.

Example workflow

1. Ensure env vars set (or a .env file present).
2. Inspect parsing output:

python3 parse_minimum_fields.py

3. Create entities on OBP (confirm with --yes or interactively):

python3 parse_minimum_fields.py --create --yes

4. Use main.py to clean and recreate system entities defined in dynamic_entities.py:

python3 main.py

Where to look for issues

• Parsed entities printed by parse_minimum_fields.py show the exact value and example used to build the dynamic entity schema.
• If a field example must be a number, ensure column H contains an unquoted numeric value (the parser will coerce when possible).

If you want me to add example .env content, a quick test script, or adjust any parsing detail, tell me which part to update next.

Funding

The OGCR Project has received funding from the European Union's Horizon Europe programme under grant agreement 101218854.