chore: feature integration branch for the ENVITED-X pipeline - #14
Draft
jdsika wants to merge 14 commits into
Draft
chore: feature integration branch for the ENVITED-X pipeline#14jdsika wants to merge 14 commits into
jdsika wants to merge 14 commits into
Conversation
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
9 times, most recently
from
May 12, 2026 16:35
a705c35 to
3d3a52a
Compare
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
3 times, most recently
from
June 10, 2026 19:42
cab84ad to
b2b3dba
Compare
rmessaou
force-pushed
the
feat/envited-x-pipeline
branch
from
July 8, 2026 09:02
97e73d0 to
ef7ad85
Compare
3 tasks
RDFC-1.0 canonicalization already makes RDF output deterministic:
isomorphic graphs always serialize identically. It does not make output
diffable. Blank nodes are numbered `c14nN` in a single global order, so
inserting one class can renumber every blank node after it and rewrite
most of the file. A one-line semantic change lands as a whole-file diff,
which makes generated OWL/SHACL hard to review and noisy to keep under
version control.
Add a `diff_stable` argument to `canonicalize_rdf_graph()` and a
`--diff-stable/--no-diff-stable` flag to the four RDF generators. When
enabled, blank-node labels are derived from each node's own neighbourhood
via Weisfeiler-Lehman refinement, so an edit relabels only the blank
nodes it actually touches.
Measured churn on a real schema (add one class, count changed lines):
generator default --diff-stable
owlgen 2091 17
shexgen 796 50
shaclgen 291 13
rdfgen 115 25
Output stays deterministic and isomorphic either way; only the choice of
label changes. Off by default, because enabling it relabels existing
output.
The refinement itself lives in `diffable-rdf`, whose only dependencies
(rdflib, pyoxigraph) are already linkml-runtime dependencies at higher
versions, so this adds no new transitive dependencies.
…-opping Bump the floor to diffable-rdf 0.3.0 and add the missing uv.lock entry: the dependency was declared in pyproject.toml but never locked, so "uv lock --check" and the "uv sync --frozen" anti-malware gate would both have failed CI. 0.3.0 also fixes two defects in the Weisfeiler-Lehman labelling this feature relies on. Disconnected blank-node components now converge independently, so an edit in one region no longer relabels an unrelated one. And the suffix used to tell structurally indistinguishable nodes apart was assigned in c14nN *text* order, so c14n10 sorted between c14n1 and c14n2 -- adding a tenth tied blank node relabelled eight of the nine already there, the exact opposite of what this labelling is for. Separately, diff_stable=True was silently ignored whenever pyoxigraph refused the graph and canonicalize_rdf_graph degraded to rdflib. Weisfeiler-Lehman refinement consumes canonical pyoxigraph quads, and that path exists precisely because there are none, so the argument could not be honoured -- but the caller was never told. "shaclgen --include-annotations --diff-stable" reaches it, via the literal predicate an annotation tag without a ':' produces, and returned output byte-identical to --no-diff-stable. It now warns, with a regression test asserting the warning and the byte-identical output that makes silence misleading.
0.4.0 carries graph.base through the library's rdflib fallback, verifying that every absolute IRI of the source survives a re-read rather than dropping the directive outright, and adds a diff_stable parameter to canonicalize_rdf_graph. The lock entry is written by hand because the workspace sets exclude-newer = "7 days", which filters any release younger than that from resolution; 0.3.0 was pinned the same way for the same reason, and both become resolvable normally on 2026-09-18. uv lock --check and uv sync --all-groups both accept the entry. https://github.com/ASCS-eV/diffable-rdf/releases/tag/v0.4.0
Asserts each correctness property against both linkml's copy and diffable_rdf, marking whichever implementation does not hold it as a strict xfail, so the file is a ratchet in both directions. Nine gaps run one way, two of them silent data corruption. One ran the other way -- the library dropped @base on the degraded path -- and that was the last property blocking delegation. diffable-rdf 0.4.0 fixed it, so that case now passes on both sides and carries no mark.
The implementation was extracted into diffable-rdf at the maintainers' request in linkml#3295, but linkml kept its own copy and the two drifted. This deletes the copy and calls the library, which is what the extraction was for. Nine correctness fixes come with it, each already asserted in test_rdf_canonicalize_defects.py and each previously a strict xfail on the linkml side: - a base ending in # no longer rewrites every IRI that merely shares its prefix (silent corruption: output parsed, meaning changed) - a shared rdf:List tail is no longer duplicated (9 triples in, 11 out) - N-Triples refuses a relative IRI instead of writing a file its own parser rejects - literals containing U+2028, U+2029, U+0085 and the other separators str.splitlines() treats as line breaks survive the line sort - degraded RDF/XML, degraded Turtle and json-ld are byte-identical across processes - every format ends with exactly one newline - a Dataset is refused rather than silently flattened Behaviour changes for callers: nt output for a graph containing a relative IRI now raises ValueError rather than writing an unparseable file, and json-ld is canonicalized rather than handed to rdflib, so it no longer warns. All four RDF generators produce byte-identical output. The library reports degradation through logging; linkml reports it through warnings so it is visible without logging configuration. _DegradedPathWarnings bridges the two, and the tests pin the properties that makes load-bearing: the warning is attributed to the caller's line, the library's logger is left as it was found, a caller who configured logging still receives the record, and warnings survive an exception.
The rules-to-SHACL-SPARQL converter added in linkml#3451 recognised a single named pattern. This adds the presence-implies-value pattern: a precondition asserting `value_presence: PRESENT` on one slot, and a postcondition constraining another slot with `equals_string` or `equals_string_in`. It reads as "if the guard slot is present, the target slot must be present and hold one of the allowed values", and generalises the existing boolean guard to arbitrary enum values. The boolean guard is now gated on the target slot's range actually being `boolean`. Without that gate a slot of range `string` carrying `equals_string: "true"` was translated as a boolean comparison, which does not match the string `"true"` in the data and so flagged conforming instances as violations. String-ranged slots now fall through to the presence-implies-value pattern and compare as strings. Pattern matching is exact: each converter requires its conditions to set precisely the operators it translates. A rule whose conditions carry anything further -- extra scalar operators, or expression-level any_of / all_of / none_of / exactly_one_of -- is skipped rather than partially translated, since dropping a term would either widen the precondition (false positives) or weaken the postcondition (false negatives). Slot resolution goes through induced slots so that `slot_usage` overrides, `slot_uri` overrides and alias-form keys resolve to the same IRI that `sh:path` emits. Co-authored-by: jdsika <carlo.van-driesten@vdl.digital>
…ersion The named-pattern converters only recognise whole-rule shapes, so a rule one operator away from a known pattern produced no constraint at all. This adds a compositional fallback, tried only after every named pattern has declined, that builds the query from the operators present rather than from a fixed template. Preconditions become a conjunction of graph patterns and FILTERs; the single postcondition becomes its negation. Together they select focus nodes satisfying every precondition while violating the postcondition, which is exactly the SHACL-SPARQL violation contract of SHACL 5.3.1, with $this pre-bound to the focus node. Operator support is declared per operator, and the builder returns None -- skipping the rule -- as soon as it meets one it does not handle, so an unsupported combination is never partially translated. This covers conditional-required and conditional-absent postconditions, numeric threshold preconditions, nested-object preconditions and has_member list membership, in any combination the operators allow. Numeric bounds are validated before interpolation. minimum_value and maximum_value have metamodel range Anything, so YAML strings, dates and .nan / .inf reach the generator unchanged; interpolating them raw either produced unparsable SPARQL that poisons the whole shapes graph at validation time, or -- for a date such as 2020-01-01 -- parsed as an arithmetic expression that silently never fires. Only int and finite float are rendered; anything else skips the rule. String literals are escaped rather than interpolated, and a slot carrying both minimum_value and maximum_value now yields both bounds instead of only the first. Nested and member slots resolve against the range class of their container, so an inner slot is no longer shadowed by a same-named slot on the outer class, and a range narrowed through slot_usage resolves its enum permissible values from the narrowed range. Co-authored-by: jdsika <carlo.van-driesten@vdl.digital>
The SHACL generator translates LinkML rules into sh:sparql constraints, but the generator documentation did not mention it, so the feature was undiscoverable and its limits undocumented. Describe the recognised named patterns and the compositional fallback, the --emit-rules flag, the skip-never-mis-translate contract and which rule attributes warn, with a worked YAML-to-Turtle example. Note that SPARQL-based constraints need a processor with SHACL-SPARQL support.
The SHACL generator translated any_of branches by dispatching
solely on `any.range` (class, type, enum, or simple datatype).
If a branch specified `pattern:` — either alone or combined
with a range — the constraint was silently dropped, producing
an empty blank node `[ ]` (trivially satisfied) instead of the
intended `[ sh:pattern "..." ]`.
This is a problem for schemas that use pattern alternatives in
`any_of`, such as the SPDX license field where valid values are
either members of a fixed enum (SPDX identifiers), IRIs, or
custom identifiers matching the LicenseRef- pattern defined in
SPDX Specification v2.3 Annex D (ABNF: license-ref =
["DocumentRef-"(idstring)":"]"LicenseRef-"(idstring)).
The fix adds a single check after the range dispatch:
if any.pattern:
g.add((range_list[-1], SH.pattern, Literal(any.pattern)))
This correctly handles:
- Pattern-only branches (no range): node gets only sh:pattern
- Range + pattern branches: node gets both sh:datatype and sh:pattern
- Range-only branches (no pattern): unchanged behaviour
The test suite now includes a dedicated schema exercising all
three cases, with assertions on both the generated RDF triples
and pyshacl validation of conforming/non-conforming data.
Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
…nstraints For an inlined-as-dict slot whose range class has an identifier/key slot, render the key slot's string-applicable constraints onto JSON Schema propertyNames (draft-06+) instead of dropping them. In the inlined-dict form the mapping key is the identifier value, so the key slot's constraints constrain the keys. JSON object keys are always strings, so only pattern, enum (equals_string_in) and a string const (equals_string) are emitted; numeric minimum/maximum, numeric const (equals_number) and allOf are excluded -- a numeric const would otherwise reject every key. structured_pattern is honored when materialize_patterns is enabled, consistent with value patterns. Backward compatible: emitted only when a string-applicable key constraint applies. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
…key constraints Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
… names Add an opt-in --normalize-prefixes flag to OWL, SHACL, and JSON-LD Context generators that normalises non-standard prefix aliases to well-known names from a static prefix map (derived from rdflib 7.x defaults, cross-checked against prefix.cc consensus). Key design decisions: - Static frozen map (MappingProxyType) instead of runtime Graph().namespaces() lookup eliminates rdflib version dependency - Both http://schema.org/ and https://schema.org/ map to 'schema' - Shared normalize_graph_prefixes() helper used by OWL and SHACL - Two-phase graph normalisation: Phase 1 normalises schema-declared prefixes, Phase 2 cleans up runtime-injected bindings - Collision detection: skip with warning when standard prefix name is already user-declared for a different namespace - Phase 2 guard prevents overwriting HTTPS bindings with HTTP variants The flag defaults to off, preserving existing behaviour. Tests cover OWL, SHACL, and context generators with sdo->schema, dce->dc, http/https edge case, custom prefix preservation, flag-off backward compatibility, cross-generator consistency, prefix collision detection, schema1 regression prevention, Phase 2 HTTPS guard, empty schema edge case, and static map integrity. Signed-off-by: jdsika <carlo.van-driesten@bmw.de> Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
from
September 11, 2026 14:23
6aa7702 to
233adc3
Compare
jdsika
changed the title
gen-owl canonicalizes its output with RDFC-1.0 before serializing, and the determinism work adds a --diff-stable option on top of it, but neither is mentioned anywhere in the generator documentation. Describe what is guaranteed without passing any flag, why RDFC-1.0's sequential blank-node numbering can still produce noisy diffs across schema edits, and what --diff-stable changes. Also record the behaviour users meet in practice but cannot discover from --help: that the same option exists on gen-rdf, gen-shacl and gen-shex, and that graphs which are not standard RDF -- literal predicates from SHACL annotation mode, relative IRIs such as the metamodel's bibo:status <testing> -- take an rdflib fallback that stays reproducible across processes, warns, and deliberately does not honour --diff-stable.
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
from
September 11, 2026 14:44
233adc3 to
b8a388e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Feature integration branch. This branch is consumed directly by downstream
ENVITED-X projects, so it carries every feature developed in this fork, applied
on top of current upstream
main. It is not an upstream submission — eachfeature is submitted individually (see the table below).
Rebuilt from scratch on upstream
main(0e401cef2). The previous state was221 commits behind and still carried three commits that have since been merged
upstream.
Contents
Fourteen commits, in dependency order, starting with the determinism work.
diffable-rdf0.4.0, conformance evidence, delegation to the librarysh:patternfor pattern constraints insideany_ofpropertyNamesfrom inlined-dict key constraints, + docs--normalize-prefixesfor well-known prefix names, + docsdocs(owl): document deterministic serialization and--diff-stableAlready merged upstream — dropped from this branch
feat(gen-shacl): generate sh:sparql constraints from LinkML rules→ feat(gen-shacl): generate sh:sparql constraints from LinkML rules linkml/linkml#3451fix(shaclgen): emit sh:minCount/maxCount 0 for zero cardinality values→ fix(shaclgen): emit sh:minCount/maxCount 0 for zero cardinality values linkml/linkml#3473feat(jsonschemagen): add --include-null/--no-include-null→ mergeddocs(json-schema): document --include-null→ cherry-pick was empty, already inmainQuality gates
Every commit is signed and carries its original authorship (the two SHACL rule
features remain authored by Rayene Messaoud).
ruff checkandruff format --checkclean on all 15 changed Python filespytest tests/linkml/test_generators tests/linkml_runtime— 3674 passed,375 skipped, 5 xfailed, 0 failed
Known caveat
The
--normalize-prefixescommits pinprefixmapsto a git revision, becausethe fix they need (linkml/prefixmaps#82) is merged but unreleased — PyPI is
still on
0.2.6. This is fine for consumers installing from git, but it breaksthe Docker image build, since
pip installof the built wheel has to fetch thedependency from GitHub and the base image has no
git. A release request is withthe prefixmaps maintainer. If a clean Docker build is needed before then, drop
the last three commits.