refactor(rdf): delegate canonicalization to the diffable-rdf library - #26
Open
jdsika wants to merge 5 commits into
Open
refactor(rdf): delegate canonicalization to the diffable-rdf library#26jdsika wants to merge 5 commits into
jdsika wants to merge 5 commits into
Conversation
RDFC-1.0 canonicalization already makes RDF output deterministic:
isomorphic graphs always serialize identically. It does not make output
diffable. Blank nodes are numbered `c14nN` in a single global order, so
inserting one class can renumber every blank node after it and rewrite
most of the file. A one-line semantic change lands as a whole-file diff,
which makes generated OWL/SHACL hard to review and noisy to keep under
version control.
Add a `diff_stable` argument to `canonicalize_rdf_graph()` and a
`--diff-stable/--no-diff-stable` flag to the four RDF generators. When
enabled, blank-node labels are derived from each node's own neighbourhood
via Weisfeiler-Lehman refinement, so an edit relabels only the blank
nodes it actually touches.
Measured churn on a real schema (add one class, count changed lines):
generator default --diff-stable
owlgen 2091 17
shexgen 796 50
shaclgen 291 13
rdfgen 115 25
Output stays deterministic and isomorphic either way; only the choice of
label changes. Off by default, because enabling it relabels existing
output.
The refinement itself lives in `diffable-rdf`, whose only dependencies
(rdflib, pyoxigraph) are already linkml-runtime dependencies at higher
versions, so this adds no new transitive dependencies.
…-opping Bump the floor to diffable-rdf 0.3.0 and add the missing uv.lock entry: the dependency was declared in pyproject.toml but never locked, so "uv lock --check" and the "uv sync --frozen" anti-malware gate would both have failed CI. 0.3.0 also fixes two defects in the Weisfeiler-Lehman labelling this feature relies on. Disconnected blank-node components now converge independently, so an edit in one region no longer relabels an unrelated one. And the suffix used to tell structurally indistinguishable nodes apart was assigned in c14nN *text* order, so c14n10 sorted between c14n1 and c14n2 -- adding a tenth tied blank node relabelled eight of the nine already there, the exact opposite of what this labelling is for. Separately, diff_stable=True was silently ignored whenever pyoxigraph refused the graph and canonicalize_rdf_graph degraded to rdflib. Weisfeiler-Lehman refinement consumes canonical pyoxigraph quads, and that path exists precisely because there are none, so the argument could not be honoured -- but the caller was never told. "shaclgen --include-annotations --diff-stable" reaches it, via the literal predicate an annotation tag without a ':' produces, and returned output byte-identical to --no-diff-stable. It now warns, with a regression test asserting the warning and the byte-identical output that makes silence misleading.
0.4.0 carries graph.base through the library's rdflib fallback, verifying that every absolute IRI of the source survives a re-read rather than dropping the directive outright, and adds a diff_stable parameter to canonicalize_rdf_graph. The lock entry is written by hand because the workspace sets exclude-newer = "7 days", which filters any release younger than that from resolution; 0.3.0 was pinned the same way for the same reason, and both become resolvable normally on 2026-09-18. uv lock --check and uv sync --all-groups both accept the entry. https://github.com/ASCS-eV/diffable-rdf/releases/tag/v0.4.0
Asserts each correctness property against both linkml's copy and diffable_rdf, marking whichever implementation does not hold it as a strict xfail, so the file is a ratchet in both directions. Nine gaps run one way, two of them silent data corruption. One ran the other way -- the library dropped @base on the degraded path -- and that was the last property blocking delegation. diffable-rdf 0.4.0 fixed it, so that case now passes on both sides and carries no mark.
The implementation was extracted into diffable-rdf at the maintainers' request in linkml#3295, but linkml kept its own copy and the two drifted. This deletes the copy and calls the library, which is what the extraction was for. Nine correctness fixes come with it, each already asserted in test_rdf_canonicalize_defects.py and each previously a strict xfail on the linkml side: - a base ending in # no longer rewrites every IRI that merely shares its prefix (silent corruption: output parsed, meaning changed) - a shared rdf:List tail is no longer duplicated (9 triples in, 11 out) - N-Triples refuses a relative IRI instead of writing a file its own parser rejects - literals containing U+2028, U+2029, U+0085 and the other separators str.splitlines() treats as line breaks survive the line sort - degraded RDF/XML, degraded Turtle and json-ld are byte-identical across processes - every format ends with exactly one newline - a Dataset is refused rather than silently flattened Behaviour changes for callers: nt output for a graph containing a relative IRI now raises ValueError rather than writing an unparseable file, and json-ld is canonicalized rather than handed to rdflib, so it no longer warns. All four RDF generators produce byte-identical output. The library reports degradation through logging; linkml reports it through warnings so it is visible without logging configuration. _DegradedPathWarnings bridges the two, and the tests pin the properties that makes load-bearing: the warning is attributed to the caller's line, the library's logger is left as it was found, a caller who configured logging still receives the record, and warnings survive an exception.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delegate RDF canonicalization to
diffable-rdfStacked on #25 (conformance evidence), which is stacked on #24 (
--diff-stable). Review that order.What this does
Deletes
linkml_runtime/utils/rdf_canonicalize.py's implementation and callsdiffable-rdfinstead.canonicalize_rdf_graphkeeps its signature; nothing that imports it changes. Net −196 lines.This is the step linkml#3295 asked for. The module was extracted into a standalone library at @cmungall's request — "the implementation should live elsewhere... independent rdflib sidecar library?" and "This should be implemented somewhere upstream of linkml, after which we'll happily consider a PR" — but linkml kept its copy, and the two drifted. #25 measured the drift. This closes it.
Why now
#25 found ten correctness properties where the two copies disagree. Nine held only in the library. One held only in linkml — the library dropped
@baseon the degraded path — and that one was the blocker: adopting the library would have been a regression forrdflib_dumper.dumps(..., prefix_map={"@base": ...}), which the metamodel exercises.That is fixed in diffable-rdf 0.4.0 (ASCS-eV/diffable-rdf#60), pinned by #24. With no gap running toward linkml, delegation is a pure improvement.
The fix is worth a sentence because the naive version is wrong. rdflib's
relativize()is a string prefix strip, not RFC 3986 §5.2.2 component resolution, so carrying a base ofhttp://ex.org/vocab#rewriteshttp://ex.org/vocab#Thingand every unrelated IRI that merely starts with those characters. Rejecting bases by shape does not work either —http://ex.org/a/bandhttp://ex.org/d?q=1corrupt too. The library carries the base, re-parses, and keeps it only if every absolute IRI of the source survives. RFC 3986 specifies resolution and never relativization, so verification is the only sound test.What linkml gains
Each was a strict xfail on the linkml side in #25 and is now unmarked:
#doesn't rewrite unrelated IRIsrdf:Listtail isn't duplicatedntrefuses a relative IRIU+2028/U+2029/U+0085/U+000B/U+000C/U+001C–1Ein literals survivestr.splitlines()split on them mid-literaljson-ldare byte-identical across processesns1/ns2names movedDatasetis refused, not flattenedDatasetis aGraphsubclass, so named graphs were silently mergedBehaviour changes
Two, both deliberate:
ntfor a graph containing a relative IRI now raisesValueError. N-Triples 1.1 §2.2 permits only absolute IRIs. The old code emitted the file anyway, and rdflib's own parser rejected it. There is no valid document to produce, so refusing is the only honest answer. This is the one change that could break a caller, and it breaks them at write time instead of at read time.json-ldis canonicalized rather than handed to rdflib, so it no longer emits a fallback warning.All four RDF generators (
gen-owl,gen-rdf,gen-shacl,gen-shex) produce byte-identical output. 1736 generator tests and 1855 runtime tests pass unchanged.The one piece of real code left
linkml reports degradation with
warnings.warnso a schema author runninggen-owlsees it without configuring logging first. The library useslogging, which is silent by default. A naive delegation loses the signal._DegradedPathWarningsbridges them, and four tests pin the properties that makes load-bearing:emit— the distance fromwarnings.warnto the caller is otherwise eightloggingframes plus a library depth that differs per message, so no fixedstacklevelworks);Review notes
test_rdf_canonicalize_defects.pystill parametrizes over both entry points with every case unmarked. That is the evidence this changed nothing it should not have, and it keeps a future re-fork from going unnoticed.test_rdf_canonicalize.pychanged target rather than assertion. Two monkeypatchedrdf_canon_mod.ox, which was always reallypyoxigraph— they now patchoxdirectly and no longer depend on module internals. The third asserted an unrelatedValueErrorpropagates using"Invalid base IRI"as its example; the library correctly treats that as a base problem and recovers from it, so the example is now a genuinely unrelated error and the recovery got its own test.References
Extraction request: linkml#3295. Determinism and diff-stability lineage: linkml#3702, linkml#3481, linkml#3212, linkml#696, linkml#3516, linkml#3721. Merged: linkml#3407, linkml#3696, linkml#3518, linkml#3524, linkml#3703.