Skip to content

refactor(rdf): delegate canonicalization to the diffable-rdf library - #26

Open
jdsika wants to merge 5 commits into
mainfrom
refactor/delegate-canonicalizer
Open

refactor(rdf): delegate canonicalization to the diffable-rdf library#26
jdsika wants to merge 5 commits into
mainfrom
refactor/delegate-canonicalizer

Conversation

@jdsika

@jdsika jdsika commented Sep 11, 2026

Copy link
Copy Markdown

Delegate RDF canonicalization to diffable-rdf

Stacked on #25 (conformance evidence), which is stacked on #24 (--diff-stable). Review that order.

What this does

Deletes linkml_runtime/utils/rdf_canonicalize.py's implementation and calls diffable-rdf instead. canonicalize_rdf_graph keeps its signature; nothing that imports it changes. Net −196 lines.

This is the step linkml#3295 asked for. The module was extracted into a standalone library at @cmungall's request — "the implementation should live elsewhere... independent rdflib sidecar library?" and "This should be implemented somewhere upstream of linkml, after which we'll happily consider a PR" — but linkml kept its copy, and the two drifted. #25 measured the drift. This closes it.

Why now

#25 found ten correctness properties where the two copies disagree. Nine held only in the library. One held only in linkml — the library dropped @base on the degraded path — and that one was the blocker: adopting the library would have been a regression for rdflib_dumper.dumps(..., prefix_map={"@base": ...}), which the metamodel exercises.

That is fixed in diffable-rdf 0.4.0 (ASCS-eV/diffable-rdf#60), pinned by #24. With no gap running toward linkml, delegation is a pure improvement.

The fix is worth a sentence because the naive version is wrong. rdflib's relativize() is a string prefix strip, not RFC 3986 §5.2.2 component resolution, so carrying a base of http://ex.org/vocab# rewrites http://ex.org/vocab#Thing and every unrelated IRI that merely starts with those characters. Rejecting bases by shape does not work either — http://ex.org/a/b and http://ex.org/d?q=1 corrupt too. The library carries the base, re-parses, and keeps it only if every absolute IRI of the source survives. RFC 3986 specifies resolution and never relativization, so verification is the only sound test.

What linkml gains

Each was a strict xfail on the linkml side in #25 and is now unmarked:

Property Was
1 A base ending in # doesn't rewrite unrelated IRIs silent corruption — parses, means something else
2 A shared rdf:List tail isn't duplicated silent corruption — 9 triples in, 11 out
3 nt refuses a relative IRI wrote a file its own parser rejects
4 U+2028/U+2029/U+0085/U+000B/U+000C/U+001C1E in literals survive str.splitlines() split on them mid-literal
5–7 Degraded RDF/XML, degraded Turtle, json-ld are byte-identical across processes traversal-ordered; ns1/ns2 names moved
8 Exactly one trailing newline, every format varied by serializer
9 A Dataset is refused, not flattened Dataset is a Graph subclass, so named graphs were silently merged

Behaviour changes

Two, both deliberate:

  • nt for a graph containing a relative IRI now raises ValueError. N-Triples 1.1 §2.2 permits only absolute IRIs. The old code emitted the file anyway, and rdflib's own parser rejected it. There is no valid document to produce, so refusing is the only honest answer. This is the one change that could break a caller, and it breaks them at write time instead of at read time.
  • json-ld is canonicalized rather than handed to rdflib, so it no longer emits a fallback warning.

All four RDF generators (gen-owl, gen-rdf, gen-shacl, gen-shex) produce byte-identical output. 1736 generator tests and 1855 runtime tests pass unchanged.

The one piece of real code left

linkml reports degradation with warnings.warn so a schema author running gen-owl sees it without configuring logging first. The library uses logging, which is silent by default. A naive delegation loses the signal.

_DegradedPathWarnings bridges them, and four tests pin the properties that makes load-bearing:

  • the warning is attributed to the caller's line, not to the adapter (this is why records are collected and re-emitted on exit rather than warned from emit — the distance from warnings.warn to the caller is otherwise eight logging frames plus a library depth that differs per message, so no fixed stacklevel works);
  • the library's logger is left as it was found, including when the adapter had to raise its level to see anything;
  • a caller who did configure logging still receives the record — the adapter adds a mechanism, it does not remove one;
  • warnings survive an exception, so the degradation that preceded a failure still reaches you.

Review notes

  • test_rdf_canonicalize_defects.py still parametrizes over both entry points with every case unmarked. That is the evidence this changed nothing it should not have, and it keeps a future re-fork from going unnoticed.
  • Three tests in test_rdf_canonicalize.py changed target rather than assertion. Two monkeypatched rdf_canon_mod.ox, which was always really pyoxigraph — they now patch ox directly and no longer depend on module internals. The third asserted an unrelated ValueError propagates using "Invalid base IRI" as its example; the library correctly treats that as a base problem and recovers from it, so the example is now a genuinely unrelated error and the recovery got its own test.

References

Extraction request: linkml#3295. Determinism and diff-stability lineage: linkml#3702, linkml#3481, linkml#3212, linkml#696, linkml#3516, linkml#3721. Merged: linkml#3407, linkml#3696, linkml#3518, linkml#3524, linkml#3703.

RDFC-1.0 canonicalization already makes RDF output deterministic:
isomorphic graphs always serialize identically. It does not make output
diffable. Blank nodes are numbered `c14nN` in a single global order, so
inserting one class can renumber every blank node after it and rewrite
most of the file. A one-line semantic change lands as a whole-file diff,
which makes generated OWL/SHACL hard to review and noisy to keep under
version control.

Add a `diff_stable` argument to `canonicalize_rdf_graph()` and a
`--diff-stable/--no-diff-stable` flag to the four RDF generators. When
enabled, blank-node labels are derived from each node's own neighbourhood
via Weisfeiler-Lehman refinement, so an edit relabels only the blank
nodes it actually touches.

Measured churn on a real schema (add one class, count changed lines):

    generator   default   --diff-stable
    owlgen         2091              17
    shexgen         796              50
    shaclgen        291              13
    rdfgen          115              25

Output stays deterministic and isomorphic either way; only the choice of
label changes. Off by default, because enabling it relabels existing
output.

The refinement itself lives in `diffable-rdf`, whose only dependencies
(rdflib, pyoxigraph) are already linkml-runtime dependencies at higher
versions, so this adds no new transitive dependencies.
…-opping

Bump the floor to diffable-rdf 0.3.0 and add the missing uv.lock entry: the
dependency was declared in pyproject.toml but never locked, so "uv lock --check"
and the "uv sync --frozen" anti-malware gate would both have failed CI.

0.3.0 also fixes two defects in the Weisfeiler-Lehman labelling this feature
relies on. Disconnected blank-node components now converge independently, so an
edit in one region no longer relabels an unrelated one. And the suffix used to
tell structurally indistinguishable nodes apart was assigned in c14nN *text*
order, so c14n10 sorted between c14n1 and c14n2 -- adding a tenth tied blank
node relabelled eight of the nine already there, the exact opposite of what this
labelling is for.

Separately, diff_stable=True was silently ignored whenever pyoxigraph refused
the graph and canonicalize_rdf_graph degraded to rdflib. Weisfeiler-Lehman
refinement consumes canonical pyoxigraph quads, and that path exists precisely
because there are none, so the argument could not be honoured -- but the caller
was never told. "shaclgen --include-annotations --diff-stable" reaches it, via
the literal predicate an annotation tag without a ':' produces, and returned
output byte-identical to --no-diff-stable. It now warns, with a regression test
asserting the warning and the byte-identical output that makes silence
misleading.
0.4.0 carries graph.base through the library's rdflib fallback, verifying
that every absolute IRI of the source survives a re-read rather than
dropping the directive outright, and adds a diff_stable parameter to
canonicalize_rdf_graph.

The lock entry is written by hand because the workspace sets
exclude-newer = "7 days", which filters any release younger than that from
resolution; 0.3.0 was pinned the same way for the same reason, and both
become resolvable normally on 2026-09-18. uv lock --check and
uv sync --all-groups both accept the entry.

https://github.com/ASCS-eV/diffable-rdf/releases/tag/v0.4.0
Asserts each correctness property against both linkml's copy and
diffable_rdf, marking whichever implementation does not hold it as a
strict xfail, so the file is a ratchet in both directions.

Nine gaps run one way, two of them silent data corruption. One ran the
other way -- the library dropped @base on the degraded path -- and that
was the last property blocking delegation. diffable-rdf 0.4.0 fixed it,
so that case now passes on both sides and carries no mark.
The implementation was extracted into diffable-rdf at the maintainers'
request in linkml#3295, but linkml kept its own copy and the two
drifted. This deletes the copy and calls the library, which is what the
extraction was for.

Nine correctness fixes come with it, each already asserted in
test_rdf_canonicalize_defects.py and each previously a strict xfail on
the linkml side:

- a base ending in # no longer rewrites every IRI that merely shares its
  prefix (silent corruption: output parsed, meaning changed)
- a shared rdf:List tail is no longer duplicated (9 triples in, 11 out)
- N-Triples refuses a relative IRI instead of writing a file its own
  parser rejects
- literals containing U+2028, U+2029, U+0085 and the other separators
  str.splitlines() treats as line breaks survive the line sort
- degraded RDF/XML, degraded Turtle and json-ld are byte-identical
  across processes
- every format ends with exactly one newline
- a Dataset is refused rather than silently flattened

Behaviour changes for callers: nt output for a graph containing a
relative IRI now raises ValueError rather than writing an unparseable
file, and json-ld is canonicalized rather than handed to rdflib, so it
no longer warns. All four RDF generators produce byte-identical output.

The library reports degradation through logging; linkml reports it
through warnings so it is visible without logging configuration.
_DegradedPathWarnings bridges the two, and the tests pin the properties
that makes load-bearing: the warning is attributed to the caller's line,
the library's logger is left as it was found, a caller who configured
logging still receives the record, and warnings survive an exception.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant