docs(owl): document deterministic serialization and --diff-stable - #27
Open
jdsika wants to merge 6 commits into
Open
docs(owl): document deterministic serialization and --diff-stable#27jdsika wants to merge 6 commits into
jdsika wants to merge 6 commits into
Conversation
RDFC-1.0 canonicalization already makes RDF output deterministic:
isomorphic graphs always serialize identically. It does not make output
diffable. Blank nodes are numbered `c14nN` in a single global order, so
inserting one class can renumber every blank node after it and rewrite
most of the file. A one-line semantic change lands as a whole-file diff,
which makes generated OWL/SHACL hard to review and noisy to keep under
version control.
Add a `diff_stable` argument to `canonicalize_rdf_graph()` and a
`--diff-stable/--no-diff-stable` flag to the four RDF generators. When
enabled, blank-node labels are derived from each node's own neighbourhood
via Weisfeiler-Lehman refinement, so an edit relabels only the blank
nodes it actually touches.
Measured churn on a real schema (add one class, count changed lines):
generator default --diff-stable
owlgen 2091 17
shexgen 796 50
shaclgen 291 13
rdfgen 115 25
Output stays deterministic and isomorphic either way; only the choice of
label changes. Off by default, because enabling it relabels existing
output.
The refinement itself lives in `diffable-rdf`, whose only dependencies
(rdflib, pyoxigraph) are already linkml-runtime dependencies at higher
versions, so this adds no new transitive dependencies.
…-opping Bump the floor to diffable-rdf 0.3.0 and add the missing uv.lock entry: the dependency was declared in pyproject.toml but never locked, so "uv lock --check" and the "uv sync --frozen" anti-malware gate would both have failed CI. 0.3.0 also fixes two defects in the Weisfeiler-Lehman labelling this feature relies on. Disconnected blank-node components now converge independently, so an edit in one region no longer relabels an unrelated one. And the suffix used to tell structurally indistinguishable nodes apart was assigned in c14nN *text* order, so c14n10 sorted between c14n1 and c14n2 -- adding a tenth tied blank node relabelled eight of the nine already there, the exact opposite of what this labelling is for. Separately, diff_stable=True was silently ignored whenever pyoxigraph refused the graph and canonicalize_rdf_graph degraded to rdflib. Weisfeiler-Lehman refinement consumes canonical pyoxigraph quads, and that path exists precisely because there are none, so the argument could not be honoured -- but the caller was never told. "shaclgen --include-annotations --diff-stable" reaches it, via the literal predicate an annotation tag without a ':' produces, and returned output byte-identical to --no-diff-stable. It now warns, with a regression test asserting the warning and the byte-identical output that makes silence misleading.
0.4.0 carries graph.base through the library's rdflib fallback, verifying that every absolute IRI of the source survives a re-read rather than dropping the directive outright, and adds a diff_stable parameter to canonicalize_rdf_graph. The lock entry is written by hand because the workspace sets exclude-newer = "7 days", which filters any release younger than that from resolution; 0.3.0 was pinned the same way for the same reason, and both become resolvable normally on 2026-09-18. uv lock --check and uv sync --all-groups both accept the entry. https://github.com/ASCS-eV/diffable-rdf/releases/tag/v0.4.0
Asserts each correctness property against both linkml's copy and diffable_rdf, marking whichever implementation does not hold it as a strict xfail, so the file is a ratchet in both directions. Nine gaps run one way, two of them silent data corruption. One ran the other way -- the library dropped @base on the degraded path -- and that was the last property blocking delegation. diffable-rdf 0.4.0 fixed it, so that case now passes on both sides and carries no mark.
The implementation was extracted into diffable-rdf at the maintainers' request in linkml#3295, but linkml kept its own copy and the two drifted. This deletes the copy and calls the library, which is what the extraction was for. Nine correctness fixes come with it, each already asserted in test_rdf_canonicalize_defects.py and each previously a strict xfail on the linkml side: - a base ending in # no longer rewrites every IRI that merely shares its prefix (silent corruption: output parsed, meaning changed) - a shared rdf:List tail is no longer duplicated (9 triples in, 11 out) - N-Triples refuses a relative IRI instead of writing a file its own parser rejects - literals containing U+2028, U+2029, U+0085 and the other separators str.splitlines() treats as line breaks survive the line sort - degraded RDF/XML, degraded Turtle and json-ld are byte-identical across processes - every format ends with exactly one newline - a Dataset is refused rather than silently flattened Behaviour changes for callers: nt output for a graph containing a relative IRI now raises ValueError rather than writing an unparseable file, and json-ld is canonicalized rather than handed to rdflib, so it no longer warns. All four RDF generators produce byte-identical output. The library reports degradation through logging; linkml reports it through warnings so it is visible without logging configuration. _DegradedPathWarnings bridges the two, and the tests pin the properties that makes load-bearing: the warning is attributed to the caller's line, the library's logger is left as it was found, a caller who configured logging still receives the record, and warnings survive an exception.
gen-owl canonicalizes its output with RDFC-1.0 before serializing, and the determinism work adds a --diff-stable option on top of it, but neither is mentioned anywhere in the generator documentation. Describe what is guaranteed without passing any flag, why RDFC-1.0's sequential blank-node numbering can still produce noisy diffs across schema edits, and what --diff-stable changes. Also record the behaviour users meet in practice but cannot discover from --help: that the same option exists on gen-rdf, gen-shacl and gen-shex, and that graphs which are not standard RDF -- literal predicates from SHACL annotation mode, relative IRIs such as the metamodel's bibo:status <testing> -- take an rdflib fallback that stays reproducible across processes, warns, and deliberately does not honour --diff-stable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Documents the deterministic serialization pipeline in
docs/generators/owl.rst.Documentation only — no behaviour change.
The determinism work is spread over three PRs and none of it is described in the
generator docs.
gen-owlhas canonicalized its output with RDFC-1.0 for a whilealready, and that is not documented either, so users cannot tell what guarantee
they get without passing any flag.
The new section covers:
unconditionally, so isomorphic graphs serialize byte-identically;
sequentially (
_:c14n0,_:c14n1, …), so inserting one statement can renumberevery blank node ordered after it;
--diff-stablechanges, and why it is off by default (turning it onrelabels existing output once);
gen-rdf,gen-shaclandgen-shex;from SHACL annotation mode, relative IRIs such as the metamodel's
bibo:status <testing>— which stays reproducible across processes, warns withRDFCanonicalizationWarning, and deliberately does not honour--diff-stable.Dependencies
Stacked on the determinism series; the diff shown here includes their commits.
Validation
=/-/^/~hierarchy
original design notes: the flag is
--diff-stable/--no-diff-stable(there isno
--deterministicflag), canonicalization inserialize()isunconditional, and the option is present on exactly four generators