From 0e92639e4076e7e15843fe11c46026001c1417af Mon Sep 17 00:00:00 2001 From: Carlo van Driesten Date: Fri, 11 Sep 2026 13:17:45 +0200 Subject: [PATCH] Carry graph.base on the fallback path and add diff_stable The rdflib fallback dropped graph.base unconditionally. A document that holds relative references and declares no base is not self-describing: RFC 3986 section 5.1.3 hands resolution to the retrieval URI, so the same bytes read from two directories produced two different graphs, and section 5.1.4 places that responsibility on the sender. The drop had a real cause. rdflib's Serializer.relativize shortens an IRI by string prefix rather than by the component algorithm RFC 3986 section 5.2.2 defines and Turtle section 6.3 requires, so under a base ending in '#', in '?', or mid-path-segment it emits a reference that resolves back to a different IRI. But the defence was too broad: it also discarded path-segment and authority-only bases, which are the ones ordinary tooling emits. RFC 3986 specifies resolution and never its inverse, so a relativization has no conformance criterion of its own and no static test can decide it. The rendering is now re-read and the base kept only if every absolute IRI of the source survives. Only loss counts: a relative source term is outside the RDF abstract syntax (RDF 1.1 Concepts section 3.2) and always resolves to something on re-reading, so a newly appearing IRI proves nothing. A base that is not itself a valid absolute IRI is never declared, since Turtle section 6.5 IRIREF admits no space, brace or quote. docs/api.md already promised this for the path ("every rendering must verify before it is returned"); only the fallback did not honour it. Also adds diff_stable to canonicalize_rdf_graph, applying the same Weisfeiler-Leman labelling as wl_relabel_quads so an edit stops renumbering blank nodes elsewhere in a file. The fallback cannot relabel, because Weisfeiler-Leman consumes pyoxigraph quads that path never produces, so it warns rather than passing silently. --- CHANGELOG.md | 45 +++ README.md | 2 +- docs/api.md | 52 +++- src/diffable_rdf/__init__.py | 7 +- src/diffable_rdf/canonicalize.py | 265 +++++++++++++---- .../serialization/test_diff_stable_option.py | 153 ++++++++++ tests/serialization/test_fallback_base_iri.py | 270 ++++++++++++++++++ 7 files changed, 738 insertions(+), 56 deletions(-) create mode 100644 tests/serialization/test_diff_stable_option.py create mode 100644 tests/serialization/test_fallback_base_iri.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 44d1aa1..d4d0b1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,51 @@ about it. ## [Unreleased] +## [0.4.0] - 2026-09-11 + +**Output bytes change** for graphs that take the rdflib fallback path and carry +a base IRI. If you serialize only standard RDF, nothing here changes your +output. Regenerate the affected artifact once and subsequent runs are stable. + +### Added + +- `canonicalize_rdf_graph` accepts `diff_stable=True`, applying the same + Weisfeiler-Leman blank-node labelling as `wl_relabel_quads` so that editing + one part of a graph no longer renumbers blank nodes elsewhere. Opt-in; + output is deterministic and isomorphic to the input either way. The rdflib + fallback path cannot relabel — Weisfeiler-Leman consumes pyoxigraph quads + that path never produces — so it logs a warning rather than passing + silently. + +### Fixed + +- The rdflib fallback no longer drops `graph.base` unconditionally. A document + holding relative references and declaring no base is not self-describing: + RFC 3986 §5.1.3 hands resolution to the retrieval URI, so the same bytes read + from two directories produced two different graphs, and §5.1.4 places that + responsibility on the sender. The base was dropped because rdflib's + `Serializer.relativize` shortens IRIs by string prefix rather than by the + component algorithm RFC 3986 §5.2.2 defines and Turtle §6.3 requires, which + corrupts terms under a base ending in `#`, in `?`, or mid-path-segment. + + The blanket drop over-corrected: it also discarded safe path-segment and + authority-only bases, which are the ones ordinary tooling actually emits. + RFC 3986 specifies resolution and never its inverse, so no static test can + decide this; the rendering is now re-read and the base kept only if every + absolute IRI of the source survives. Only loss counts — a relative source + term is outside the RDF abstract syntax (RDF 1.1 Concepts §3.2) and always + resolves to something on re-reading. Each drop logs a warning naming the base + and an IRI that forced it. + + This was already the documented contract for this path in `docs/api.md` + ("every rendering must verify before it is returned"); only the fallback + did not honour it. + +- A base that is not itself a valid absolute IRI is never declared. rdflib + stores whatever base string it is handed, and Turtle §6.5 `IRIREF` admits no + space, brace or quote, so such a directive yields a document a strict parser + rejects outright. + ## [0.3.0] - 2026-09-11 Two kinds of change here, and the difference matters when you upgrade. diff --git a/README.md b/README.md index 2f07eab..4dd7ee0 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ in another order, or with the blank nodes renamed — produces the same bytes. | Function | Use it for | |---|---| | `deterministic_turtle(graph)` | Diff-stable, idiomatic Turtle. The default choice for files kept in version control. | -| `canonicalize_rdf_graph(graph, output_format="turtle")` | Deterministic serialization using RDFC-1.0 blank-node labels: N-Triples, N-Quads, RDF/XML, TriG, N3, JSON-LD. Its Turtle is laid out differently from `deterministic_turtle`'s — same terms, different presentation. | +| `canonicalize_rdf_graph(graph, output_format="turtle")` | Deterministic serialization using RDFC-1.0 blank-node labels: N-Triples, N-Quads, RDF/XML, TriG, N3, JSON-LD. Its Turtle is laid out differently from `deterministic_turtle`'s — same terms, different presentation. Pass `diff_stable=True` for blank-node labels that keep an edit local. | | `deterministic_json(obj)` | Ordering an existing JSON or JSON-LD document, without touching RDF. | | `well_known_prefix_map()` | Normalizing prefix aliases (`sdo` → `schema`) to rdflib's curated names. | | `wl_blank_node_labels(quads)` | Diff-stable labels for blank nodes in quads you have already canonicalized. | diff --git a/docs/api.md b/docs/api.md index b39ba11..2741ebd 100644 --- a/docs/api.md +++ b/docs/api.md @@ -136,7 +136,11 @@ print(turtle) ```python -def canonicalize_rdf_graph(graph: rdflib.Graph, output_format: str = "turtle") -> str +def canonicalize_rdf_graph( + graph: rdflib.Graph, + output_format: str = "turtle", + diff_stable: bool = False, +) -> str ``` Serializes one graph deterministically in the requested format. Use this when @@ -150,7 +154,22 @@ not expose a standalone RDFC processor or a selectable hash algorithm. This is the lower-level entry point: blank nodes keep their RDFC-1.0 `c14nN` labels, which are deterministic but sequential, so inserting a triple can renumber the rest. For output kept in version control, prefer -`deterministic_turtle`, or apply `wl_relabel_quads` in your own pipeline. +`deterministic_turtle`, or pass `diff_stable=True`. + +**`diff_stable=True`** labels blank nodes by Weisfeiler-Leman refinement +instead, so a label depends on a node's own neighbourhood rather than on the +whole graph. Editing one part of a graph then leaves the rest of the file +untouched, which is what makes a version-controlled diff readable. The option +is opt-in and changes nothing else: output is deterministic either way, and +both renderings are isomorphic to the input, since RDF 1.1 Concepts §3.4 gives +blank-node identifiers no meaning beyond a single document. It is the same +relabelling `wl_relabel_quads` applies, without having to build a pipeline +around it. + +A graph that reaches the rdflib fallback cannot be relabelled — Weisfeiler-Leman +consumes the pyoxigraph quads that graph could not produce — so the call logs a +warning and returns rdflib-canonicalized labels, which are deterministic but +not diff-stable. It never passes silently. **The two entry points lay Turtle out differently.** Both are correct and both preserve every term exactly; only the presentation differs, so the same graph @@ -266,6 +285,35 @@ instead would invent a graph name the input never had. N-Quads goes into a Dataset's default graph for the same reason, and says exactly what N-Triples says. +**`graph.base` is carried on the fallback path too, when it survives.** The +rule is the same as above — every rendering must verify before it is returned — +but the reason it has to be checked is different. The fallback writes through +rdflib, whose `Serializer.relativize` shortens an IRI by string prefix rather +than by the component algorithm RFC 3986 §5.2.2 defines and Turtle §6.3 +requires. Under a base ending in `#`, in `?`, or mid-path-segment it therefore +emits a reference that resolves back to a *different* IRI. + +RFC 3986 specifies resolution and never its inverse, so a relativization has no +conformance criterion of its own and no static test can decide it. The +rendering is instead re-read, and the base is kept only if every **absolute** +IRI of the source graph is still there. Only loss counts: a *relative* source +term is outside the RDF abstract syntax (RDF 1.1 Concepts §3.2) and always +resolves to something on re-reading, so a newly appearing IRI proves nothing. A +base that is not itself a valid absolute IRI is never declared at all, since +Turtle §6.5 `IRIREF` admits no space, brace or quote and an invalid directive +costs more than it saves. Each drop logs a warning naming the base and an IRI +that forced it. + +Keeping it matters because dropping it is not neutral. A document holding +relative references and declaring no base is not self-describing: RFC 3986 +§5.1.3 hands resolution to the retrieval URI, so the reader's own location +becomes part of the graph — the same bytes read from two directories yield two +different graphs. §5.1.4 puts that responsibility on the sender. Where a base +cannot be kept, absolute terms are preserved in preference to relative ones. + +The line-oriented formats are excluded: N-Triples and N-Quads have no base +directive to declare, and rdflib warns and ignores one. + **The line-oriented formats refuse a graph they cannot represent.** N-Triples and N-Quads accept only absolute IRIs — "IRIs may be written only as absolute IRIs", N-Triples 1.1 §2.2 — and a graph reaches the fallback precisely because diff --git a/src/diffable_rdf/__init__.py b/src/diffable_rdf/__init__.py index 9749dcf..a01c7f6 100644 --- a/src/diffable_rdf/__init__.py +++ b/src/diffable_rdf/__init__.py @@ -7,10 +7,11 @@ Public API: deterministic_turtle(graph) Diff-stable, idiomatic Turtle. The usual entry point. - canonicalize_rdf_graph(graph, output_format="turtle") + canonicalize_rdf_graph(graph, output_format="turtle", diff_stable=False) Deterministic serialization using RDFC-1.0 labels in Turtle, N-Triples, N-Quads, RDF/XML, TriG, N3 or JSON-LD. Any other format name is delegated to - rdflib with no determinism guarantee. + rdflib with no determinism guarantee. Pass diff_stable=True for + Weisfeiler-Leman blank-node labels that keep an edit local. deterministic_json(obj, indent=3, preserve_list_order_keys=None) Deterministically ordered JSON, keeping arrays whose order carries JSON-LD meaning. @@ -43,4 +44,4 @@ "__version__", ] -__version__ = "0.3.0" +__version__ = "0.4.0" diff --git a/src/diffable_rdf/canonicalize.py b/src/diffable_rdf/canonicalize.py index 34d279a..4f24891 100644 --- a/src/diffable_rdf/canonicalize.py +++ b/src/diffable_rdf/canonicalize.py @@ -40,6 +40,7 @@ import io import json import logging +from collections.abc import Callable from xml.etree import ElementTree import pyoxigraph as ox @@ -52,6 +53,7 @@ from .graph_input import _require_single_graph from .jsonld import deterministic_json from .namespaces import bind_source_namespaces, prepare_namespaces +from .wl import wl_relabel_quads logger = logging.getLogger(__name__) @@ -414,7 +416,9 @@ def _with_single_trailing_newline(text: str) -> str: return stripped + "\n" if stripped else "" -def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) -> str: +def _deterministic_fallback_serialize( + graph: rdflib.Graph, output_format: str, diff_stable: bool = False +) -> str: """Serialize a graph that pyoxigraph cannot canonicalize, deterministically. pyoxigraph rejects some graphs that rdflib accepts -- notably graphs @@ -432,10 +436,18 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - Relative IRIs are preserved verbatim (not resolved against the base): the goal is deterministic output, and silently rewriting them would - mask what is really a data problem in the source graph. The source - graph's ``base`` is deliberately not carried across either -- rdflib - relativizes against it by naive string prefixing, which corrupts terms - under a hash base (see :func:`deterministic_turtle`). + mask what is really a data problem in the source graph. The source + graph's ``base`` is carried onto formats that can declare one, but only + when re-reading the result still yields every absolute IRI of the source; + rdflib relativizes by string prefix rather than by RFC 3986 section 5.2.2 + component resolution, which corrupts terms under a hash, query or + partial-segment base. See :func:`_render_preserving_base`. + + ``diff_stable`` cannot be honoured here: Weisfeiler-Leman relabelling + consumes pyoxigraph quads, and this path exists precisely because + pyoxigraph would not accept the graph. Requesting it warns rather than + passing silently, so a caller is never told a stability guarantee applies + to bytes that did not receive it. Turtle-family output is rendered without ``( … )`` collection syntax, because that syntax cannot express a list whose tail is referenced more @@ -443,8 +455,18 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - :param graph: The rdflib Graph that pyoxigraph could not parse. :param output_format: Target serialization format (e.g. ``"turtle"``, ``"nt"``). + :param diff_stable: What the caller asked for; only used to warn that this + path cannot deliver it. :return: Deterministic string serialization of the graph. """ + if diff_stable: + logger.warning( + "diff_stable was requested but this graph took the rdflib fallback path, " + "where Weisfeiler-Leman relabelling cannot run because it operates on the " + "pyoxigraph quads this graph could not produce; blank-node labels are " + "canonicalized by rdflib instead, so output remains deterministic but " + "labels are not diff-stable across edits" + ) if output_format.lower() in _LINE_ORIENTED_FORMATS: # rdflib's N-Triples serializer reuses Turtle's term rendering and does # not enforce the absolute-IRI rule, so it will happily write a @@ -485,9 +507,14 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - # know the other's internals. from .turtle import _NoCollectionTurtleSerializer - buffer = io.BytesIO() - _NoCollectionTurtleSerializer(canonical).serialize(buffer, encoding="utf-8") - serialized = buffer.getvalue().decode("utf-8") + def render_turtle() -> str: + buffer = io.BytesIO() + _NoCollectionTurtleSerializer(canonical).serialize(buffer, encoding="utf-8") + return buffer.getvalue().decode("utf-8") + + # The text is Turtle whichever collection-capable alias was asked for, + # so it is Turtle that has to read back. + serialized = _render_preserving_base(graph, canonical, render_turtle, output_format, "turtle") else: # Serialize an independent single Graph. ``to_canonical_graph`` # returns a context-aware dataset container, while this branch needs @@ -506,27 +533,33 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - if ox_target is not None else output_format ) - try: - serialized = canonical.serialize(format=serializer_format) - except PluginException: - # rdflib's own "no such format" error, which the public contract - # documents as propagating. - raise - except Exception as exc: - raise ValueError( - f"cannot serialize this graph as {output_format!r}: {exc}. This graph " - "took the fallback path because pyoxigraph could not parse it, and " - f"rdflib's {serializer_format!r} serializer cannot represent it either. " - "turtle, ttl, trig and n3 can carry any graph that reaches this path." - ) from exc - if not serialized.strip() and len(canonical) > 0: - # A delegated serializer that writes nothing for a non-empty graph - # is silent total loss; refuse rather than return it. - raise ValueError( - f"rdflib's {serializer_format!r} serializer produced an empty document for a " - f"graph of {len(canonical)} triples. Use turtle, ttl, trig or n3, which " - "carry any graph that reaches the fallback path." - ) + def render_delegated() -> str: + try: + text = canonical.serialize(format=serializer_format) + except PluginException: + # rdflib's own "no such format" error, which the public contract + # documents as propagating. + raise + except Exception as exc: + raise ValueError( + f"cannot serialize this graph as {output_format!r}: {exc}. This graph " + "took the fallback path because pyoxigraph could not parse it, and " + f"rdflib's {serializer_format!r} serializer cannot represent it either. " + "turtle, ttl, trig and n3 can carry any graph that reaches this path." + ) from exc + if not text.strip() and len(canonical) > 0: + # A delegated serializer that writes nothing for a non-empty graph + # is silent total loss; refuse rather than return it. + raise ValueError( + f"rdflib's {serializer_format!r} serializer produced an empty document for a " + f"graph of {len(canonical)} triples. Use turtle, ttl, trig or n3, which " + "carry any graph that reaches the fallback path." + ) + return text + + serialized = _render_preserving_base( + graph, canonical, render_delegated, output_format, serializer_format + ) if output_format.lower() in _LINE_ORIENTED_FORMATS: # Split on newline characters only. N-Triples permits Unicode line # separators inside quoted literals; RDFLib escapes ``\n`` and ``\r``, @@ -586,6 +619,137 @@ def _is_safe_prefix_iri(iri: str) -> bool: return _is_absolute_iri(iri) +def _iri_sets(graph: rdflib.Graph) -> tuple[set[str], set[str]]: + """Return direct IRI terms and literal datatype IRIs separately. + + :param graph: The graph to read terms from. + :return: ``(direct IRI terms, literal datatype IRIs)``. + """ + direct: set[str] = set() + datatypes: set[str] = set() + for triple in graph: + for term in triple: + if isinstance(term, rdflib.URIRef): + direct.add(str(term)) + elif isinstance(term, rdflib.Literal) and term.datatype is not None: + datatypes.add(str(term.datatype)) + # RDF 1.1 treats a plain string and an xsd:string literal as the same + # literal. Keep that allowance local to datatype positions: an + # xsd:string URIRef used directly in a triple remains significant. + datatypes.discard(str(rdflib.XSD.string)) + return direct, datatypes + + +def _absolute_iris_lost(source: rdflib.Graph, serialized: str, parse_format: str) -> set[str]: + """Return the absolute IRIs of ``source`` that re-reading ``serialized`` does not yield. + + Only *absolute* terms are examined, and only loss counts. RDF 1.1 Concepts + §3.2 requires IRIs in the abstract syntax to be absolute, so those are the + terms a serialization is obliged to preserve. A relative term is outside + that data model; it reaches this path only because pyoxigraph refused the + graph, and on re-reading it always resolves to *some* absolute IRI + (RFC 3986 §5.1.3 makes the retrieval URI the base when a document declares + none). Newly appearing IRIs are therefore unavoidable and prove nothing, + while a *missing* absolute IRI is exactly the damage that relativizing + against an unsuitable base does. + + :param source: The graph that was serialized. + :param serialized: The text produced for it. + :param parse_format: The rdflib parser name for ``serialized``. + :return: Absolute source IRIs absent from the re-parsed graph; empty if none. + """ + source_direct, source_datatypes = _iri_sets(source) + must_survive = {iri for iri in source_direct | source_datatypes if _is_absolute_iri(iri)} + + reparsed = rdflib.Graph() + try: + reparsed.parse(data=serialized, format=parse_format) + except Exception: + # Output that will not parse is a worse result than a dropped base, so + # report total loss and let the caller retry the plainer rendering. If + # that one is unreadable too, its own verification reports it. + return must_survive + + reparsed_direct, reparsed_datatypes = _iri_sets(reparsed) + return must_survive - (reparsed_direct | reparsed_datatypes) + + +def _render_preserving_base( + graph: rdflib.Graph, + canonical: rdflib.Graph, + render: Callable[[], str], + output_format: str, + parse_format: str, +) -> str: + """Render ``canonical``, carrying ``graph.base`` when doing so preserves the graph. + + A document that contains relative references and declares no base is not + self-describing: RFC 3986 §5.1.3 hands resolution to the retrieval URI, and + §5.1.4 states that "a sender of a representation containing relative + references is responsible for ensuring that a base URI for those references + can be established". This path emits relative references whenever the + source graph holds them, so dropping the directive outright moves the + reader's own location into the graph's meaning. + + Carrying it unconditionally is not safe either. rdflib relativizes by + string prefix (``rdflib.serializer.Serializer.relativize``) rather than by + the component algorithm RFC 3986 §5.2.2 defines and Turtle §6.3 requires, + so under a base ending in ``#``, ``?`` or a partial path segment it emits a + reference that resolves to a different IRI than the one it was given. + + RFC 3986 specifies resolution and never the inverse, so a relativization + has no conformance criterion of its own and the only sound test is to + resolve the result back. That is what this does: render with the base, + check that every absolute term survives re-reading, and fall back to a + rendering without it when one does not. + + :param graph: The source graph, read for its base and its expected terms. + :param canonical: The graph being rendered; its ``base`` is set here. + :param render: Serializes ``canonical`` as it currently stands. + :param output_format: The caller's format name, used in the warning. + :param parse_format: The rdflib parser name for ``render``'s output. + :return: The rendering that preserves the graph's absolute IRIs. + """ + base = str(graph.base) if graph.base else None + # N-Triples and N-Quads have no base directive to carry (N-Triples 1.1 + # §2.2); rdflib warns and ignores one, so never offer it. + if base is None or output_format.lower() in _LINE_ORIENTED_FORMATS: + return render() + # rdflib stores whatever base it was handed, including text that is not an + # IRI at all. Writing that into a directive produces a document no strict + # parser will read -- Turtle §6.5 IRIREF admits neither spaces nor braces + # nor quotes -- which is a worse outcome than the relativization this is + # trying to preserve. Asking pyoxigraph is the check, exactly as + # :func:`_is_safe_prefix_iri` does for prefix declarations. + if not _is_absolute_iri(base): + logger.warning( + "the graph's base %r is not a valid absolute IRI, so it cannot be declared " + "in %s output (Turtle section 6.5 IRIREF, RFC 3986 section 4.3); serializing " + "without the base directive", + base, + output_format, + ) + return render() + + canonical.base = base + serialized = render() + lost = _absolute_iris_lost(graph, serialized, parse_format) + if not lost: + return serialized + + logger.warning( + "carrying the graph's base IRI %r into %s output would change %d IRI(s), such as %r, " + "because rdflib relativizes by string prefix rather than by RFC 3986 section 5.2.2 " + "component resolution; serializing without the base directive", + base, + output_format, + len(lost), + sorted(lost)[0], + ) + canonical.base = None + return render() + + def _assert_round_trips(source: rdflib.Graph, serialized: str, output_format: str) -> None: """Raise if ``serialized`` does not say the same thing as ``source``. @@ -634,24 +798,8 @@ def _assert_round_trips(source: rdflib.Graph, serialized: str, output_format: st # reference rdflib mis-resolves would otherwise slip past: pyoxigraph # resolves it correctly, so comparing only canonical forms compares two # correct readings and sees nothing wrong. - def iri_sets(graph: rdflib.Graph) -> tuple[set[str], set[str]]: - """Return direct IRI terms and literal datatype IRIs separately.""" - direct: set[str] = set() - datatypes: set[str] = set() - for triple in graph: - for term in triple: - if isinstance(term, rdflib.URIRef): - direct.add(str(term)) - elif isinstance(term, rdflib.Literal) and term.datatype is not None: - datatypes.add(str(term.datatype)) - # RDF 1.1 treats a plain string and an xsd:string literal as the same - # literal. Keep that allowance local to datatype positions: an - # xsd:string URIRef used directly in a triple remains significant. - datatypes.discard(str(rdflib.XSD.string)) - return direct, datatypes - - source_direct, source_datatypes = iri_sets(source) - reparsed_direct, reparsed_datatypes = iri_sets(reparsed) + source_direct, source_datatypes = _iri_sets(source) + reparsed_direct, reparsed_datatypes = _iri_sets(reparsed) missing = (source_direct - reparsed_direct) | (source_datatypes - reparsed_datatypes) invented = (reparsed_direct - source_direct) | (reparsed_datatypes - source_datatypes) if missing or invented: @@ -684,6 +832,7 @@ def iri_sets(graph: rdflib.Graph) -> tuple[set[str], set[str]]: def canonicalize_rdf_graph( graph: rdflib.Graph, output_format: str = "turtle", + diff_stable: bool = False, ) -> str: """Serialize an rdflib Graph deterministically using RDFC-1.0 canonicalization. @@ -718,6 +867,12 @@ def canonicalize_rdf_graph( ConjunctiveGraph containers are not supported; select an individual graph context. :param output_format: Target serialization format (e.g. ``"turtle"``, ``"nt"``). + :param diff_stable: Label blank nodes by their local neighbourhood + (Weisfeiler-Leman) instead of by RDFC-1.0's whole-graph function, so + that editing one part of a graph does not renumber blank nodes + elsewhere. Output stays deterministic and isomorphic either way; only + the choice of label changes. Not available on the rdflib fallback + path, which warns when it is requested. :return: Deterministic string serialization of the graph. :raises TypeError: If ``graph`` is a Dataset or ConjunctiveGraph container. """ @@ -738,7 +893,7 @@ def canonicalize_rdf_graph( # serializer. What it cannot fix is a plugin whose *traversal* order # varies; see the guarantee wording in the docstring above. return _with_single_trailing_newline( - _deterministic_fallback_serialize(graph, output_format) + _deterministic_fallback_serialize(graph, output_format, diff_stable) ) if ox_format == ox.RdfFormat.RDF_XML: @@ -760,7 +915,7 @@ def canonicalize_rdf_graph( "deterministic (blank-node labels are canonicalized via rdflib) but is not " "canonicalized with pyoxigraph RDFC-1.0." ) - result = _deterministic_fallback_serialize(graph, output_format) + result = _deterministic_fallback_serialize(graph, output_format, diff_stable) if ox_format == ox.RdfFormat.RDF_XML: # RDFLib escapes literal carriage returns as `` ``. Applying # finalization here keeps XML 1.0 representability local to this @@ -775,8 +930,18 @@ def canonicalize_rdf_graph( # 3. Canonicalize blank node labels with RDFC-1.0. dataset.canonicalize(ox.CanonicalizationAlgorithm.RDFC_1_0) - # 4. Sort triples for deterministic ordering. + # 3b. Optionally re-label blank nodes for diff stability. RDFC-1.0 labels + # are a function of the whole graph, so one added triple can renumber every + # blank node in the document and turn a one-line edit into a whole-file + # diff. Weisfeiler-Leman labels depend only on a blank node's local + # neighbourhood, so unrelated regions keep their labels. Output stays + # deterministic and isomorphic either way; this only changes which label + # each blank node receives. quads = list(dataset) + if diff_stable: + quads = wl_relabel_quads(quads) + + # 4. Sort triples for deterministic ordering. sorted_triples = sorted( (ox.Triple(q.subject, q.predicate, q.object) for q in quads), key=lambda t: (str(t.subject), str(t.predicate), str(t.object)), diff --git a/tests/serialization/test_diff_stable_option.py b/tests/serialization/test_diff_stable_option.py new file mode 100644 index 0000000..8dc72ac --- /dev/null +++ b/tests/serialization/test_diff_stable_option.py @@ -0,0 +1,153 @@ +"""Contracts for the ``diff_stable`` option of :func:`canonicalize_rdf_graph`. + +RDFC-1.0 labels blank nodes as a function of the whole graph, so a one-triple +edit can renumber every blank node in a document. Weisfeiler-Leman labels +depend only on a node's local neighbourhood, which keeps unrelated regions of +a file untouched. Both are deterministic; they differ only in how far an edit +propagates through the output. +""" + +from __future__ import annotations + +import difflib +import logging + +import pytest +from rdflib import BNode, Graph, Literal, Namespace, URIRef +from rdflib.compare import isomorphic + +from diffable_rdf import canonicalize_rdf_graph + +EX = Namespace("http://example.org/") +LOGGER_NAME = "diffable_rdf.canonicalize" +FORMATS = ["turtle", "ttl", "trig", "n3", "xml", "nt", "json-ld"] +PARSER = {"ttl": "turtle", "n3": "turtle"} + + +def _nested_graph(count: int) -> Graph: + """Build a graph of ``count`` independent blank-node branches. + + :param count: How many subject/blank-node pairs to create. + :return: The constructed graph. + """ + graph = Graph() + for index in range(count): + node = BNode() + graph.add((EX[f"s{index}"], EX.has, node)) + graph.add((node, EX.value, Literal(index))) + return graph + + +@pytest.mark.parametrize("output_format", FORMATS) +def test_diff_stable_output_is_reproducible(output_format: str) -> None: + """Requesting diff stability does not cost determinism.""" + graph = _nested_graph(6) + + first = canonicalize_rdf_graph(graph, output_format=output_format, diff_stable=True) + second = canonicalize_rdf_graph(graph, output_format=output_format, diff_stable=True) + + assert first == second + + +@pytest.mark.parametrize("output_format", FORMATS) +def test_diff_stable_output_says_the_same_thing(output_format: str) -> None: + """Relabelling is a renaming, so the graph is unchanged. + + RDF 1.1 Concepts section 3.4 gives blank-node identifiers no meaning + beyond a document, so any consistent renaming yields the same graph. + """ + graph = _nested_graph(6) + + plain = canonicalize_rdf_graph(graph, output_format=output_format) + stable = canonicalize_rdf_graph(graph, output_format=output_format, diff_stable=True) + + parser = PARSER.get(output_format, output_format) + assert isomorphic(Graph().parse(data=plain, format=parser), graph) + assert isomorphic(Graph().parse(data=stable, format=parser), graph) + + +def test_diff_stable_defaults_to_off() -> None: + """The option is opt-in, so the default output is unchanged.""" + graph = _nested_graph(6) + + assert canonicalize_rdf_graph(graph) == canonicalize_rdf_graph(graph, diff_stable=False) + + +def _changed_lines(before: str, after: str) -> int: + """Count added and removed lines between two documents. + + :param before: The earlier document. + :param after: The later document. + :return: How many lines the diff touches. + """ + diff = difflib.unified_diff(before.splitlines(), after.splitlines(), n=0) + return sum(1 for line in diff if line[:1] in {"+", "-"} and not line.startswith(("+++", "---"))) + + +def test_diff_stable_confines_an_edit_to_the_part_that_changed() -> None: + """Adding one branch leaves the other branches' labels alone. + + This is the whole point of the option, so it is asserted as a strict + improvement over the default rather than as a fixed number. + """ + graph = _nested_graph(6) + extended = Graph() + for triple in graph: + extended.add(triple) + added = BNode() + extended.add((EX.s99, EX.has, added)) + extended.add((added, EX.value, Literal(99))) + + plain_churn = _changed_lines( + canonicalize_rdf_graph(graph), + canonicalize_rdf_graph(extended), + ) + stable_churn = _changed_lines( + canonicalize_rdf_graph(graph, diff_stable=True), + canonicalize_rdf_graph(extended, diff_stable=True), + ) + + assert stable_churn < plain_churn + + +def test_diff_stable_on_the_fallback_path_is_reported(caplog: pytest.LogCaptureFixture) -> None: + """The fallback names diff stability as unavailable, and still serializes. + + Weisfeiler-Leman relabelling consumes pyoxigraph quads; this path exists + because pyoxigraph refused the graph, so there are none. Passing silently + would tell a caller a guarantee applies to bytes that never received it. + """ + graph = Graph() + node = BNode() + graph.add((URIRef("relative-term"), EX.p, Literal("v"))) + graph.add((EX.s, EX.has, node)) + graph.add((node, EX.value, Literal(1))) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + result = canonicalize_rdf_graph(graph, output_format="turtle", diff_stable=True) + + messages = [record.getMessage() for record in caplog.records] + assert [message for message in messages if "diff_stable was requested" in message] + + # The graph still serializes: the option is unavailable, not fatal. It + # holds a relative IRI, so it cannot be isomorphic to its own re-reading + # (RFC 3986 section 5.1.3 resolves that term at the reader); the blank + # node structure is what diff stability would have touched. + reparsed = Graph().parse(data=result, format="turtle") + assert len(reparsed) == len(graph) + assert sum(1 for triple in reparsed for term in triple if isinstance(term, BNode)) == 2 + + +def test_the_fallback_is_silent_when_diff_stability_was_not_asked_for( + caplog: pytest.LogCaptureFixture, +) -> None: + """The unavailability warning is tied to the request, not to the path.""" + graph = Graph() + graph.add((URIRef("relative-term"), EX.p, Literal("v"))) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + assert not [ + record.getMessage() for record in caplog.records if "diff_stable was requested" in record.getMessage() + ] diff --git a/tests/serialization/test_fallback_base_iri.py b/tests/serialization/test_fallback_base_iri.py new file mode 100644 index 0000000..41b6637 --- /dev/null +++ b/tests/serialization/test_fallback_base_iri.py @@ -0,0 +1,270 @@ +"""Base-IRI contracts for the rdflib fallback path. + +Graphs that pyoxigraph refuses reach a plain rdflib serializer, which +relativizes by string prefix (``rdflib.serializer.Serializer.relativize``) +rather than by the component algorithm RFC 3986 section 5.2.2 defines and +Turtle section 6.3 requires. Carrying a base is therefore correct for some +bases and destructive for others, and RFC 3986 specifies only resolution -- +never its inverse -- so the sole sound test is to resolve the output back. +""" + +from __future__ import annotations + +import logging + +import pytest +from rdflib import Graph, Literal, Namespace, URIRef +from rdflib.compare import isomorphic + +from diffable_rdf import canonicalize_rdf_graph + +EX = Namespace("http://example.org/") +LOGGER_NAME = "diffable_rdf.canonicalize" + +# A relative IRI is what forces the fallback: pyoxigraph rejects it because +# RDF 1.1 Concepts section 3.2 requires IRIs in the abstract syntax to be +# absolute, while rdflib accepts it. +FALLBACK_TRIGGER = URIRef("relative-term") + + +def _fallback_graph(base: str, subjects: list[str]) -> Graph: + """Build a graph with the given base that pyoxigraph will refuse. + + :param base: The base IRI to attach to the graph. + :param subjects: Subject IRIs, absolute or relative, to add. + :return: A graph guaranteed to take the rdflib fallback path. + """ + graph = Graph(base=base) + for subject in subjects: + graph.add((URIRef(subject), EX.p, Literal("v"))) + graph.add((FALLBACK_TRIGGER, EX.p, Literal("v"))) + return graph + + +def _absolute_terms(graph: Graph) -> set[str]: + """Return every absolute IRI appearing anywhere in ``graph``. + + :param graph: The graph to read. + :return: The set of absolute IRI strings. + """ + return { + str(term) + for triple in graph + for term in triple + if isinstance(term, URIRef) and "://" in str(term) + } + + +# Each case pairs a base with terms and states whether the base can be +# declared without changing any of them. The "drop" cases are the ones a +# shape heuristic would get wrong: rejecting only bases that end in "#" +# still corrupts the partial-segment and query cases. +PRESERVING_BASES = [ + pytest.param("http://example.org/d/", ["http://example.org/d/a", "http://example.org/d/b"], id="path-segment"), + pytest.param("http://example.org/d/", ["a", "http://example.org/d/b"], id="path-segment-with-relative-term"), + pytest.param("http://ex.org/", ["http://ex.org/x"], id="authority-only"), + pytest.param("http://ex.org/d/", ["http://ex.org/d/"], id="term-equal-to-base"), +] +CORRUPTING_BASES = [ + pytest.param("http://ex.org/d#", ["http://ex.org/d#a", "http://ex.org/d#b"], id="fragment"), + pytest.param("http://ex.org/a/b", ["http://ex.org/a/bc"], id="partial-segment"), + pytest.param("http://ex.org/d?q=1", ["http://ex.org/d?q=1x"], id="query"), + pytest.param("http://ex.org/d#", ["a", "http://ex.org/d#b"], id="fragment-with-relative-term"), +] + + +@pytest.mark.parametrize(("base", "subjects"), PRESERVING_BASES) +def test_a_preserving_base_is_declared(base: str, subjects: list[str]) -> None: + """The fallback declares a base that does not change any term. + + A document holding relative references with no declared base is not + self-describing (RFC 3986 section 5.1.4), so the base is kept wherever + keeping it is safe. + """ + graph = _fallback_graph(base, subjects) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert "@base" in result + assert _absolute_terms(Graph().parse(data=result, format="turtle")) >= _absolute_terms(graph) + + +@pytest.mark.parametrize(("base", "subjects"), CORRUPTING_BASES) +def test_a_corrupting_base_is_dropped(base: str, subjects: list[str]) -> None: + """The fallback drops a base whose declaration would change a term. + + rdflib's prefix-string relativization is not RFC 3986 section 5.2.2 + resolution, so under these bases it emits references that resolve to + different IRIs than the ones it was given. + """ + graph = _fallback_graph(base, subjects) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert "@base" not in result + assert _absolute_terms(Graph().parse(data=result, format="turtle")) >= _absolute_terms(graph) + + +@pytest.mark.parametrize(("base", "subjects"), PRESERVING_BASES + CORRUPTING_BASES) +def test_every_absolute_term_survives_whatever_the_base(base: str, subjects: list[str]) -> None: + """No base shape loses an absolute IRI, whichever branch is taken. + + This is the invariant the decision exists to protect; the presence or + absence of the directive is only how it is achieved. + """ + graph = _fallback_graph(base, subjects) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert _absolute_terms(Graph().parse(data=result, format="turtle")) >= _absolute_terms(graph) + + +@pytest.mark.parametrize(("base", "subjects"), CORRUPTING_BASES) +def test_dropping_a_base_names_the_term_it_would_have_lost( + base: str, subjects: list[str], caplog: pytest.LogCaptureFixture +) -> None: + """Dropping a base reports which IRI forced the decision.""" + graph = _fallback_graph(base, subjects) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + messages = [record.getMessage() for record in caplog.records] + dropped = [message for message in messages if "without the base directive" in message] + assert len(dropped) == 1 + assert base in dropped[0] + assert any(subject in dropped[0] for subject in subjects if "://" in subject) + + +@pytest.mark.parametrize(("base", "subjects"), PRESERVING_BASES) +def test_keeping_a_base_is_silent( + base: str, subjects: list[str], caplog: pytest.LogCaptureFixture +) -> None: + """A base that preserves every term is kept without a warning.""" + graph = _fallback_graph(base, subjects) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + assert not [ + record.getMessage() + for record in caplog.records + if "without the base directive" in record.getMessage() + ] + + +@pytest.mark.parametrize("output_format", ["turtle", "ttl", "trig", "n3", "xml"]) +def test_the_base_decision_holds_across_formats(output_format: str) -> None: + """Every format that can declare a base keeps the graph's terms.""" + preserving = _fallback_graph("http://example.org/d/", ["http://example.org/d/a"]) + corrupting = _fallback_graph("http://ex.org/d#", ["http://ex.org/d#a"]) + + for graph in (preserving, corrupting): + result = canonicalize_rdf_graph(graph, output_format=output_format) + parser = "xml" if output_format == "xml" else "turtle" + assert _absolute_terms(Graph().parse(data=result, format=parser)) >= _absolute_terms(graph) + + +@pytest.mark.parametrize("output_format", ["nt", "ntriples", "nquads"]) +def test_line_oriented_output_never_declares_a_base(output_format: str) -> None: + """N-Triples and N-Quads have no base directive to declare. + + Their grammars (N-Triples 1.1 section 2.2) admit only absolute IRIs, so a + base is neither expressible nor needed. rdflib warns and ignores one; it + is never offered. + """ + graph = Graph(base="http://example.org/d/") + graph.add((URIRef("http://example.org/d/a"), EX.p, Literal("v"))) + graph.add((URIRef("http://example.org/d/b"), URIRef("literal-predicate-forces-fallback"), Literal("v"))) + + with pytest.raises(ValueError, match="not an absolute IRI"): + canonicalize_rdf_graph(graph, output_format=output_format) + + +@pytest.mark.parametrize( + "base", + [ + pytest.param("http://ex.org/a b/", id="space"), + pytest.param("http://ex.org/x{y}/", id="braces"), + pytest.param('http://ex.org/a"b/', id="quote"), + pytest.param("not a base at all", id="not-an-iri"), + ], +) +def test_a_base_that_is_not_a_valid_iri_is_never_declared(base: str) -> None: + """An unwritable base is dropped rather than emitted as invalid syntax. + + rdflib stores whatever base string it is handed. Turtle section 6.5 admits + no space, brace or quote inside an ``IRIREF``, so declaring such a base + yields a document a strict parser rejects outright -- strictly worse than + the relativization the directive was meant to support. + """ + graph = Graph(base=base) + graph.add((URIRef("http://ex.org/keepme"), EX.p, Literal("v"))) + graph.add((FALLBACK_TRIGGER, EX.p, Literal("v"))) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert "@base" not in result + assert "http://ex.org/keepme" in _absolute_terms(Graph().parse(data=result, format="turtle")) + + +def test_an_invalid_base_says_why_it_was_dropped(caplog: pytest.LogCaptureFixture) -> None: + """Dropping an unwritable base reports the base that could not be used.""" + graph = Graph(base="http://ex.org/a b/") + graph.add((URIRef("http://ex.org/keepme"), EX.p, Literal("v"))) + graph.add((FALLBACK_TRIGGER, EX.p, Literal("v"))) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + assert [ + record.getMessage() + for record in caplog.records + if "is not a valid absolute IRI" in record.getMessage() + ] + + +def test_a_declared_base_makes_the_document_independent_of_where_it_is_read() -> None: + """A carried base fixes resolution inside the document, not at the reader. + + RFC 3986 section 5.1.1 ranks a base embedded in the content above the + retrieval URI of section 5.1.3. Reading the same bytes from two different + locations must therefore yield the same graph -- otherwise the reader's + own location becomes part of the graph's meaning. + """ + graph = _fallback_graph("http://example.org/d/", ["a", "http://example.org/d/b"]) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + assert "@base" in result + + here = Graph().parse(data=result, format="turtle", publicID="http://reader-one.example/somewhere/") + there = Graph().parse(data=result, format="turtle", publicID="file:///a/totally/different/place/") + + assert isomorphic(here, there) + assert {str(term) for triple in here for term in triple} == { + str(term) for triple in there for term in triple + } + + +def test_without_a_declared_base_the_reader_location_leaks_in() -> None: + """Establish that the guarantee above is not vacuous. + + When a base cannot be declared -- here because keeping it would lose + ``http://ex.org/d#b`` -- any relative reference left in the document + resolves against wherever it happens to be read from. There is no correct + answer for such a graph: it holds relative IRIs, which RDF 1.1 Concepts + section 3.2 places outside the abstract syntax. Preserving the absolute + terms is the choice made, and this records its cost. + """ + graph = _fallback_graph("http://ex.org/d#", ["a", "http://ex.org/d#b"]) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + assert "@base" not in result + + here = Graph().parse(data=result, format="turtle", publicID="http://reader-one.example/somewhere/") + there = Graph().parse(data=result, format="turtle", publicID="file:///a/totally/different/place/") + + assert not isomorphic(here, there) + # The terms that RDF actually defines are identical in both readings. + assert _absolute_terms(here) >= _absolute_terms(graph) + assert _absolute_terms(there) >= _absolute_terms(graph)