diff --git a/CHANGELOG.md b/CHANGELOG.md index 44d1aa1..d4d0b1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,51 @@ about it. ## [Unreleased] +## [0.4.0] - 2026-09-11 + +**Output bytes change** for graphs that take the rdflib fallback path and carry +a base IRI. If you serialize only standard RDF, nothing here changes your +output. Regenerate the affected artifact once and subsequent runs are stable. + +### Added + +- `canonicalize_rdf_graph` accepts `diff_stable=True`, applying the same + Weisfeiler-Leman blank-node labelling as `wl_relabel_quads` so that editing + one part of a graph no longer renumbers blank nodes elsewhere. Opt-in; + output is deterministic and isomorphic to the input either way. The rdflib + fallback path cannot relabel — Weisfeiler-Leman consumes pyoxigraph quads + that path never produces — so it logs a warning rather than passing + silently. + +### Fixed + +- The rdflib fallback no longer drops `graph.base` unconditionally. A document + holding relative references and declaring no base is not self-describing: + RFC 3986 §5.1.3 hands resolution to the retrieval URI, so the same bytes read + from two directories produced two different graphs, and §5.1.4 places that + responsibility on the sender. The base was dropped because rdflib's + `Serializer.relativize` shortens IRIs by string prefix rather than by the + component algorithm RFC 3986 §5.2.2 defines and Turtle §6.3 requires, which + corrupts terms under a base ending in `#`, in `?`, or mid-path-segment. + + The blanket drop over-corrected: it also discarded safe path-segment and + authority-only bases, which are the ones ordinary tooling actually emits. + RFC 3986 specifies resolution and never its inverse, so no static test can + decide this; the rendering is now re-read and the base kept only if every + absolute IRI of the source survives. Only loss counts — a relative source + term is outside the RDF abstract syntax (RDF 1.1 Concepts §3.2) and always + resolves to something on re-reading. Each drop logs a warning naming the base + and an IRI that forced it. + + This was already the documented contract for this path in `docs/api.md` + ("every rendering must verify before it is returned"); only the fallback + did not honour it. + +- A base that is not itself a valid absolute IRI is never declared. rdflib + stores whatever base string it is handed, and Turtle §6.5 `IRIREF` admits no + space, brace or quote, so such a directive yields a document a strict parser + rejects outright. + ## [0.3.0] - 2026-09-11 Two kinds of change here, and the difference matters when you upgrade. diff --git a/README.md b/README.md index 2f07eab..4dd7ee0 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ in another order, or with the blank nodes renamed — produces the same bytes. | Function | Use it for | |---|---| | `deterministic_turtle(graph)` | Diff-stable, idiomatic Turtle. The default choice for files kept in version control. | -| `canonicalize_rdf_graph(graph, output_format="turtle")` | Deterministic serialization using RDFC-1.0 blank-node labels: N-Triples, N-Quads, RDF/XML, TriG, N3, JSON-LD. Its Turtle is laid out differently from `deterministic_turtle`'s — same terms, different presentation. | +| `canonicalize_rdf_graph(graph, output_format="turtle")` | Deterministic serialization using RDFC-1.0 blank-node labels: N-Triples, N-Quads, RDF/XML, TriG, N3, JSON-LD. Its Turtle is laid out differently from `deterministic_turtle`'s — same terms, different presentation. Pass `diff_stable=True` for blank-node labels that keep an edit local. | | `deterministic_json(obj)` | Ordering an existing JSON or JSON-LD document, without touching RDF. | | `well_known_prefix_map()` | Normalizing prefix aliases (`sdo` → `schema`) to rdflib's curated names. | | `wl_blank_node_labels(quads)` | Diff-stable labels for blank nodes in quads you have already canonicalized. | diff --git a/docs/api.md b/docs/api.md index b39ba11..2741ebd 100644 --- a/docs/api.md +++ b/docs/api.md @@ -136,7 +136,11 @@ print(turtle) ```python -def canonicalize_rdf_graph(graph: rdflib.Graph, output_format: str = "turtle") -> str +def canonicalize_rdf_graph( + graph: rdflib.Graph, + output_format: str = "turtle", + diff_stable: bool = False, +) -> str ``` Serializes one graph deterministically in the requested format. Use this when @@ -150,7 +154,22 @@ not expose a standalone RDFC processor or a selectable hash algorithm. This is the lower-level entry point: blank nodes keep their RDFC-1.0 `c14nN` labels, which are deterministic but sequential, so inserting a triple can renumber the rest. For output kept in version control, prefer -`deterministic_turtle`, or apply `wl_relabel_quads` in your own pipeline. +`deterministic_turtle`, or pass `diff_stable=True`. + +**`diff_stable=True`** labels blank nodes by Weisfeiler-Leman refinement +instead, so a label depends on a node's own neighbourhood rather than on the +whole graph. Editing one part of a graph then leaves the rest of the file +untouched, which is what makes a version-controlled diff readable. The option +is opt-in and changes nothing else: output is deterministic either way, and +both renderings are isomorphic to the input, since RDF 1.1 Concepts §3.4 gives +blank-node identifiers no meaning beyond a single document. It is the same +relabelling `wl_relabel_quads` applies, without having to build a pipeline +around it. + +A graph that reaches the rdflib fallback cannot be relabelled — Weisfeiler-Leman +consumes the pyoxigraph quads that graph could not produce — so the call logs a +warning and returns rdflib-canonicalized labels, which are deterministic but +not diff-stable. It never passes silently. **The two entry points lay Turtle out differently.** Both are correct and both preserve every term exactly; only the presentation differs, so the same graph @@ -266,6 +285,35 @@ instead would invent a graph name the input never had. N-Quads goes into a Dataset's default graph for the same reason, and says exactly what N-Triples says. +**`graph.base` is carried on the fallback path too, when it survives.** The +rule is the same as above — every rendering must verify before it is returned — +but the reason it has to be checked is different. The fallback writes through +rdflib, whose `Serializer.relativize` shortens an IRI by string prefix rather +than by the component algorithm RFC 3986 §5.2.2 defines and Turtle §6.3 +requires. Under a base ending in `#`, in `?`, or mid-path-segment it therefore +emits a reference that resolves back to a *different* IRI. + +RFC 3986 specifies resolution and never its inverse, so a relativization has no +conformance criterion of its own and no static test can decide it. The +rendering is instead re-read, and the base is kept only if every **absolute** +IRI of the source graph is still there. Only loss counts: a *relative* source +term is outside the RDF abstract syntax (RDF 1.1 Concepts §3.2) and always +resolves to something on re-reading, so a newly appearing IRI proves nothing. A +base that is not itself a valid absolute IRI is never declared at all, since +Turtle §6.5 `IRIREF` admits no space, brace or quote and an invalid directive +costs more than it saves. Each drop logs a warning naming the base and an IRI +that forced it. + +Keeping it matters because dropping it is not neutral. A document holding +relative references and declaring no base is not self-describing: RFC 3986 +§5.1.3 hands resolution to the retrieval URI, so the reader's own location +becomes part of the graph — the same bytes read from two directories yield two +different graphs. §5.1.4 puts that responsibility on the sender. Where a base +cannot be kept, absolute terms are preserved in preference to relative ones. + +The line-oriented formats are excluded: N-Triples and N-Quads have no base +directive to declare, and rdflib warns and ignores one. + **The line-oriented formats refuse a graph they cannot represent.** N-Triples and N-Quads accept only absolute IRIs — "IRIs may be written only as absolute IRIs", N-Triples 1.1 §2.2 — and a graph reaches the fallback precisely because diff --git a/src/diffable_rdf/__init__.py b/src/diffable_rdf/__init__.py index 9749dcf..a01c7f6 100644 --- a/src/diffable_rdf/__init__.py +++ b/src/diffable_rdf/__init__.py @@ -7,10 +7,11 @@ Public API: deterministic_turtle(graph) Diff-stable, idiomatic Turtle. The usual entry point. - canonicalize_rdf_graph(graph, output_format="turtle") + canonicalize_rdf_graph(graph, output_format="turtle", diff_stable=False) Deterministic serialization using RDFC-1.0 labels in Turtle, N-Triples, N-Quads, RDF/XML, TriG, N3 or JSON-LD. Any other format name is delegated to - rdflib with no determinism guarantee. + rdflib with no determinism guarantee. Pass diff_stable=True for + Weisfeiler-Leman blank-node labels that keep an edit local. deterministic_json(obj, indent=3, preserve_list_order_keys=None) Deterministically ordered JSON, keeping arrays whose order carries JSON-LD meaning. @@ -43,4 +44,4 @@ "__version__", ] -__version__ = "0.3.0" +__version__ = "0.4.0" diff --git a/src/diffable_rdf/canonicalize.py b/src/diffable_rdf/canonicalize.py index 34d279a..4f24891 100644 --- a/src/diffable_rdf/canonicalize.py +++ b/src/diffable_rdf/canonicalize.py @@ -40,6 +40,7 @@ import io import json import logging +from collections.abc import Callable from xml.etree import ElementTree import pyoxigraph as ox @@ -52,6 +53,7 @@ from .graph_input import _require_single_graph from .jsonld import deterministic_json from .namespaces import bind_source_namespaces, prepare_namespaces +from .wl import wl_relabel_quads logger = logging.getLogger(__name__) @@ -414,7 +416,9 @@ def _with_single_trailing_newline(text: str) -> str: return stripped + "\n" if stripped else "" -def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) -> str: +def _deterministic_fallback_serialize( + graph: rdflib.Graph, output_format: str, diff_stable: bool = False +) -> str: """Serialize a graph that pyoxigraph cannot canonicalize, deterministically. pyoxigraph rejects some graphs that rdflib accepts -- notably graphs @@ -432,10 +436,18 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - Relative IRIs are preserved verbatim (not resolved against the base): the goal is deterministic output, and silently rewriting them would - mask what is really a data problem in the source graph. The source - graph's ``base`` is deliberately not carried across either -- rdflib - relativizes against it by naive string prefixing, which corrupts terms - under a hash base (see :func:`deterministic_turtle`). + mask what is really a data problem in the source graph. The source + graph's ``base`` is carried onto formats that can declare one, but only + when re-reading the result still yields every absolute IRI of the source; + rdflib relativizes by string prefix rather than by RFC 3986 section 5.2.2 + component resolution, which corrupts terms under a hash, query or + partial-segment base. See :func:`_render_preserving_base`. + + ``diff_stable`` cannot be honoured here: Weisfeiler-Leman relabelling + consumes pyoxigraph quads, and this path exists precisely because + pyoxigraph would not accept the graph. Requesting it warns rather than + passing silently, so a caller is never told a stability guarantee applies + to bytes that did not receive it. Turtle-family output is rendered without ``( … )`` collection syntax, because that syntax cannot express a list whose tail is referenced more @@ -443,8 +455,18 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - :param graph: The rdflib Graph that pyoxigraph could not parse. :param output_format: Target serialization format (e.g. ``"turtle"``, ``"nt"``). + :param diff_stable: What the caller asked for; only used to warn that this + path cannot deliver it. :return: Deterministic string serialization of the graph. """ + if diff_stable: + logger.warning( + "diff_stable was requested but this graph took the rdflib fallback path, " + "where Weisfeiler-Leman relabelling cannot run because it operates on the " + "pyoxigraph quads this graph could not produce; blank-node labels are " + "canonicalized by rdflib instead, so output remains deterministic but " + "labels are not diff-stable across edits" + ) if output_format.lower() in _LINE_ORIENTED_FORMATS: # rdflib's N-Triples serializer reuses Turtle's term rendering and does # not enforce the absolute-IRI rule, so it will happily write a @@ -485,9 +507,14 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - # know the other's internals. from .turtle import _NoCollectionTurtleSerializer - buffer = io.BytesIO() - _NoCollectionTurtleSerializer(canonical).serialize(buffer, encoding="utf-8") - serialized = buffer.getvalue().decode("utf-8") + def render_turtle() -> str: + buffer = io.BytesIO() + _NoCollectionTurtleSerializer(canonical).serialize(buffer, encoding="utf-8") + return buffer.getvalue().decode("utf-8") + + # The text is Turtle whichever collection-capable alias was asked for, + # so it is Turtle that has to read back. + serialized = _render_preserving_base(graph, canonical, render_turtle, output_format, "turtle") else: # Serialize an independent single Graph. ``to_canonical_graph`` # returns a context-aware dataset container, while this branch needs @@ -506,27 +533,33 @@ def _deterministic_fallback_serialize(graph: rdflib.Graph, output_format: str) - if ox_target is not None else output_format ) - try: - serialized = canonical.serialize(format=serializer_format) - except PluginException: - # rdflib's own "no such format" error, which the public contract - # documents as propagating. - raise - except Exception as exc: - raise ValueError( - f"cannot serialize this graph as {output_format!r}: {exc}. This graph " - "took the fallback path because pyoxigraph could not parse it, and " - f"rdflib's {serializer_format!r} serializer cannot represent it either. " - "turtle, ttl, trig and n3 can carry any graph that reaches this path." - ) from exc - if not serialized.strip() and len(canonical) > 0: - # A delegated serializer that writes nothing for a non-empty graph - # is silent total loss; refuse rather than return it. - raise ValueError( - f"rdflib's {serializer_format!r} serializer produced an empty document for a " - f"graph of {len(canonical)} triples. Use turtle, ttl, trig or n3, which " - "carry any graph that reaches the fallback path." - ) + def render_delegated() -> str: + try: + text = canonical.serialize(format=serializer_format) + except PluginException: + # rdflib's own "no such format" error, which the public contract + # documents as propagating. + raise + except Exception as exc: + raise ValueError( + f"cannot serialize this graph as {output_format!r}: {exc}. This graph " + "took the fallback path because pyoxigraph could not parse it, and " + f"rdflib's {serializer_format!r} serializer cannot represent it either. " + "turtle, ttl, trig and n3 can carry any graph that reaches this path." + ) from exc + if not text.strip() and len(canonical) > 0: + # A delegated serializer that writes nothing for a non-empty graph + # is silent total loss; refuse rather than return it. + raise ValueError( + f"rdflib's {serializer_format!r} serializer produced an empty document for a " + f"graph of {len(canonical)} triples. Use turtle, ttl, trig or n3, which " + "carry any graph that reaches the fallback path." + ) + return text + + serialized = _render_preserving_base( + graph, canonical, render_delegated, output_format, serializer_format + ) if output_format.lower() in _LINE_ORIENTED_FORMATS: # Split on newline characters only. N-Triples permits Unicode line # separators inside quoted literals; RDFLib escapes ``\n`` and ``\r``, @@ -586,6 +619,137 @@ def _is_safe_prefix_iri(iri: str) -> bool: return _is_absolute_iri(iri) +def _iri_sets(graph: rdflib.Graph) -> tuple[set[str], set[str]]: + """Return direct IRI terms and literal datatype IRIs separately. + + :param graph: The graph to read terms from. + :return: ``(direct IRI terms, literal datatype IRIs)``. + """ + direct: set[str] = set() + datatypes: set[str] = set() + for triple in graph: + for term in triple: + if isinstance(term, rdflib.URIRef): + direct.add(str(term)) + elif isinstance(term, rdflib.Literal) and term.datatype is not None: + datatypes.add(str(term.datatype)) + # RDF 1.1 treats a plain string and an xsd:string literal as the same + # literal. Keep that allowance local to datatype positions: an + # xsd:string URIRef used directly in a triple remains significant. + datatypes.discard(str(rdflib.XSD.string)) + return direct, datatypes + + +def _absolute_iris_lost(source: rdflib.Graph, serialized: str, parse_format: str) -> set[str]: + """Return the absolute IRIs of ``source`` that re-reading ``serialized`` does not yield. + + Only *absolute* terms are examined, and only loss counts. RDF 1.1 Concepts + §3.2 requires IRIs in the abstract syntax to be absolute, so those are the + terms a serialization is obliged to preserve. A relative term is outside + that data model; it reaches this path only because pyoxigraph refused the + graph, and on re-reading it always resolves to *some* absolute IRI + (RFC 3986 §5.1.3 makes the retrieval URI the base when a document declares + none). Newly appearing IRIs are therefore unavoidable and prove nothing, + while a *missing* absolute IRI is exactly the damage that relativizing + against an unsuitable base does. + + :param source: The graph that was serialized. + :param serialized: The text produced for it. + :param parse_format: The rdflib parser name for ``serialized``. + :return: Absolute source IRIs absent from the re-parsed graph; empty if none. + """ + source_direct, source_datatypes = _iri_sets(source) + must_survive = {iri for iri in source_direct | source_datatypes if _is_absolute_iri(iri)} + + reparsed = rdflib.Graph() + try: + reparsed.parse(data=serialized, format=parse_format) + except Exception: + # Output that will not parse is a worse result than a dropped base, so + # report total loss and let the caller retry the plainer rendering. If + # that one is unreadable too, its own verification reports it. + return must_survive + + reparsed_direct, reparsed_datatypes = _iri_sets(reparsed) + return must_survive - (reparsed_direct | reparsed_datatypes) + + +def _render_preserving_base( + graph: rdflib.Graph, + canonical: rdflib.Graph, + render: Callable[[], str], + output_format: str, + parse_format: str, +) -> str: + """Render ``canonical``, carrying ``graph.base`` when doing so preserves the graph. + + A document that contains relative references and declares no base is not + self-describing: RFC 3986 §5.1.3 hands resolution to the retrieval URI, and + §5.1.4 states that "a sender of a representation containing relative + references is responsible for ensuring that a base URI for those references + can be established". This path emits relative references whenever the + source graph holds them, so dropping the directive outright moves the + reader's own location into the graph's meaning. + + Carrying it unconditionally is not safe either. rdflib relativizes by + string prefix (``rdflib.serializer.Serializer.relativize``) rather than by + the component algorithm RFC 3986 §5.2.2 defines and Turtle §6.3 requires, + so under a base ending in ``#``, ``?`` or a partial path segment it emits a + reference that resolves to a different IRI than the one it was given. + + RFC 3986 specifies resolution and never the inverse, so a relativization + has no conformance criterion of its own and the only sound test is to + resolve the result back. That is what this does: render with the base, + check that every absolute term survives re-reading, and fall back to a + rendering without it when one does not. + + :param graph: The source graph, read for its base and its expected terms. + :param canonical: The graph being rendered; its ``base`` is set here. + :param render: Serializes ``canonical`` as it currently stands. + :param output_format: The caller's format name, used in the warning. + :param parse_format: The rdflib parser name for ``render``'s output. + :return: The rendering that preserves the graph's absolute IRIs. + """ + base = str(graph.base) if graph.base else None + # N-Triples and N-Quads have no base directive to carry (N-Triples 1.1 + # §2.2); rdflib warns and ignores one, so never offer it. + if base is None or output_format.lower() in _LINE_ORIENTED_FORMATS: + return render() + # rdflib stores whatever base it was handed, including text that is not an + # IRI at all. Writing that into a directive produces a document no strict + # parser will read -- Turtle §6.5 IRIREF admits neither spaces nor braces + # nor quotes -- which is a worse outcome than the relativization this is + # trying to preserve. Asking pyoxigraph is the check, exactly as + # :func:`_is_safe_prefix_iri` does for prefix declarations. + if not _is_absolute_iri(base): + logger.warning( + "the graph's base %r is not a valid absolute IRI, so it cannot be declared " + "in %s output (Turtle section 6.5 IRIREF, RFC 3986 section 4.3); serializing " + "without the base directive", + base, + output_format, + ) + return render() + + canonical.base = base + serialized = render() + lost = _absolute_iris_lost(graph, serialized, parse_format) + if not lost: + return serialized + + logger.warning( + "carrying the graph's base IRI %r into %s output would change %d IRI(s), such as %r, " + "because rdflib relativizes by string prefix rather than by RFC 3986 section 5.2.2 " + "component resolution; serializing without the base directive", + base, + output_format, + len(lost), + sorted(lost)[0], + ) + canonical.base = None + return render() + + def _assert_round_trips(source: rdflib.Graph, serialized: str, output_format: str) -> None: """Raise if ``serialized`` does not say the same thing as ``source``. @@ -634,24 +798,8 @@ def _assert_round_trips(source: rdflib.Graph, serialized: str, output_format: st # reference rdflib mis-resolves would otherwise slip past: pyoxigraph # resolves it correctly, so comparing only canonical forms compares two # correct readings and sees nothing wrong. - def iri_sets(graph: rdflib.Graph) -> tuple[set[str], set[str]]: - """Return direct IRI terms and literal datatype IRIs separately.""" - direct: set[str] = set() - datatypes: set[str] = set() - for triple in graph: - for term in triple: - if isinstance(term, rdflib.URIRef): - direct.add(str(term)) - elif isinstance(term, rdflib.Literal) and term.datatype is not None: - datatypes.add(str(term.datatype)) - # RDF 1.1 treats a plain string and an xsd:string literal as the same - # literal. Keep that allowance local to datatype positions: an - # xsd:string URIRef used directly in a triple remains significant. - datatypes.discard(str(rdflib.XSD.string)) - return direct, datatypes - - source_direct, source_datatypes = iri_sets(source) - reparsed_direct, reparsed_datatypes = iri_sets(reparsed) + source_direct, source_datatypes = _iri_sets(source) + reparsed_direct, reparsed_datatypes = _iri_sets(reparsed) missing = (source_direct - reparsed_direct) | (source_datatypes - reparsed_datatypes) invented = (reparsed_direct - source_direct) | (reparsed_datatypes - source_datatypes) if missing or invented: @@ -684,6 +832,7 @@ def iri_sets(graph: rdflib.Graph) -> tuple[set[str], set[str]]: def canonicalize_rdf_graph( graph: rdflib.Graph, output_format: str = "turtle", + diff_stable: bool = False, ) -> str: """Serialize an rdflib Graph deterministically using RDFC-1.0 canonicalization. @@ -718,6 +867,12 @@ def canonicalize_rdf_graph( ConjunctiveGraph containers are not supported; select an individual graph context. :param output_format: Target serialization format (e.g. ``"turtle"``, ``"nt"``). + :param diff_stable: Label blank nodes by their local neighbourhood + (Weisfeiler-Leman) instead of by RDFC-1.0's whole-graph function, so + that editing one part of a graph does not renumber blank nodes + elsewhere. Output stays deterministic and isomorphic either way; only + the choice of label changes. Not available on the rdflib fallback + path, which warns when it is requested. :return: Deterministic string serialization of the graph. :raises TypeError: If ``graph`` is a Dataset or ConjunctiveGraph container. """ @@ -738,7 +893,7 @@ def canonicalize_rdf_graph( # serializer. What it cannot fix is a plugin whose *traversal* order # varies; see the guarantee wording in the docstring above. return _with_single_trailing_newline( - _deterministic_fallback_serialize(graph, output_format) + _deterministic_fallback_serialize(graph, output_format, diff_stable) ) if ox_format == ox.RdfFormat.RDF_XML: @@ -760,7 +915,7 @@ def canonicalize_rdf_graph( "deterministic (blank-node labels are canonicalized via rdflib) but is not " "canonicalized with pyoxigraph RDFC-1.0." ) - result = _deterministic_fallback_serialize(graph, output_format) + result = _deterministic_fallback_serialize(graph, output_format, diff_stable) if ox_format == ox.RdfFormat.RDF_XML: # RDFLib escapes literal carriage returns as `` ``. Applying # finalization here keeps XML 1.0 representability local to this @@ -775,8 +930,18 @@ def canonicalize_rdf_graph( # 3. Canonicalize blank node labels with RDFC-1.0. dataset.canonicalize(ox.CanonicalizationAlgorithm.RDFC_1_0) - # 4. Sort triples for deterministic ordering. + # 3b. Optionally re-label blank nodes for diff stability. RDFC-1.0 labels + # are a function of the whole graph, so one added triple can renumber every + # blank node in the document and turn a one-line edit into a whole-file + # diff. Weisfeiler-Leman labels depend only on a blank node's local + # neighbourhood, so unrelated regions keep their labels. Output stays + # deterministic and isomorphic either way; this only changes which label + # each blank node receives. quads = list(dataset) + if diff_stable: + quads = wl_relabel_quads(quads) + + # 4. Sort triples for deterministic ordering. sorted_triples = sorted( (ox.Triple(q.subject, q.predicate, q.object) for q in quads), key=lambda t: (str(t.subject), str(t.predicate), str(t.object)), diff --git a/tests/serialization/test_diff_stable_option.py b/tests/serialization/test_diff_stable_option.py new file mode 100644 index 0000000..8dc72ac --- /dev/null +++ b/tests/serialization/test_diff_stable_option.py @@ -0,0 +1,153 @@ +"""Contracts for the ``diff_stable`` option of :func:`canonicalize_rdf_graph`. + +RDFC-1.0 labels blank nodes as a function of the whole graph, so a one-triple +edit can renumber every blank node in a document. Weisfeiler-Leman labels +depend only on a node's local neighbourhood, which keeps unrelated regions of +a file untouched. Both are deterministic; they differ only in how far an edit +propagates through the output. +""" + +from __future__ import annotations + +import difflib +import logging + +import pytest +from rdflib import BNode, Graph, Literal, Namespace, URIRef +from rdflib.compare import isomorphic + +from diffable_rdf import canonicalize_rdf_graph + +EX = Namespace("http://example.org/") +LOGGER_NAME = "diffable_rdf.canonicalize" +FORMATS = ["turtle", "ttl", "trig", "n3", "xml", "nt", "json-ld"] +PARSER = {"ttl": "turtle", "n3": "turtle"} + + +def _nested_graph(count: int) -> Graph: + """Build a graph of ``count`` independent blank-node branches. + + :param count: How many subject/blank-node pairs to create. + :return: The constructed graph. + """ + graph = Graph() + for index in range(count): + node = BNode() + graph.add((EX[f"s{index}"], EX.has, node)) + graph.add((node, EX.value, Literal(index))) + return graph + + +@pytest.mark.parametrize("output_format", FORMATS) +def test_diff_stable_output_is_reproducible(output_format: str) -> None: + """Requesting diff stability does not cost determinism.""" + graph = _nested_graph(6) + + first = canonicalize_rdf_graph(graph, output_format=output_format, diff_stable=True) + second = canonicalize_rdf_graph(graph, output_format=output_format, diff_stable=True) + + assert first == second + + +@pytest.mark.parametrize("output_format", FORMATS) +def test_diff_stable_output_says_the_same_thing(output_format: str) -> None: + """Relabelling is a renaming, so the graph is unchanged. + + RDF 1.1 Concepts section 3.4 gives blank-node identifiers no meaning + beyond a document, so any consistent renaming yields the same graph. + """ + graph = _nested_graph(6) + + plain = canonicalize_rdf_graph(graph, output_format=output_format) + stable = canonicalize_rdf_graph(graph, output_format=output_format, diff_stable=True) + + parser = PARSER.get(output_format, output_format) + assert isomorphic(Graph().parse(data=plain, format=parser), graph) + assert isomorphic(Graph().parse(data=stable, format=parser), graph) + + +def test_diff_stable_defaults_to_off() -> None: + """The option is opt-in, so the default output is unchanged.""" + graph = _nested_graph(6) + + assert canonicalize_rdf_graph(graph) == canonicalize_rdf_graph(graph, diff_stable=False) + + +def _changed_lines(before: str, after: str) -> int: + """Count added and removed lines between two documents. + + :param before: The earlier document. + :param after: The later document. + :return: How many lines the diff touches. + """ + diff = difflib.unified_diff(before.splitlines(), after.splitlines(), n=0) + return sum(1 for line in diff if line[:1] in {"+", "-"} and not line.startswith(("+++", "---"))) + + +def test_diff_stable_confines_an_edit_to_the_part_that_changed() -> None: + """Adding one branch leaves the other branches' labels alone. + + This is the whole point of the option, so it is asserted as a strict + improvement over the default rather than as a fixed number. + """ + graph = _nested_graph(6) + extended = Graph() + for triple in graph: + extended.add(triple) + added = BNode() + extended.add((EX.s99, EX.has, added)) + extended.add((added, EX.value, Literal(99))) + + plain_churn = _changed_lines( + canonicalize_rdf_graph(graph), + canonicalize_rdf_graph(extended), + ) + stable_churn = _changed_lines( + canonicalize_rdf_graph(graph, diff_stable=True), + canonicalize_rdf_graph(extended, diff_stable=True), + ) + + assert stable_churn < plain_churn + + +def test_diff_stable_on_the_fallback_path_is_reported(caplog: pytest.LogCaptureFixture) -> None: + """The fallback names diff stability as unavailable, and still serializes. + + Weisfeiler-Leman relabelling consumes pyoxigraph quads; this path exists + because pyoxigraph refused the graph, so there are none. Passing silently + would tell a caller a guarantee applies to bytes that never received it. + """ + graph = Graph() + node = BNode() + graph.add((URIRef("relative-term"), EX.p, Literal("v"))) + graph.add((EX.s, EX.has, node)) + graph.add((node, EX.value, Literal(1))) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + result = canonicalize_rdf_graph(graph, output_format="turtle", diff_stable=True) + + messages = [record.getMessage() for record in caplog.records] + assert [message for message in messages if "diff_stable was requested" in message] + + # The graph still serializes: the option is unavailable, not fatal. It + # holds a relative IRI, so it cannot be isomorphic to its own re-reading + # (RFC 3986 section 5.1.3 resolves that term at the reader); the blank + # node structure is what diff stability would have touched. + reparsed = Graph().parse(data=result, format="turtle") + assert len(reparsed) == len(graph) + assert sum(1 for triple in reparsed for term in triple if isinstance(term, BNode)) == 2 + + +def test_the_fallback_is_silent_when_diff_stability_was_not_asked_for( + caplog: pytest.LogCaptureFixture, +) -> None: + """The unavailability warning is tied to the request, not to the path.""" + graph = Graph() + graph.add((URIRef("relative-term"), EX.p, Literal("v"))) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + assert not [ + record.getMessage() for record in caplog.records if "diff_stable was requested" in record.getMessage() + ] diff --git a/tests/serialization/test_fallback_base_iri.py b/tests/serialization/test_fallback_base_iri.py new file mode 100644 index 0000000..41b6637 --- /dev/null +++ b/tests/serialization/test_fallback_base_iri.py @@ -0,0 +1,270 @@ +"""Base-IRI contracts for the rdflib fallback path. + +Graphs that pyoxigraph refuses reach a plain rdflib serializer, which +relativizes by string prefix (``rdflib.serializer.Serializer.relativize``) +rather than by the component algorithm RFC 3986 section 5.2.2 defines and +Turtle section 6.3 requires. Carrying a base is therefore correct for some +bases and destructive for others, and RFC 3986 specifies only resolution -- +never its inverse -- so the sole sound test is to resolve the output back. +""" + +from __future__ import annotations + +import logging + +import pytest +from rdflib import Graph, Literal, Namespace, URIRef +from rdflib.compare import isomorphic + +from diffable_rdf import canonicalize_rdf_graph + +EX = Namespace("http://example.org/") +LOGGER_NAME = "diffable_rdf.canonicalize" + +# A relative IRI is what forces the fallback: pyoxigraph rejects it because +# RDF 1.1 Concepts section 3.2 requires IRIs in the abstract syntax to be +# absolute, while rdflib accepts it. +FALLBACK_TRIGGER = URIRef("relative-term") + + +def _fallback_graph(base: str, subjects: list[str]) -> Graph: + """Build a graph with the given base that pyoxigraph will refuse. + + :param base: The base IRI to attach to the graph. + :param subjects: Subject IRIs, absolute or relative, to add. + :return: A graph guaranteed to take the rdflib fallback path. + """ + graph = Graph(base=base) + for subject in subjects: + graph.add((URIRef(subject), EX.p, Literal("v"))) + graph.add((FALLBACK_TRIGGER, EX.p, Literal("v"))) + return graph + + +def _absolute_terms(graph: Graph) -> set[str]: + """Return every absolute IRI appearing anywhere in ``graph``. + + :param graph: The graph to read. + :return: The set of absolute IRI strings. + """ + return { + str(term) + for triple in graph + for term in triple + if isinstance(term, URIRef) and "://" in str(term) + } + + +# Each case pairs a base with terms and states whether the base can be +# declared without changing any of them. The "drop" cases are the ones a +# shape heuristic would get wrong: rejecting only bases that end in "#" +# still corrupts the partial-segment and query cases. +PRESERVING_BASES = [ + pytest.param("http://example.org/d/", ["http://example.org/d/a", "http://example.org/d/b"], id="path-segment"), + pytest.param("http://example.org/d/", ["a", "http://example.org/d/b"], id="path-segment-with-relative-term"), + pytest.param("http://ex.org/", ["http://ex.org/x"], id="authority-only"), + pytest.param("http://ex.org/d/", ["http://ex.org/d/"], id="term-equal-to-base"), +] +CORRUPTING_BASES = [ + pytest.param("http://ex.org/d#", ["http://ex.org/d#a", "http://ex.org/d#b"], id="fragment"), + pytest.param("http://ex.org/a/b", ["http://ex.org/a/bc"], id="partial-segment"), + pytest.param("http://ex.org/d?q=1", ["http://ex.org/d?q=1x"], id="query"), + pytest.param("http://ex.org/d#", ["a", "http://ex.org/d#b"], id="fragment-with-relative-term"), +] + + +@pytest.mark.parametrize(("base", "subjects"), PRESERVING_BASES) +def test_a_preserving_base_is_declared(base: str, subjects: list[str]) -> None: + """The fallback declares a base that does not change any term. + + A document holding relative references with no declared base is not + self-describing (RFC 3986 section 5.1.4), so the base is kept wherever + keeping it is safe. + """ + graph = _fallback_graph(base, subjects) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert "@base" in result + assert _absolute_terms(Graph().parse(data=result, format="turtle")) >= _absolute_terms(graph) + + +@pytest.mark.parametrize(("base", "subjects"), CORRUPTING_BASES) +def test_a_corrupting_base_is_dropped(base: str, subjects: list[str]) -> None: + """The fallback drops a base whose declaration would change a term. + + rdflib's prefix-string relativization is not RFC 3986 section 5.2.2 + resolution, so under these bases it emits references that resolve to + different IRIs than the ones it was given. + """ + graph = _fallback_graph(base, subjects) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert "@base" not in result + assert _absolute_terms(Graph().parse(data=result, format="turtle")) >= _absolute_terms(graph) + + +@pytest.mark.parametrize(("base", "subjects"), PRESERVING_BASES + CORRUPTING_BASES) +def test_every_absolute_term_survives_whatever_the_base(base: str, subjects: list[str]) -> None: + """No base shape loses an absolute IRI, whichever branch is taken. + + This is the invariant the decision exists to protect; the presence or + absence of the directive is only how it is achieved. + """ + graph = _fallback_graph(base, subjects) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert _absolute_terms(Graph().parse(data=result, format="turtle")) >= _absolute_terms(graph) + + +@pytest.mark.parametrize(("base", "subjects"), CORRUPTING_BASES) +def test_dropping_a_base_names_the_term_it_would_have_lost( + base: str, subjects: list[str], caplog: pytest.LogCaptureFixture +) -> None: + """Dropping a base reports which IRI forced the decision.""" + graph = _fallback_graph(base, subjects) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + messages = [record.getMessage() for record in caplog.records] + dropped = [message for message in messages if "without the base directive" in message] + assert len(dropped) == 1 + assert base in dropped[0] + assert any(subject in dropped[0] for subject in subjects if "://" in subject) + + +@pytest.mark.parametrize(("base", "subjects"), PRESERVING_BASES) +def test_keeping_a_base_is_silent( + base: str, subjects: list[str], caplog: pytest.LogCaptureFixture +) -> None: + """A base that preserves every term is kept without a warning.""" + graph = _fallback_graph(base, subjects) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + assert not [ + record.getMessage() + for record in caplog.records + if "without the base directive" in record.getMessage() + ] + + +@pytest.mark.parametrize("output_format", ["turtle", "ttl", "trig", "n3", "xml"]) +def test_the_base_decision_holds_across_formats(output_format: str) -> None: + """Every format that can declare a base keeps the graph's terms.""" + preserving = _fallback_graph("http://example.org/d/", ["http://example.org/d/a"]) + corrupting = _fallback_graph("http://ex.org/d#", ["http://ex.org/d#a"]) + + for graph in (preserving, corrupting): + result = canonicalize_rdf_graph(graph, output_format=output_format) + parser = "xml" if output_format == "xml" else "turtle" + assert _absolute_terms(Graph().parse(data=result, format=parser)) >= _absolute_terms(graph) + + +@pytest.mark.parametrize("output_format", ["nt", "ntriples", "nquads"]) +def test_line_oriented_output_never_declares_a_base(output_format: str) -> None: + """N-Triples and N-Quads have no base directive to declare. + + Their grammars (N-Triples 1.1 section 2.2) admit only absolute IRIs, so a + base is neither expressible nor needed. rdflib warns and ignores one; it + is never offered. + """ + graph = Graph(base="http://example.org/d/") + graph.add((URIRef("http://example.org/d/a"), EX.p, Literal("v"))) + graph.add((URIRef("http://example.org/d/b"), URIRef("literal-predicate-forces-fallback"), Literal("v"))) + + with pytest.raises(ValueError, match="not an absolute IRI"): + canonicalize_rdf_graph(graph, output_format=output_format) + + +@pytest.mark.parametrize( + "base", + [ + pytest.param("http://ex.org/a b/", id="space"), + pytest.param("http://ex.org/x{y}/", id="braces"), + pytest.param('http://ex.org/a"b/', id="quote"), + pytest.param("not a base at all", id="not-an-iri"), + ], +) +def test_a_base_that_is_not_a_valid_iri_is_never_declared(base: str) -> None: + """An unwritable base is dropped rather than emitted as invalid syntax. + + rdflib stores whatever base string it is handed. Turtle section 6.5 admits + no space, brace or quote inside an ``IRIREF``, so declaring such a base + yields a document a strict parser rejects outright -- strictly worse than + the relativization the directive was meant to support. + """ + graph = Graph(base=base) + graph.add((URIRef("http://ex.org/keepme"), EX.p, Literal("v"))) + graph.add((FALLBACK_TRIGGER, EX.p, Literal("v"))) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + + assert "@base" not in result + assert "http://ex.org/keepme" in _absolute_terms(Graph().parse(data=result, format="turtle")) + + +def test_an_invalid_base_says_why_it_was_dropped(caplog: pytest.LogCaptureFixture) -> None: + """Dropping an unwritable base reports the base that could not be used.""" + graph = Graph(base="http://ex.org/a b/") + graph.add((URIRef("http://ex.org/keepme"), EX.p, Literal("v"))) + graph.add((FALLBACK_TRIGGER, EX.p, Literal("v"))) + + with caplog.at_level(logging.WARNING, logger=LOGGER_NAME): + canonicalize_rdf_graph(graph, output_format="turtle") + + assert [ + record.getMessage() + for record in caplog.records + if "is not a valid absolute IRI" in record.getMessage() + ] + + +def test_a_declared_base_makes_the_document_independent_of_where_it_is_read() -> None: + """A carried base fixes resolution inside the document, not at the reader. + + RFC 3986 section 5.1.1 ranks a base embedded in the content above the + retrieval URI of section 5.1.3. Reading the same bytes from two different + locations must therefore yield the same graph -- otherwise the reader's + own location becomes part of the graph's meaning. + """ + graph = _fallback_graph("http://example.org/d/", ["a", "http://example.org/d/b"]) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + assert "@base" in result + + here = Graph().parse(data=result, format="turtle", publicID="http://reader-one.example/somewhere/") + there = Graph().parse(data=result, format="turtle", publicID="file:///a/totally/different/place/") + + assert isomorphic(here, there) + assert {str(term) for triple in here for term in triple} == { + str(term) for triple in there for term in triple + } + + +def test_without_a_declared_base_the_reader_location_leaks_in() -> None: + """Establish that the guarantee above is not vacuous. + + When a base cannot be declared -- here because keeping it would lose + ``http://ex.org/d#b`` -- any relative reference left in the document + resolves against wherever it happens to be read from. There is no correct + answer for such a graph: it holds relative IRIs, which RDF 1.1 Concepts + section 3.2 places outside the abstract syntax. Preserving the absolute + terms is the choice made, and this records its cost. + """ + graph = _fallback_graph("http://ex.org/d#", ["a", "http://ex.org/d#b"]) + + result = canonicalize_rdf_graph(graph, output_format="turtle") + assert "@base" not in result + + here = Graph().parse(data=result, format="turtle", publicID="http://reader-one.example/somewhere/") + there = Graph().parse(data=result, format="turtle", publicID="file:///a/totally/different/place/") + + assert not isomorphic(here, there) + # The terms that RDF actually defines are identical in both readings. + assert _absolute_terms(here) >= _absolute_terms(graph) + assert _absolute_terms(there) >= _absolute_terms(graph)