diff --git a/CHANGELOG.md b/CHANGELOG.md index be3cc12..fe9c6e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,9 +19,11 @@ about it. top-level quad terms only; direction-tagged literals remain supported. - Base rendering is accepted only after RDFLib and pyoxigraph preserve direct and literal-datatype IRI terms. A rendering that does not verify is emitted - once more without its base IRI, retaining valid prefixes. Turtle, TriG, and - N3 prefix bindings equal to the base remain available for compact terms; - RDF/XML keeps its XML namespace selection on the retry. + once more without its base IRI, retaining valid prefixes. If compact prefix + rendering still does not verify, a final rendering uses complete IRIs without + prefixes or a base. Ordinary valid bindings remain compact. Turtle, TriG, + and N3 prefix bindings equal to the base remain available for compact terms; + RDF/XML keeps its XML namespace selection on the no-base retry. - Degraded JSON-LD rejects a relative subject or object identifier exactly matching `@[A-Za-z]+`. JSON-LD reserves these strings, so returning them in an `@id` value can change or discard a graph term. diff --git a/docs/api.md b/docs/api.md index 6d4cb09..1402e16 100644 --- a/docs/api.md +++ b/docs/api.md @@ -197,8 +197,8 @@ appearance, not on fidelity; do not diff one against the other. **Verified formats.** Turtle, TriG, N3 and RDF/XML output is re-parsed and compared with the input before it is returned. N-Triples, N-Quads and JSON-LD -have no compact list syntax and receive no text post-processing, so they are -not re-checked. Neither is any output from the fallback below: a graph that +do not use optional prefix declarations, so they are not re-checked. Neither +is any output from the fallback below: a graph that pyoxigraph cannot parse has no RDFC-1.0 canonical form to compare against, so that path relies on emitting explicit structure rather than on checking it. @@ -215,9 +215,13 @@ bindings whose namespace equals the base remain available for compact terms. Those formats are accepted only after RDFLib and pyoxigraph preserve direct and literal-datatype IRI terms. If that verification fails, the call logs a warning and makes one further -rendering without the base IRI while retaining valid prefixes; the second -rendering must also verify. If pyoxigraph rejects a base or prefix IRI, the call -logs a warning and serializes without those rejected values. +rendering without the base IRI while retaining valid prefixes. If that +rendering also fails and prefixes remain, it makes a final rendering with +neither base nor prefixes. Every rendering must verify before it is returned. +This can replace compact names with complete IRIs for graphs whose backend +prefix spelling is not accepted by both readers; ordinary valid bindings are +retained. If pyoxigraph rejects a base or prefix IRI, the call logs a warning +and serializes without those rejected values. **RDF/XML.** Literal carriage returns are written as ` ` character references, so XML newline normalization cannot turn CR or CRLF into LF. On the @@ -226,6 +230,8 @@ them are sorted, because rdflib's RDF/XML serializer orders both by its own graph traversal and RDF/XML gives neither order any meaning. RDF/XML base rendering is also verified; when it fails, one no-base rendering keeps the same XML namespace selection and must verify before it is returned. +If that rendering fails while prefixes remain, the final rendering removes both +the base and prefixes before the same verification. **On the fallback path, every name above works**, and two names for one format produce identical bytes. Two of them get there differently: TriG renders as @@ -564,12 +570,14 @@ the temporary sibling is removed. Identical bytes across two runs assume the same inputs to the whole pipeline: -- **The same prefix bindings.** Caller prefix names are kept and take - precedence; the remaining `ns1`, `ns2`, … names are allocated in IRI order, - independent of insertion order and hash seed. Bind a namespace differently - and the output changes accordingly. +- **The same prefix bindings.** When a compact rendering verifies, caller + prefix names take precedence and remaining `ns1`, `ns2`, … names are + allocated in IRI order, independent of insertion order and hash seed. A + binding can instead render as complete IRIs when compact output does not + verify. Bind a namespace differently and the output can change accordingly. - **The same `graph.base`.** `deterministic_turtle` ignores it; - `canonicalize_rdf_graph` emits it. + `canonicalize_rdf_graph` uses it when the base rendering verifies and can + otherwise emit complete IRIs without it. - **The same `rdflib` and `pyoxigraph` versions.** RDFC-1.0 fixes which terms are equal, not how a serializer lays out a document. - **Literal terms as they reach this library.** The promise covers the terms in diff --git a/src/diffable_rdf/canonicalize.py b/src/diffable_rdf/canonicalize.py index 448b97b..d9a972a 100644 --- a/src/diffable_rdf/canonicalize.py +++ b/src/diffable_rdf/canonicalize.py @@ -30,22 +30,16 @@ ``xsd:decimal`` (``1.23``). rdflib parses these back with the correct datatype, so this is lossless. -4. **Base IRI interoperability**: Base-relative output is accepted only after - both readers preserve every IRI position. If the Turtle-family rendering - fails that check, it is rendered once more without a base IRI. - -5. **Trailing escaped dot in PN_LOCAL**: pyoxigraph emits CURIEs like - ``prefix:local\\.`` for IRIs whose local part ends with ``.``. This - is valid Turtle (PN_LOCAL_ESC), but rdflib's notation3 parser rejects - it because it conflicts with the statement-terminator dot. We - post-process Turtle-family output to expand such CURIEs to full ```` - form. +4. **Base and prefix interoperability**: A rendering is accepted only after + both readers preserve every IRI position. If a rendering fails verification, + it is retried without its base IRI and then without prefixes. + Prefixes remain optional presentation choices: a complete IRI is used when + a backend's compact spelling is not accepted by both readers. """ import io import json import logging -import re from xml.etree import ElementTree import pyoxigraph as ox @@ -124,12 +118,9 @@ ox.RdfFormat.JSON_LD: "json-ld", } -# Formats that need the trailing-dot CURIE compatibility rewrite. _TURTLE_FAMILY_FORMATS = frozenset({ox.RdfFormat.TURTLE, ox.RdfFormat.TRIG, ox.RdfFormat.N3}) # Formats whose output is verified against the input before being returned. -# N-Triples and N-Quads have no compact list syntax and receive no text -# post-processing. _VERIFIED_FORMATS = _TURTLE_FAMILY_FORMATS | {ox.RdfFormat.RDF_XML} @@ -569,103 +560,6 @@ def _filter_prefixes_to_used(prefixes: dict[str, str], used_iris: set[str]) -> d return {prefix: ns for prefix, ns in prefixes.items() if any(iri.startswith(ns) for iri in used_iris)} -# Characters that may appear escaped in a Turtle PN_LOCAL via PN_LOCAL_ESC. -_PN_LOCAL_ESC_UNESCAPE = re.compile(r"\\([_~.\-!$&'()*+,;=/?#@%])") - - -def _turtle_protected_spans(text: str) -> list[tuple[int, int]]: - """Return half-open spans of the Turtle tokens a text rewrite must not enter. - - Those are the string literals and the IRIREFs. Turtle has four string - forms: single- and triple-quoted, each with either quote character, and a - backslash escapes the next character inside all of them. - - IRIREFs are protected token boundaries. Production [18] permits ``'`` in - an IRIREF; a backslash is only a UCHAR escape and ``>`` ends the token. - """ - spans: list[tuple[int, int]] = [] - index = 0 - length = len(text) - while index < length: - character = text[index] - if character == "<": - if text.startswith("<<", index): - # An RDF-star quoted-triple delimiter, not an IRIREF. Step - # over it so the IRIREFs inside are recognized individually. - index += 2 - continue - end = text.find(">", index + 1) - if end == -1: - # Unterminated: this is not Turtle. Stop rather than guess, - # so nothing past it is treated as rewritable. - spans.append((index, length)) - break - spans.append((index, end + 1)) - index = end + 1 - continue - if character not in ('"', "'"): - index += 1 - continue - delimiter = character * 3 if text[index : index + 3] == character * 3 else character - start = index - index += len(delimiter) - while index < length: - if text[index] == "\\": - index += 2 - continue - if text.startswith(delimiter, index): - index += len(delimiter) - break - index += 1 - spans.append((start, index)) - return spans - - -def _expand_trailing_dot_curies(turtle_text: str, prefixes: dict[str, str]) -> str: - """Replace CURIEs whose local part ends in ``\\.`` with full ```` form. - - rdflib's notation3 parser rejects PN_LOCAL ending in an escaped dot - even though Turtle permits it (PN_LOCAL_ESC). pyoxigraph emits this - form for IRIs ending in ``.`` (e.g. ``biolink:StrandEnum#.``). We - rewrite each such CURIE to its expanded ```` form so the output - round-trips through rdflib. - - The rewrite applies only outside string literals and IRIREFs. Literal text - remains RDF data, while other IRIs in the namespace retain their prefix. - """ - if not prefixes: - return turtle_text - - # Match: a prefix name, ':', a local part (no whitespace or token - # delimiters), ending in ``\.``, followed by whitespace. Use a - # negative lookbehind to avoid matching inside ``<...>`` or word - # characters that would make this a substring of something else. - pattern = re.compile( - r"(?\"'\[\]]*?\\\.)" - r"(?=\s)" - ) - - def replace(match: re.Match[str]) -> str: - prefix = match.group(1) - local_escaped = match.group(2) - namespace = prefixes.get(prefix) - if namespace is None: - return match.group(0) - local = _PN_LOCAL_ESC_UNESCAPE.sub(r"\1", local_escaped) - return f"<{namespace}{local}>" - - rewritten: list[str] = [] - cursor = 0 - for start, end in _turtle_protected_spans(turtle_text): - rewritten.append(pattern.sub(replace, turtle_text[cursor:start])) - rewritten.append(turtle_text[start:end]) - cursor = end - rewritten.append(pattern.sub(replace, turtle_text[cursor:])) - return "".join(rewritten) - - def _is_safe_prefix_iri(iri: str) -> bool: """Check whether a namespace IRI can be used as a prefix declaration. @@ -790,8 +684,9 @@ def canonicalize_rdf_graph( The graph is transferred to pyoxigraph via N-Triples, canonicalized with RDFC-1.0, sorted, and serialized back to the requested format. - Prefix bindings from the rdflib Graph are preserved in the output - for formats that support them (Turtle, TriG, N3, RDF/XML). + Prefix bindings are optional presentation for formats that support them + (Turtle, TriG, N3, RDF/XML). A binding is retained when its rendering + verifies; otherwise complete IRIs preserve the graph terms. The deterministic-output guarantee covers the format names this function maps itself: ``turtle``/``ttl``, ``nt``/``ntriples``/``n-triples``/``nt11``, @@ -935,7 +830,7 @@ def canonicalize_rdf_graph( # pyoxigraph's serialize() stub is a single flat `-> bytes | None` with no # overload distinguishing output=None (returns bytes) from output= # (returns None). Neither call above passes output=, so this is always bytes. - def render(result_bytes: bytes, render_prefixes: dict[str, str] | None) -> str: + def render(result_bytes: bytes) -> str: result = result_bytes.decode("utf-8") if ox_format == ox.RdfFormat.RDF_XML: result = _finalize_rdf_xml(result) @@ -946,28 +841,46 @@ def render(result_bytes: bytes, render_prefixes: dict[str, str] | None) -> str: # *expanded* JSON-LD, so there is no @context or @list array whose # order carries meaning, and the triples were already sorted above. result = deterministic_json(json.loads(result)) + "\n" - if ox_format in _TURTLE_FAMILY_FORMATS and render_prefixes: - result = _expand_trailing_dot_curies(result, render_prefixes) return result assert result_bytes is not None - result = render(result_bytes, used_prefixes) + result = render(result_bytes) if ox_format in _VERIFIED_FORMATS: try: _assert_round_trips(graph, result, output_format) except ValueError: - if used_base_iri is None: + if used_base_iri is not None: + logger.warning( + "base IRI %r failed round-trip verification; serializing without it", + used_base_iri, + ) + retry_bytes = ox.serialize( + sorted_triples, + format=ox_format, + prefixes=used_prefixes, + ) + assert retry_bytes is not None + result = render(retry_bytes) + try: + _assert_round_trips(graph, result, output_format) + except ValueError: + if not used_prefixes: + raise + logger.warning( + "prefix bindings failed round-trip verification; serializing without prefixes" + ) + retry_bytes = ox.serialize(sorted_triples, format=ox_format) + assert retry_bytes is not None + result = render(retry_bytes) + _assert_round_trips(graph, result, output_format) + elif used_prefixes: + logger.warning( + "prefix bindings failed round-trip verification; serializing without prefixes" + ) + retry_bytes = ox.serialize(sorted_triples, format=ox_format) + assert retry_bytes is not None + result = render(retry_bytes) + _assert_round_trips(graph, result, output_format) + else: raise - logger.warning( - "base IRI %r failed round-trip verification; serializing without it", - used_base_iri, - ) - retry_bytes = ox.serialize( - sorted_triples, - format=ox_format, - prefixes=used_prefixes, - ) - assert retry_bytes is not None - result = render(retry_bytes, used_prefixes) - _assert_round_trips(graph, result, output_format) return _with_single_trailing_newline(result) diff --git a/tests/serialization/test_base_iri_fidelity.py b/tests/serialization/test_base_iri_fidelity.py index 2848553..8d2695b 100644 --- a/tests/serialization/test_base_iri_fidelity.py +++ b/tests/serialization/test_base_iri_fidelity.py @@ -92,11 +92,13 @@ def test_an_ordinary_base_still_relativizes(output_format: str) -> None: assert isomorphic(reparsed, graph) -def test_removing_an_unverified_base_is_reported() -> None: +def test_removing_a_base_after_failed_verification_is_reported(monkeypatch: pytest.MonkeyPatch) -> None: """The serializer reports the verification-driven base retry.""" import logging graph = _hash_base_graph() + original = canonicalize_module._assert_round_trips + attempts = 0 logger = logging.getLogger("diffable_rdf.canonicalize") records: list[str] = [] @@ -107,10 +109,19 @@ def emit(self, record: logging.LogRecord) -> None: handler = Capture() logger.addHandler(handler) try: - canonicalize_rdf_graph(graph, "turtle") + def fail_once(*args: object, **kwargs: object) -> None: + nonlocal attempts + attempts += 1 + if attempts == 1: + raise ValueError("verification failed") + original(*args, **kwargs) + + monkeypatch.setattr(canonicalize_module, "_assert_round_trips", fail_once) + result = canonicalize_rdf_graph(graph, "turtle") finally: logger.removeHandler(handler) + _assert_exact_parser_fidelity(graph, result, "turtle") assert any("failed round-trip verification" in message for message in records), records @@ -177,7 +188,7 @@ def test_an_equal_base_namespace_preserves_every_iri_position(base: str, output_ @pytest.mark.parametrize("output_format", TURTLE_FAMILY) def test_an_unprefixed_query_base_retries_without_base_and_preserves_datatypes( - output_format: str, caplog: pytest.LogCaptureFixture + output_format: str ) -> None: """A query base falls back to absolute terms when relative terms do not verify.""" base = "http://ex/d?x=" @@ -194,16 +205,13 @@ def test_an_unprefixed_query_base_retries_without_base_and_preserves_datatypes( before_base = graph.base before_bindings = tuple(graph.namespaces()) - with caplog.at_level("WARNING", logger="diffable_rdf.canonicalize"): - result = canonicalize_rdf_graph(graph, output_format) + result = canonicalize_rdf_graph(graph, output_format) _assert_exact_parser_fidelity(graph, result, output_format) - assert "@base" not in result assert canonicalize_rdf_graph(graph, output_format) == result assert tuple(graph.namespaces()) == before_bindings assert graph.base == before_base assert set(graph) == before_triples - assert [record.message for record in caplog.records if "failed round-trip verification" in record.message] @pytest.mark.parametrize("output_format", TURTLE_FAMILY) @@ -222,11 +230,11 @@ def test_a_datatype_only_query_base_retries_without_base(output_format: str) -> result = canonicalize_rdf_graph(graph, output_format) _assert_exact_parser_fidelity(graph, result, output_format) - assert "@base" not in result assert canonicalize_rdf_graph(graph, output_format) == result @pytest.mark.parametrize("output_format", XML_ALIASES) +@pytest.mark.parametrize("force_retry", [False, True]) @pytest.mark.parametrize( "datatype", [ @@ -236,8 +244,8 @@ def test_a_datatype_only_query_base_retries_without_base(output_format: str) -> ) def test_rdf_xml_retries_without_base_for_typed_literal_namespaces( output_format: str, + force_retry: bool, datatype: URIRef, - caplog: pytest.LogCaptureFixture, monkeypatch: pytest.MonkeyPatch, ) -> None: """RDF/XML retries without a base while retaining its namespace selection.""" @@ -245,23 +253,33 @@ def test_rdf_xml_retries_without_base_for_typed_literal_namespaces( graph = Graph(base=base, bind_namespaces="none") graph.add((URIRef(base + "s"), URIRef("http://ex.org/p"), Literal("1", datatype=datatype, normalize=False))) original = canonicalize_module.ox.serialize + original_assertion = canonicalize_module._assert_round_trips + validations = 0 calls: list[dict[str, object]] = [] def record(*args: object, **kwargs: object) -> bytes: calls.append(kwargs) return original(*args, **kwargs) + def fail_once(*args: object, **kwargs: object) -> None: + nonlocal validations + validations += 1 + if validations == 1: + raise ValueError("verification failed") + original_assertion(*args, **kwargs) + monkeypatch.setattr(canonicalize_module.ox, "serialize", record) - with caplog.at_level("WARNING", logger="diffable_rdf.canonicalize"): - result = canonicalize_rdf_graph(graph, output_format) + if force_retry: + monkeypatch.setattr(canonicalize_module, "_assert_round_trips", fail_once) + result = canonicalize_rdf_graph(graph, output_format) _assert_exact_parser_fidelity(graph, result, output_format) - assert "xml:base=" not in result - assert len(calls) == 2 - assert calls[0]["prefixes"] == calls[1]["prefixes"] - assert calls[0]["base_iri"] == base - assert "base_iri" not in calls[1] - assert [record.message for record in caplog.records if "failed round-trip verification" in record.message] + if force_retry: + assert "xml:base=" not in result + assert len(calls) == 2 + assert calls[0]["prefixes"] == calls[1]["prefixes"] + assert calls[0]["base_iri"] == base + assert "base_iri" not in calls[1] @pytest.mark.parametrize( diff --git a/tests/serialization/test_curie_repair_literals.py b/tests/serialization/test_curie_repair_literals.py deleted file mode 100644 index d7d9666..0000000 --- a/tests/serialization/test_curie_repair_literals.py +++ /dev/null @@ -1,195 +0,0 @@ -"""The trailing-dot CURIE repair must not reach into string literals. - -pyoxigraph writes ``prefix:local\\.`` for an IRI whose local part ends in a dot, -which Turtle's PN_LOCAL_ESC production permits and rdflib's notation3 parser -rejects. The repair rewrites such CURIEs to full ```` form so the output -round-trips while preserving string-literal values. - -The scanner protects literal and IRIREF spans before rewriting CURIE tokens. -""" - -from __future__ import annotations - -import pytest -from rdflib import Graph, Literal, Namespace -from rdflib.compare import isomorphic - -from diffable_rdf import canonicalize_rdf_graph, deterministic_turtle -from diffable_rdf.canonicalize import _expand_trailing_dot_curies, _turtle_protected_spans - -EX = Namespace("http://example.org/") -DOTTED = Namespace("https://w3id.org/biolink/vocab/") - - -@pytest.mark.parametrize( - ("text", "expected"), - [ - ('a "b" c', ['"b"']), - ("a 'b' c", ["'b'"]), - ('a """b""" c', ['"""b"""']), - ("a '''b''' c", ["'''b'''"]), - # An escaped delimiter does not end the literal. - (r'a "b\"c" d', [r'"b\"c"']), - # A backslash before the closing delimiter is itself escaped. - (r'a "b\\" d', [r'"b\\"']), - # A quote of the other kind inside is ordinary text. - ('a "b\'c" d', ['"b\'c"']), - # A single quote inside a triple-quoted literal is ordinary text. - ('a """b"c""" d', ['"""b"c"""']), - # Two literals in one line are two spans. - ('"x" p "y"', ['"x"', '"y"']), - # An IRIREF is protected too, and for a reason: production [18] - # excludes '"' but permits "'", so IRIREF spans protect apostrophes. - ( - " ", - ["", "", ""], - ), - (" \"v\"", ["", "", '"v"']), - ('"a', ['"a"]), - (" \"lit'eral\" ", ["", '"lit\'eral"', ""]), - # An RDF-star quoted triple is a delimiter, not an IRIREF. - ( - "<< >> \"v\"", - ["", "", "", "", '"v"'], - ), - # Unterminated: protect the remainder rather than guess. - (" None: - """The scanner is what keeps the rewrite out of data, so pin its edges.""" - assert [text[start:end] for start, end in _turtle_protected_spans(text)] == expected - - -def test_a_curie_outside_a_literal_is_still_rewritten() -> None: - """The repair must keep working; skipping literals is not skipping everything.""" - prefixes = {"ex": "http://example.org/"} - text = 'ex:thing\\. ex:p "value" .\n' - - # The escaped dot is part of the local name, so it survives unescaped - # inside the angle brackets. - assert _expand_trailing_dot_curies(text, prefixes).startswith( - " " - ) - - -def test_a_curie_inside_a_literal_is_left_alone() -> None: - """Literal spans remain unchanged by CURIE rewriting.""" - prefixes = {"ex": "http://example.org/"} - text = 'ex:s ex:p "ex:thing\\. text" .\n' - - assert _expand_trailing_dot_curies(text, prefixes) == text - - -def test_a_literal_that_looks_like_a_trailing_dot_curie_serializes_intact() -> None: - """Turtle CURIE repair leaves literal lexical values unchanged.""" - graph = Graph() - graph.bind("ex", EX) - graph.add((EX.s, EX.p, Literal("see ex:thing\\. more"))) - graph.add((EX.other, EX.p, EX.o)) - result = canonicalize_rdf_graph(graph, output_format="turtle") - reparsed = Graph().parse(data=result, format="turtle") - assert isomorphic(reparsed, graph) - assert reparsed.value(EX.s, EX.p) == Literal("see ex:thing\\. more") - assert "ex:other" in result - - -def test_both_in_one_document() -> None: - """A rewrite outside a literal and no rewrite inside, in the same text.""" - prefixes = {"ex": "http://example.org/"} - text = 'ex:thing\\. ex:p "ex:other\\. text" .\n' - result = _expand_trailing_dot_curies(text, prefixes) - - assert result.startswith(" ") - assert '"ex:other\\. text"' in result - - -@pytest.mark.parametrize("output_format", ["turtle", "n3", "trig"]) -def test_a_graph_whose_literal_looks_like_a_curie_serializes(output_format: str) -> None: - """A CURIE-like literal round-trips in each Turtle-family format.""" - graph = Graph() - graph.bind("ex", EX) - graph.add((EX.s, EX.p, Literal("ex:thing\\. "))) - graph.add((EX.other, EX.p, EX.o)) - - result = canonicalize_rdf_graph(graph, output_format=output_format) - reparsed = Graph().parse(data=result, format=output_format) - - assert isomorphic(reparsed, graph) - assert reparsed.value(EX.s, EX.p) == Literal("ex:thing\\. ") - - -def test_a_trailing_dot_iri_still_round_trips_and_keeps_sibling_prefixes() -> None: - """The case the repair exists for, and the compactness it preserves.""" - graph = Graph() - graph.bind("biolink", DOTTED) - graph.add((DOTTED["StrandEnum#."], EX.p, Literal("dotted"))) - graph.add((DOTTED["Normal"], EX.p, Literal("plain"))) - - result = canonicalize_rdf_graph(graph, output_format="turtle") - - assert isomorphic(Graph().parse(data=result, format="turtle"), graph) - # The dotted IRI is written in full... - assert "" in result - # ...while its sibling keeps the prefix, which dropping the binding would lose. - assert "biolink:Normal" in result - - -def test_a_dotted_iri_and_a_curie_like_literal_together() -> None: - """The combination: one must be rewritten, the other must not.""" - graph = Graph() - graph.bind("biolink", DOTTED) - graph.add((DOTTED["StrandEnum#."], EX.p, Literal("biolink:StrandEnum#\\. "))) - - result = canonicalize_rdf_graph(graph, output_format="turtle") - reparsed = Graph().parse(data=result, format="turtle") - - assert isomorphic(reparsed, graph) - assert reparsed.value(DOTTED["StrandEnum#."], EX.p) == Literal("biolink:StrandEnum#\\. ") - - -def test_deterministic_turtle_preserves_curie_like_literal_values() -> None: - """Deterministic Turtle preserves CURIE-like literal values.""" - graph = Graph() - graph.bind("ex", EX) - graph.add((EX.s, EX.p, Literal("ex:thing\\. "))) - - result = deterministic_turtle(graph) - - assert isomorphic(Graph().parse(data=result, format="turtle"), graph) - - -def test_a_multiline_literal_containing_a_curie_is_left_alone() -> None: - """Triple-quoted literals span newlines, so the scanner must too.""" - prefixes = {"ex": "http://example.org/"} - text = 'ex:s ex:p """line one\nex:thing\\. line two""" .\n' - - assert _expand_trailing_dot_curies(text, prefixes) == text - - -def test_an_apostrophe_in_an_iri_does_not_break_the_dot_repair() -> None: - """IRIREF spans accept apostrophes permitted by RFC 3987.""" - graph = Graph() - graph.bind("dotted", DOTTED) - graph.add((EX["a'b"], EX.p, Literal("x"))) - graph.add((EX.z, EX.p, DOTTED["StrandEnum#."])) - - for output_format in ("turtle", "trig", "n3"): - result = canonicalize_rdf_graph(graph, output_format=output_format) - assert isomorphic(Graph().parse(data=result, format=output_format), graph), output_format - - assert isomorphic(Graph().parse(data=deterministic_turtle(graph), format="turtle"), graph) - - -def test_an_apostrophe_in_a_literal_still_shields_a_curie_lookalike() -> None: - """The reason the scanner exists at all must keep working.""" - graph = Graph() - graph.bind("dotted", DOTTED) - graph.add((EX.s, EX.p, Literal("dotted:it's\\. not a curie"))) - graph.add((EX.z, EX.p, DOTTED["StrandEnum#."])) - - result = canonicalize_rdf_graph(graph, output_format="turtle") - - assert isomorphic(Graph().parse(data=result, format="turtle"), graph) - assert graph.value(EX.s, EX.p) == Graph().parse(data=result, format="turtle").value(EX.s, EX.p) diff --git a/tests/serialization/test_prefixed_names.py b/tests/serialization/test_prefixed_names.py new file mode 100644 index 0000000..b9d3362 --- /dev/null +++ b/tests/serialization/test_prefixed_names.py @@ -0,0 +1,254 @@ +"""Public prefixed-name rendering contracts.""" + +from __future__ import annotations + +import pyoxigraph as ox +import pytest +from rdflib import Graph, Literal, URIRef +from rdflib.compare import isomorphic + +import diffable_rdf.canonicalize as canonicalize_module +from diffable_rdf import canonicalize_rdf_graph, deterministic_turtle + +EX = "http://example.org/" +TURTLE_FAMILY = ("turtle", "trig", "n3") +OX_FORMAT = { + "turtle": ox.RdfFormat.TURTLE, + "trig": ox.RdfFormat.TRIG, + "n3": ox.RdfFormat.N3, + "xml": ox.RdfFormat.RDF_XML, +} + + +def _assert_reader_fidelity(graph: Graph, rendered: str, output_format: str) -> None: + """Assert both supported readers preserve every term in the graph.""" + reparsed = Graph().parse(data=rendered, format=output_format) + assert isomorphic(reparsed, graph) + expected = set(ox.parse(graph.serialize(format="nt"), format=ox.RdfFormat.N_TRIPLES)) + assert set(ox.parse(rendered, format=OX_FORMAT[output_format])) == expected + + +def _graph_with_term(namespace: str, local: str, position: str, prefix: str) -> Graph: + """Build a graph containing the named term in one RDF term position.""" + graph = Graph(bind_namespaces="none") + graph.bind(prefix, namespace) + term = URIRef(namespace + local) + subject = URIRef(EX + "subject") + predicate = URIRef(EX + "predicate") + object_ = URIRef(EX + "object") + if position == "subject": + subject = term + elif position == "predicate": + predicate = term + elif position == "object": + object_ = term + else: + object_ = Literal("value", datatype=term) + graph.add((subject, predicate, object_)) + return graph + + +@pytest.mark.parametrize("output_format", TURTLE_FAMILY) +@pytest.mark.parametrize("position", ("subject", "predicate", "object", "datatype")) +@pytest.mark.parametrize( + ("prefix", "namespace", "local"), + [ + pytest.param("é", EX + "unicode/", "x.", id="unicode-prefix"), + pytest.param("ex", EX + "comma/", "x,.", id="comma"), + pytest.param("ex", EX + "semicolon/", "x;().", id="punctuation"), + pytest.param("ex", EX + "apostrophe/", "x'.", id="apostrophe"), + pytest.param("ex", EX + "percent/", "x%2C.", id="percent-encoded"), + ], +) +def test_prefixed_names_preserve_every_term_position( + output_format: str, position: str, prefix: str, namespace: str, local: str +) -> None: + """Optional compact names preserve terms in every RDF position.""" + graph = _graph_with_term(namespace, local, position, prefix) + before_triples = set(graph) + before_base = graph.base + before_bindings = tuple(graph.namespaces()) + + rendered = canonicalize_rdf_graph(graph, output_format) + + _assert_reader_fidelity(graph, rendered, output_format) + assert canonicalize_rdf_graph(graph, output_format) == rendered + assert set(graph) == before_triples + assert graph.base == before_base + assert tuple(graph.namespaces()) == before_bindings + + +@pytest.mark.parametrize("output_format", TURTLE_FAMILY) +def test_punctuation_prefixed_names_preserve_every_term(output_format: str) -> None: + """Punctuation in a compact name cannot change a graph term.""" + namespace = EX + "dot/" + graph = _graph_with_term(namespace, "x,.", "subject", "ex") + graph.add((URIRef(namespace + "ordinary"), URIRef(EX + "predicate"), Literal("value"))) + + rendered = canonicalize_rdf_graph(graph, output_format) + + _assert_reader_fidelity(graph, rendered, output_format) + + +@pytest.mark.parametrize("output_format", TURTLE_FAMILY) +def test_overlapping_prefixes_and_aliases_remain_optional(output_format: str) -> None: + """Aliases and overlapping namespaces cannot alter graph terms.""" + graph = Graph(bind_namespaces="none") + graph.bind("short", EX) + graph.bind("long", EX + "overlap/") + graph.bind("alias", EX + "overlap/") + graph.add((URIRef(EX + "overlap/x,."), URIRef(EX + "predicate"), URIRef(EX + "overlap/ordinary"))) + + rendered = canonicalize_rdf_graph(graph, output_format) + + _assert_reader_fidelity(graph, rendered, output_format) + assert canonicalize_rdf_graph(graph, output_format) == rendered + + +@pytest.mark.parametrize("output_format", TURTLE_FAMILY) +def test_curie_looking_literal_content_is_data(output_format: str) -> None: + """Single-line and multiline literal text remains unchanged.""" + namespace = EX + "literal/" + graph = _graph_with_term(namespace, "x'.", "object", "ex") + graph.add((URIRef(EX + "single"), URIRef(EX + "predicate"), Literal(r"ex:x\,\. remains text"))) + graph.add((URIRef(EX + "multi"), URIRef(EX + "predicate"), Literal("line one\nex:x\\. line two"))) + + rendered = canonicalize_rdf_graph(graph, output_format) + reparsed = Graph().parse(data=rendered, format=output_format) + + _assert_reader_fidelity(graph, rendered, output_format) + assert reparsed.value(URIRef(EX + "single"), URIRef(EX + "predicate")) == Literal(r"ex:x\,\. remains text") + assert reparsed.value(URIRef(EX + "multi"), URIRef(EX + "predicate")) == Literal("line one\nex:x\\. line two") + + +def test_an_ordinary_binding_keeps_its_compact_rendering() -> None: + """A first verified rendering retains ordinary caller bindings.""" + graph = _graph_with_term(EX + "ordinary/", "name", "subject", "ex") + + rendered = canonicalize_rdf_graph(graph, "turtle") + + _assert_reader_fidelity(graph, rendered, "turtle") + assert "@prefix ex: ." in rendered + assert "ex:name" in rendered + + +def test_deterministic_turtle_preserves_curie_looking_literal_content() -> None: + """The deterministic Turtle entry point preserves literal lexical forms.""" + graph = Graph(bind_namespaces="none") + graph.bind("ex", EX) + graph.add((URIRef(EX + "subject"), URIRef(EX + "predicate"), Literal(r"ex:x\,\. text"))) + + rendered = deterministic_turtle(graph) + + assert isomorphic(Graph().parse(data=rendered, format="turtle"), graph) + + +@pytest.mark.parametrize( + ("output_format", "ox_format"), + [ + pytest.param("turtle", ox.RdfFormat.TURTLE, id="turtle"), + pytest.param("xml", ox.RdfFormat.RDF_XML, id="xml"), + ], +) +def test_verification_failures_remove_options_in_order( + output_format: str, ox_format: ox.RdfFormat, monkeypatch: pytest.MonkeyPatch +) -> None: + """Each failed verification removes one serialization option.""" + base = "http://example.org/query?x=" + namespace = EX + "dot/" + graph = Graph(base=base, bind_namespaces="none") + graph.bind("ex", namespace) + graph.add((URIRef(namespace + "x,."), URIRef(base + "predicate"), URIRef(base + "object"))) + original_assertion = canonicalize_module._assert_round_trips + original_serialize = canonicalize_module.ox.serialize + validations = 0 + calls: list[dict[str, object]] = [] + + def fail_before_final(*args: object, **kwargs: object) -> None: + nonlocal validations + validations += 1 + if validations < 3: + raise ValueError("verification failed") + original_assertion(*args, **kwargs) + + def record(*args: object, **kwargs: object) -> bytes: + calls.append(kwargs) + return original_serialize(*args, **kwargs) + + monkeypatch.setattr(canonicalize_module, "_assert_round_trips", fail_before_final) + monkeypatch.setattr(canonicalize_module.ox, "serialize", record) + rendered = canonicalize_rdf_graph(graph, output_format) + + _assert_reader_fidelity(graph, rendered, output_format) + assert validations == 3 + assert calls == [ + {"format": ox_format, "prefixes": {"ex": namespace}, "base_iri": base}, + {"format": ox_format, "prefixes": {"ex": namespace}}, + {"format": ox_format}, + ] + + +@pytest.mark.parametrize("output_format", TURTLE_FAMILY) +def test_a_valid_compact_spelling_keeps_a_problematic_binding( + output_format: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """A successful first verification retains the caller binding.""" + namespace = EX + "retained/" + graph = _graph_with_term(namespace, "x,.", "subject", "ex") + rendered = ( + f"@prefix ex: <{namespace}> .\n" + f"<{namespace}x,.> <{EX}predicate> <{EX}object> .\n" + ).encode() + calls: list[dict[str, object]] = [] + + def record(*args: object, **kwargs: object) -> bytes: + calls.append(kwargs) + return rendered + + monkeypatch.setattr(canonicalize_module.ox, "serialize", record) + result = canonicalize_rdf_graph(graph, output_format) + + _assert_reader_fidelity(graph, result, output_format) + assert "@prefix ex:" in result + assert len(calls) == 1 + + +def test_final_prefix_attempt_propagates_validation_failure(monkeypatch: pytest.MonkeyPatch) -> None: + """The final option stage must verify before returning.""" + graph = Graph(base="http://example.org/query?x=", bind_namespaces="none") + graph.bind("ex", EX) + graph.add((URIRef(EX + "subject"), URIRef(EX + "predicate"), Literal("value"))) + validations = 0 + + def fail(*args: object, **kwargs: object) -> None: + nonlocal validations + validations += 1 + raise ValueError("verification failed") + + monkeypatch.setattr(canonicalize_module, "_assert_round_trips", fail) + with pytest.raises(ValueError, match="verification failed"): + canonicalize_rdf_graph(graph, "turtle") + assert validations == 3 + + +def test_retry_options_do_not_affect_later_renderings(monkeypatch: pytest.MonkeyPatch) -> None: + """A retry selects options only for its current serialization.""" + retry_graph = Graph(base="http://example.org/query?x=", bind_namespaces="none") + retry_graph.bind("ex", EX + "ordinary/") + retry_graph.add((URIRef(EX + "ordinary/subject"), URIRef(EX + "predicate"), Literal("value"))) + original = canonicalize_module._assert_round_trips + validations = 0 + + def fail_once(*args: object, **kwargs: object) -> None: + nonlocal validations + validations += 1 + if validations < 3: + raise ValueError("verification failed") + original(*args, **kwargs) + + monkeypatch.setattr(canonicalize_module, "_assert_round_trips", fail_once) + canonicalize_rdf_graph(retry_graph, "turtle") + ordinary = _graph_with_term(EX + "ordinary/", "name", "subject", "ex") + rendered = canonicalize_rdf_graph(ordinary, "turtle") + _assert_reader_fidelity(ordinary, rendered, "turtle") + assert "@prefix ex: ." in rendered