feat(cli)!: return the canonical diff document from --format json (#693)

[willhea]

Closes #693. Part of #691.

What changed

./diff_bill.py compare --format json returned the engine's internal diff dictionary
while POST /api/compare?output=json returned the canonical contract. Both now route
through compare/xml.py and return the same document. ./diff_pdf.py gains the same
--format flag, which it had no JSON output to conflict with.

compare/xml.py gains compare_xml_trees, the JSON sibling of the existing
compare_xml_trees_html. The internal shape stays reachable as a library call
(bill_diff_to_dict is unchanged and keeps its pipeline caller), so no --format internal escape hatch was needed to preserve it.

Breaking, on purpose. A caller parsing --format json gets a different document,
and --include-unchanged is now a usage error rather than a flag. Consumers were
checked in the issue: nothing in this repository read the shape except
scripts/compare_differs.py, which is updated here, and BillTrax vendors the engine
and imports it as a library rather than running the command.

Flags

Flag	Before	After
--include-unchanged	Set include_unchanged on both formats	Removed
--financial	Filtered on both formats, and also enriched on --format json	Filters only
--filter	Help named match_path, an internal field	Help describes what it matches
-o/--output	"Output JSON file", though the default has been html since --format was added	"Output file"

--include-unchanged goes because the canonical document cannot represent an unchanged
node: xml_diff_to_canonical drops every unchanged entry. Re-measured here on
tests/corpus/118-hr-8752, driving the library path the flag fed: it moved
summary.unchanged from 0 to 286 while the document's changes array stayed at 39
entries either way. The count referred to 286 entries the document does not contain.

--financial had two meanings chosen by a sibling flag (#694): --format json passed
financial=args.financial, so the flag filtered and enriched, while --format html
filtered on the flag with enrichment hardcoded on. Enrichment is unconditional in
compare/xml.py, and the canonical document has carried no money on a change since
#671, so one meaning is left.

--filter is worth a second look. It matches the normalized path the engine pairs
nodes on, which is not the path the document publishes: that one keeps the original
casing and leads with the division (["TITLE I—DEPARTMENT OF DEFENSE", "Military construction, army"] against ["department of defense", "military construction, army"]). The help now describes the behaviour instead of naming either field. The gap
itself is #689's shape and is left alone here.

Verification

tests/test_cli_api_parity.py is finding 9 of the parity audit, which found that
nothing ran one input through both surfaces and compared. It runs one committed fixture
pair through the command and through the endpoint and requires the same canonical
document, in both XML and PDF, then validates the command's output against
schema/canonical-diff.schema.json.

The comparison is of the parsed documents, not of the bytes. An earlier revision of this
branch asserted cli_text == json.dumps(response.json(), indent=2) and called that byte
identity; it was not, because it re-serialized the parsed response with the command's own
formatter, so the only thing it could catch beyond document equality was a difference in
key order. The two surfaces genuinely serialize differently, measured on
tests/corpus/118-hr-8752:

CLI  551,433 bytes  b'{\n  "schema_version": "3.0",\n  "generator": {\n    "name": "d'
API  380,599 bytes  b'{"schema_version":"3.0","generator":{"name":"deltatrack","ve'
raw bytes equal: False        parsed documents equal: True

Byte identity is also not the invariant worth holding. Getting it would mean indenting
the HTTP response to match a file on disk, roughly 45% more payload for every API caller,
to prove something the contract does not ask for: schema/canonical-diff.md specifies a
document, and #691 asks the surfaces for the same answer rather than the same whitespace.
The module docstring records that measurement so the next reader does not re-add a byte
comparison that would fail on formatting while saying nothing about agreement.

Both directions were proven rather than assumed:

• Reverting the routing in cmd_compare turns all three XML legs red (parity, the
  filename-scope test, and schema validation). Re-run after the assertion above was
  removed, to confirm it had contributed nothing to the negative control.
• Dropping the label derivation from render_pdf_diff_json turns the PDF parity leg
  red.
• Cutting filter_text= and financial_only= out of cmd_compare turns all six
  flag-plumbing gates red, across test_diff_bill.py and test_financial_diff.py.

The gate's input is copied under stems carrying no <n>_ ordinal, because the two
surfaces derive version identity from a filename by different algorithms
(label_from_stem strips the prefix and reads the ordinal;
web/app.py::_label_from_filename does neither). That divergence is #692, not this
issue, and a second test holds it to the versions key alone on the committed corpus
stems, so the exclusion is one named key wide rather than an unbounded escape.

Gates run: full suite green (uv run pytest), ruff check, ruff format --check. The
new module is named in ci.yml's remaining-slow step, which
test_every_slow_module_is_run_by_a_workflow requires and which caught its absence.

scripts/render_examples.py regenerates the four committed example reports
byte-identically, confirming the HTML path did not move.

Not done, and why

summary.unchanged. The issue raised it as a smaller question worth answering here.
It is left alone, because it is outside the scope this branch was given and because
changing what every document emits is a contract decision with its own baselines to
update. Two measurements are worth having before that decision is made, and this change
sharpens both:

1. The two pipelines do not agree on the summary's key set at all. On the same fixture
   pair the XML document carries {"added": 18, "removed": 2, "modified": 18, "unchanged": 0, "moved": 1} and the PDF document carries {"modified": 18, "removed": 2, "moved": 1, "added": 16}. The XML side always emits five keys; the
   PDF side emits a Counter over the kinds actually present, so it omits zeros and has
   no unchanged kind to emit. So this is a pipeline-neutrality break in a contract
   whose whole point is pipeline neutrality, not only an undocumented key.
2. After this change, unchanged is provably constant at 0 on every document any
   shipped entry point produces, since nothing can set include_unchanged through
   compare/xml.py any more. That makes "stop emitting it" strictly stronger than
   "document it", which was not true when the issue posed the either/or.

Filed as #706 (the diff summary reports different change categories for the same bill
depending on whether it was compared from XML or PDF), with both measurements and the
three options costed, rather than decided in passing here.

The intermediate dict in compare/xml.py stays, per #698, which is sequenced after
#653.

Note for review

include_unchanged was also removed from compare_xml_trees_html and
_build_from_trees, not only from the CLI. The CLI was its only caller, and what it did
there was produce a document whose summary.unchanged disagreed with its own changes
array. filter_diff keeps its parameter: the internal shape is allowed to carry
unchanged nodes, and the canonical adapter is the boundary where they stop.

🤖 Generated with Claude Code