feat(queryexec): Differ, a keyed value comparison of two projected tables

AGENTS.md

@@ -192,6 +192,26 @@ Then update `docs/project/spec-compliance.md` mapping: semantic rule → impleme
 
 ---
 
+### 6.1 Bumping the extension-library pin
+
+The vendored `OpenSysML Libraries` (`scripts/extension-libraries-pin.sh`, synced by
+`scripts/sync-extension-libraries.sh`) are an input to gates far from `internal/workspace/libs`, and
+the full CI tier is expensive, so run them locally before the PR, every time the pin moves:
+
+1. `go generate ./internal/workspace/libs` — `stdlib.snapshot` embeds the libraries; commit it.
+2. `make extension-libraries-pin-check` — Java-free, about a minute: the sync check, the snapshot,
+   `tests/hygiene TestEngineContractCoversEveryBoundName` (a new binding in
+   `internal/ir/queryplan/compiler.go` needs an `engine-contract.json` entry upstream, in the same
+   library PR as the declaration), the provenance guards of both pilot-differential baselines, and
+   the REPL/query tests (a new library name enters the REPL's "did you mean" suggestions and can
+   collide with a fixture).
+3. If a provenance guard fails, the baselines record the previous library digest: run
+   `go run -C tools ./cmd/pilot-diff` and `./cmd/pilot-apijson-diff` (the pilot toolchain is in the
+   blueprint), adjudicate every moved entry against the committed baseline, write the adjudication
+   into `docs/project/pilot-differential.md` / `api-json-differential.md`, then re-record with
+   `-update` and run `make docs-counts`. A library change that adds declarations always moves
+   `pilotLibraryElements`; anything else moving is an implementation difference to explain.
+
 ## 7. Code Style & Commits
 
 - **Formatting:** `gofmt` is mandatory (CI-enforced). Follow *Effective Go*. Document exported types/functions.

Makefile

@@ -1,4 +1,4 @@
-.PHONY: all build docker vulncheck sca-tools audit audit-npm audit-python audit-rust audit-java extension-libraries-check extension-libraries-sync build-sysml build-re build-wasm-re build-lsp build-grpc build-mcp build-engine build-core build-syntax build-sysml-wasm build-release-wasm build-wasm build-wasm-wasip1 build-wasm-js wasm-check static-check windows-versioninfo-check man man-check install-tree pgo-profile conformance conformance-pkg conformance-mcp conformance-rust conformance-julia conformance-matlab test test-shard coverage coverage-shard coverage-merge lint clean install help ontology-table ontology-table-check metaclass-graph metaclass-graph-check python-metamodel python-metamodel-check fuml-expected python-test python-coverage scripts-coverage node-coverage vscode-coverage jupyterlab-coverage mcp-npm-coverage python-install jupyter-kernel-install jupyter-kernel-test proto proto-buf python-proto proto-ts proto-rust proto-lint proto-breaking jupyterlab-build jupyterlab-test jupyterlab-install vscode-grammar vscode-build vscode-package docs docs-install docs-serve docs-engine-assets docs-landing-assets docs-counts diagnostics-catalogue diagnostics-check docs-check changelog-check changelog-render self-model mcp-registry-check
+.PHONY: all build docker vulncheck sca-tools audit audit-npm audit-python audit-rust audit-java extension-libraries-check extension-libraries-sync extension-libraries-pin-check build-sysml build-re build-wasm-re build-lsp build-grpc build-mcp build-engine build-core build-syntax build-sysml-wasm build-release-wasm build-wasm build-wasm-wasip1 build-wasm-js wasm-check static-check windows-versioninfo-check man man-check install-tree pgo-profile conformance conformance-pkg conformance-mcp conformance-rust conformance-julia conformance-matlab test test-shard coverage coverage-shard coverage-merge lint clean install help ontology-table ontology-table-check metaclass-graph metaclass-graph-check python-metamodel python-metamodel-check fuml-expected python-test python-coverage scripts-coverage node-coverage vscode-coverage jupyterlab-coverage mcp-npm-coverage python-install jupyter-kernel-install jupyter-kernel-test proto proto-buf python-proto proto-ts proto-rust proto-lint proto-breaking jupyterlab-build jupyterlab-test jupyterlab-install vscode-grammar vscode-build vscode-package docs docs-install docs-serve docs-engine-assets docs-landing-assets docs-counts diagnostics-catalogue diagnostics-check docs-check changelog-check changelog-render self-model mcp-registry-check
 
 # Version information
 # Only release tags describe a build; the moving `nightly` tag is not a version.
@@ -451,6 +451,14 @@ extension-libraries-check: ## Check the vendored extension libraries match their
 extension-libraries-sync: ## Re-vendor the extension libraries from the pinned upstream commit
 	./scripts/sync-extension-libraries.sh
 
+extension-libraries-pin-check: ## After moving the extension-library pin: every Java-free gate the full tier runs over the vendored libraries
+	./scripts/sync-extension-libraries.sh --check
+	go run -C $(TOOLS_DIR) ./gen/snapshot -check
+	go test -count=1 ./tests/hygiene -run 'TestEngineContract'
+	go test -C $(TOOLS_DIR) -count=1 ./referee/diff ./referee/apijson -run 'TestCommittedBaselineStatesThisRepositorysProvenance|TestProvenance'
+	go test -count=1 ./internal/frontend/repl ./internal/workspace/libs ./internal/ir/queryplan ./internal/doc/queryexec
+	@echo "✓ the vendored extension libraries pass every pin gate"
+
 stdlib-snapshot: ## Regenerate the embedded snapshot of the bundled library after editing $(LIBS_DIR)/stdlib
 	go generate ./$(LIBS_DIR)
 

changes/unreleased/differ.added.md

@@ -0,0 +1 @@
+- `DocumentQueries::Differ(left, right, key, column)`: a keyed value comparison of two projected tables — one row (`key`, `left`, `right`) per key value on both sides whose `column` cells differ. A side that projects no such column is the path error `unknown-column`; a key value on two rows of one side is the model error `duplicate-key`, every finding kept.

cmd/sysml-grpc/access_test.go

@@ -268,9 +268,7 @@ func TestGRPCRobustnessAuthStartup(t *testing.T) {
 	t.Run("health binds the same host", func(t *testing.T) {
 		s := startService(t, "-port", "0", "-health-port", freePort(t))
 		s.address(30 * time.Second)
-		if logs := s.logs.String(); !strings.Contains(logs, "Health check server listening\" addr=localhost:") {
-			t.Errorf("the health listener did not bind loopback:\n%s", logs)
-		}
+		s.waitLog("Health check server listening\" addr=localhost:", 30*time.Second)
 	})
 	t.Run("non-loopback without a token is refused", func(t *testing.T) {
 		s := startService(t, "-host", "0.0.0.0", "-port", "0", "-health-port", "0")

cmd/sysml-grpc/lifecycle_test.go

@@ -181,6 +181,19 @@ func (s *service) address(timeout time.Duration) string {
 	}
 }
 
+// waitLog waits until the service has logged a line containing want: the health
+// listener announces itself from its own goroutine, after the main address.
+func (s *service) waitLog(want string, timeout time.Duration) {
+	s.t.Helper()
+	deadline := time.Now().Add(timeout)
+	for !strings.Contains(s.logs.String(), want) {
+		if time.Now().After(deadline) {
+			s.t.Fatalf("the service did not log %q within %s\nlogs: %s", want, timeout, s.logs.String())
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+}
+
 // waitStatus waits for the process to end and returns its exit status. A
 // process still running when the timeout expires is a leak, and a failure.
 func (s *service) waitStatus(timeout time.Duration) int {

docs/internals/design/query-outcomes.md

@@ -28,6 +28,7 @@ to `∅` is the ambiguity QVTS exists to remove.
 | `Named` | the element | `unknown-element`: the longest prefix that resolves, the segment it lacks, the nearest spellings; an alias that denotes nothing | `ambiguous-element`: the elements, by qualified name; the finding names the owning namespace |
 | `WhereType`, `WhereMetadata` | the selected rows | `unknown-classification` | — |
 | `RelatedElements` | the neighbours, possibly none | `unknown-relationship` | — |
+| `Differ` | the `(key, left, right)` rows whose column differs, possibly none | `unknown-column`: the side (`Parameter`) that projects no such column (`Property`), its columns (`Candidates`) | `duplicate-key`: one `duplicateKey` finding per row repeating a key value on one side, `multiplicity` where a key cell is not single-valued |
 | `Project`, `WhereFeature` on a feature the element lacks | an empty cell, with `tolerate = true` | `missing-feature`: the element (`Target`), the feature (`Property`), the features it does have (`Candidates`); `unknown-property` when no row carries it | — |
 | `Project`, `WhereFeature` on a metadata path (`Tag::key`) the element lacks | *today an empty cell* | *to do:* decide whether an absent annotation is a path error | — |
 | `Project` on a `[1]` feature with two values | *today the first value* | — | *to do:* `multiplicity` |

docs/manual/query-cookbook.md

@@ -1354,6 +1354,40 @@ $ sysml cookbook.sysml -run-query "Cookbook::VerifiedButUnsatisfied root=Cookboo
 to be unprojected or to project the same columns, and is a typed error
 otherwise, since its rows share one table.
 
+### Keyed comparison: `Differ`
+
+`Except` and `Union` identify rows, so a value that changed on an element present on
+both sides is invisible to them. `Differ(left, right, key, column)` compares two projected
+tables by a key column: one row per key value present on both sides whose `column` cells
+differ, with the columns `key`, `left` and `right` — the key, then the cell of each side.
+Cells compare as sequences under KerML `==` (elements by identity, numbers by value), so a
+many-valued cell differs when its values or their order do; keys match under the same
+equality, so an integer `1` and a real `1.0` are one key. Keys on one side only are not
+rows here; `Except` reports those. A `Tree`-arranged left side keeps its depths.
+
+```sysml
+calc def PartMasses :> Query {
+	in root : Element;
+	Project(
+		source = WhereType(source = OwnedElements(source = root), type = "PartUsage"),
+		properties = ("name", "mass")
+	)
+}
+
+calc def ChangedMasses :> Query {
+	in before : Element;
+	in after : Element;
+	Differ(left = PartMasses(root = before), right = PartMasses(root = after), key = "name", column = "mass")
+}
+```
+
+Both sides must project `key` and `column`; a side that does not is the typed error
+`unknown-column`, a path error at `Differ(key, column)`. A key named `left` or `right`
+would collide with the result's side columns and is an `invalid-argument`. A key value carried by two rows
+of one side, or a key cell that is not single-valued, is the model error `duplicate-key`
+(`duplicateKey` / `multiplicity` findings, one per offending row, every row kept) rather
+than a silent choice between them.
+
 ## Requirement hierarchy
 
 A requirement tree is the requirement definitions and usages under a root —

docs/project/api-json-differential.md

@@ -242,9 +242,18 @@ The pilot's export failed on 59 files (`pilotFailed`: its own export throws on a
 | `examples/pilot-corpora/kerml-examples` | 58 | 6399 | 1166 | 1842 | 73725 | 2504 | 63918 | 5784 |
 | `examples/spec-examples` | 511 | 7074 | 990 | 2656 | 76404 | 1567 | 74068 | 5295 |
 | `tests/testdata` | 40 | 1460 | 175 | 511 | 16445 | 428 | 15857 | 1042 |
-| `examples` | 53 | 26937 | 6046 | 15045 | 332702 | 8290 | 322123 | 19997 |
+| `examples` | 53 | 26937 | 6046 | 15045 | 332672 | 8320 | 322123 | 19997 |
 | `tools/referee/diff/testdata` | 4 | 53 | 0 | 0 | 555 | 1 | 454 | 41 |
-| `docs/manual/examples` | 11 | 7199 | 920 | 2110 | 88779 | 2034 | 89601 | 4816 |
+| `docs/manual/examples` | 11 | 7199 | 920 | 2110 | 88741 | 2072 | 89601 | 4816 |
+
+**Defaulted parameters of an invocation** (re-recorded with the extension libraries at
+`DocumentQueries::WhereType(…, exact = false)` and `Differ`): the pilot's `InvocationExpression.parameter`
+and `.input` list every parameter of the called calculation, a defaulted one the call does not bind
+included, while this repository's writer lists the bound ones — so the 34 `WhereType` calls of
+`examples` and `docs/manual/examples` moved from present-equal to present-different on both
+properties (`InvocationExpression.input` 368 → 334 equal, `.parameter` 382 → 348) when the library
+gained the parameter. The further pilot-library elements (198 for `exact`, 810 for `Differ`) are the
+new declarations themselves. No other row moved. Listing the defaulted parameters is a writer follow-up.
 
 The scorecard is mostly **ours-missing**: this repository's writer emits a subset of the metamodel's properties, and the referee now measures which, per metaclass. The full ranked scorecard with one concrete example per differing row is `build/pilot-apijson-diff/pilot-apijson-diff.txt` after a run; the baseline keeps every row's seven counts.