Add a conformance suite for HermitCrab, with its adequacy argument by johnml1135 · Pull Request #480 · sillsdev/machine

johnml1135 · 2026-08-19T19:22:23Z

Quick summary

Parser maintainers can compare full parse signatures across HermitCrab and compatible engines.
The suite covers 44 fixtures and 536 cases with explicit expectations and auditable coverage evidence.
It leaves HermitCrab's analysis and synthesis algorithms unchanged.

Where to look

conformance/PROTOCOL.md -- engine adapter contract.
conformance/generated/hc-conformance-manifest.v1.json -- fixture paths and hashes.
src/SIL.Machine.Morphology.HermitCrab.Conformance/SemanticCoverage/AnalyzerMetadataInspector.cs -- hash-pinned generator admission remains probe-gated.
src/SIL.Machine.Morphology.HermitCrab.Conformance/MaterializedRunner.cs -- crash fixtures require an ordinary fixture to complete.

Deliberately not included

Counterfactual and coverage-evidence sweeps remain deferred; known oracle gaps are tracked.

Validation

dotnet build and dotnet build -c Release -- succeeded, 0 warnings.
dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --no-build -- 593 passed, 1 skipped (symlink test needs elevation), 0 failed.
dotnet run --no-build -c Release --project src/SIL.Machine.Morphology.HermitCrab.Conformance -- --fixtures conformance -- 43 passed, 0 failed.
... -- --semantic-coverage --repository-root . -- 0 new gaps, 0 stale lines.
python conformance/parity-check.py -- all checks passed.
dotnet csharpier check . -- 868 files, clean.
pwsh -NoProfile -File scripts/comment-hygiene.ps1 -BaseRef origin/master -- clean.

Decisions, and why

PCRE.NET (now on master) ships an IIncrementalGenerator. The compilation-graph loader admits it only by content hash, and a probe runs it and fails closed on any generated output.
An expect_crash fixture passes only if the same engine completes an ordinary fixture in the same run, so a broken adapter command cannot pass as a crash (conformance/PROTOCOL.md).
*.fwdata is marked binary in .gitattributes; its pinned hash depended on core.autocrlf, which is why CI failed a test that passed locally.
Hand-maintained fixture totals were removed from the README and docs; the manifest is the count.
Four fixture grammars keep trailing whitespace: their hashes are pinned in the manifest.

🤖 Generated with Claude Code

This change is

codecov-commenter · 2026-08-20T05:05:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.03%. Comparing base (7109531) to head (f412c25).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #480      +/-   ##
==========================================
- Coverage   74.07%   74.03%   -0.05%     
==========================================
  Files         456      555      +99     
  Lines       38164    55873   +17709     
  Branches     5228     7592    +2364     
==========================================
+ Hits        28270    41363   +13093     
- Misses       8734    12659    +3925     
- Partials     1160     1851     +691

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

johnml1135 · 2026-09-15T17:40:35Z

Added two independently justified conformance fixtures in a20bce1: final-template-partial-discriminators (10 words; #507) and template-category-sharing (6 words; #505). Fresh Release build: zero warnings/errors. Both fixtures pass with memoization on and off, zero skipped; 22 focused manifest/coverage tests pass. Four grammar counterfactuals produce the independently predicted mismatches, documented under conformance/docs. These are grammar-sensitivity controls, not proof that pruning, cache hits, or same-key feature collisions execute. No parser code changed. Generated inventory also restores previously omitted rows for the already-existing Exact-inverse fixture; its valid zudiua loss remains tracked in #504, not relabeled as correct.

johnml1135 · 2026-09-15T18:49:09Z

Added the shared-template unconstrained-suffix conformance fixture in f150e2a. The grammar has a slot-only feature-unconstrained suffix shared by mandatory TV/IV templates, an ordinary N-to-IV verbalizer, four positive rows, and six rejection controls. The companion rationale explains complete forward derivations and loader reachability (not a FieldWorks UI round trip).

Fresh C# harness: all 10 rows pass in both template orders, memoization on/off (40 comparisons; no skips). Focused template/manifest tests: 22 passed, no skips. Regenerated manifest is current at 40 fixtures; coverage/rule reports updated. No Machine parser code changed.

PanGloss main now contains the narrow gate-only alignment (fc357daf), with this fixture mirrored unchanged until its old Machine submodule pin is upgraded. Direct state assertions provide red/green evidence; full-parse fixture checks pass both before and after the change. Issue #505 remains open for collision/correlation and broader merge-equivalence evidence.

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

PCRE.NET 1.6.0 landed on master (a4b2974, the base this branch rebased onto) with a real Roslyn IIncrementalGenerator, PCRE.Analyzers.PcreCallsInterceptorGenerator, verified here by direct PE metadata inspection. SIL.Machine's PackageReference to it flows transitively into every project this branch's compilation-graph loader audits (machine, hc, hc-tool, hc-conformance), so CSharpCommandLineInputParser's existing generator check correctly, but too bluntly, rejected it everywhere as "unsupported-source-generator" -- breaking ~10 tests across CoverageCliAuthorityTests, DotnetMsbuildCompilationGraphLoaderTests, and GrammarCoverageGateTests. d979b662 already hit this same failure while authoring the four-fixture commit and worked around it by skipping semantic-coverage regeneration rather than fixing the loader. AnalyzerMetadataInspector already had a narrow admission path for known .NET SDK reference-pack generators (SdkOwnedSourceGeneratorPendingProbe), which does not verify anything at admission time -- it defers the real check to RoslynCompilationGraph.RunPendingGeneratorProbe, which actually runs the generator against each node's compilation and fails closed if it ever emits a source, diagnostic, or exception. This change extends that same probe-gated admission to a new VettedThirdPartySourceGeneratorPendingProbe disposition, keyed by the analyzer's SHA-256 (not by name or directory, so any future PCRE.NET version bump changes the hash and reverts to the closed-by-default rejection until a maintainer re-verifies). PCRE.NET.Analyzers.dll 1.6.0's hash is pinned after confirming `dotnet build -t:Rebuild --property:EmitCompilerGeneratedFiles=true` emits zero generated files for every audited project. Updates the two tests that pin exact generator counts per node (DotnetMsbuildCompilationGraphLoaderTests, RoslynCompilationGraphTests) to the now-measured totals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…, simultaneous mode, iterative epenthesis edge-cases/discontinuous-morph-environment: Allomorph.IsWordValid checks an allomorph's environment once per contiguous piece of the morph (an infix splits the root). zuexbu/zuixdu hold the environment at the merged span's edges but fail at an inner piece boundary, so a merged-span port accepts what hc.dll rejects. edge-cases/simultaneous-feeding and simultaneous-feeding-control-iterative: byte-identical grammars except multipleApplicationOrder; gigugu parses only under simultaneous application, gigugi only under iterative. edge-cases/iterative-epenthesis-cascade: two empty-LHS iterative rules in one stratum where the first rule's insertion creates the second rule's site; the one-node cursor accepts uotaa/uotaata and rejects uotatata, which a collect-all implementation produces. Regenerated: rule-interaction-pairs.tsv, interface-inventory.tsv, interaction-chains.tsv, generated/hc-conformance-manifest.v1.json. The semantic-coverage gate could not run on the authoring machine (its Roslyn analyzer loader rejects PCRE.NET.Analyzers); --counterfactual and --coverage-traceability were not regenerated in this commit. Each fixture's positive parse now carries an "exercises:" tag (infixation for discontinuous-morph-environment; the matching RewriteRule Simultaneous/ Iterative construct for the other three), so ConformanceFixtureGateTests. EveryFixtureContributesAtLeastOneCoverageRow's per-language coverage.csv check sees a row for all four -- a review found all four previously silent there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016AzbiMyms1YL83LbdRNJwx Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

suffixing-extension-slot-ordering: post-nasal voicing of the second copy (pambam) and its unmutated negative. metathesis-phase-isolation: h-deletion in the second copy after full and CV reduplication, a fixed-vowel CV reduplicant (ModifyFromInput), and both combined. Expected parses are hand-derived forward and pass on the unchanged engine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Regenerates every generated artifact (coverage.csv, rules.csv, the JSON manifest, interface-inventory.tsv, interaction-chains.tsv, rule-interaction-pairs.tsv, interface-witness.tsv, construct-claim-corroboration.tsv, grammar-coverage-ledger.tsv, fold-in-candidates.tsv, dataflow-obligations.tsv, engine-gate-inventory.tsv, gate-obligations.tsv, and the evidence cards) from the full 44-fixture corpus, via each artifact's own --write-* flag (see conformance/README.md). None of this branch's earlier commits past the first ever regenerated these together against the final fixture set, so several were stale independently of each other. Also updates seven pinned tests whose exact counts drifted from 36 to 44 fixtures across four later commits in this branch (final-template-partial-discriminators, template-category-sharing, shared-template-unconstrained-suffix, discontinuous-morph-environment, iterative-epenthesis-cascade, simultaneous-feeding, simultaneous-feeding-control-iterative) without the pin ever being bumped: ConstructClaimCorroborationTests, CoverageGapRatchetTests, GrammarCoverageLedgerTests, InterfaceWitnessLedgerTests, OrderingGeneratorTests, EngineGateInventoryLedgerTests, and GateObligationLedgerTests (renaming two of the latter's tests for the counts they now pin: EvidencedRowsAreExactlyTheseSixteenAndEachCarriesAFixtureAndWord, SixGatesHaveBothArmsEvidenced). Each new value is checked against the manifest's own fixture list, not just accepted -- see each test's own updated comment for the accounting, and CoverageGapRatchetTests' in particular for the two Ordering gaps (edge-cases/chained-output-feature-override- loss) that predate this branch and are not fixture growth. conformance/README.md and conformance/docs/what-it-is.md no longer carry hand-copied fixture/case totals that had already drifted from the manifest twice; they now point at the manifest and its own --generate-manifest/--check-manifest flags instead. Not regenerated: semantic-coverage-counterfactuals.tsv and semantic-coverage-evidence.tsv (the --write-counterfactual and --write-coverage-evidence artifacts). d979b662 already deferred these on this branch; the full test suite is green without touching them, and a fresh --write-counterfactual sweep (one reparse per fixture per neutralized surface) did not complete within this session's time budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

johnml1135 requested a review from ddaspit August 19, 2026 19:27

johnml1135 force-pushed the integrate-conformance-framework branch from d826869 to 8f9dcea Compare August 19, 2026 19:46

johnml1135 force-pushed the integrate-conformance-framework branch 2 times, most recently from d0b5448 to 9dfb2f6 Compare August 20, 2026 09:10

johnml1135 mentioned this pull request Aug 20, 2026

Add HermitCrab grammar health diagnostics #475

Open

johnml1135 force-pushed the integrate-conformance-framework branch 2 times, most recently from de95458 to 4823a05 Compare September 4, 2026 00:15

johnml1135 force-pushed the integrate-conformance-framework branch from 4823a05 to 3128301 Compare September 15, 2026 12:29

johnml1135 mentioned this pull request Sep 15, 2026

HermitCrab: track exact analysis feature inversion and chained override parse loss #504

Open

This was referenced Sep 15, 2026

HermitCrab: establish discriminating conformance coverage for analysis-state and template/slot merges #505

Open

HermitCrab: complete cross-engine correctness controls for final-template pruning #507

Open

This was referenced Sep 24, 2026

Prune reduplication matches whose copies cannot agree #519

Open

POS-scoped deletion blocks metathesis analysis of other words (ParseWord incomplete) #520

Open

johnml1135 force-pushed the integrate-conformance-framework branch from 93f470c to 2933fcc Compare September 26, 2026 02:55

johnml1135 and others added 8 commits September 26, 2026 00:04

Add final-template and template-sharing conformance controls

e2b51cf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test: cover shared unconstrained template suffix

bd82bae

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

johnml1135 force-pushed the integrate-conformance-framework branch from 2933fcc to f412c25 Compare September 26, 2026 04:15

Navigation Menu

Uh oh!

Add a conformance suite for HermitCrab, with its adequacy argument - #480

Add a conformance suite for HermitCrab, with its adequacy argument#480
johnml1135 wants to merge 8 commits into
masterfrom
integrate-conformance-framework

johnml1135 commented Aug 19, 2026 •

edited by ddaspit

Loading

Uh oh!

codecov-commenter commented Aug 20, 2026 •

edited

Loading

Uh oh!

johnml1135 commented Sep 15, 2026

Uh oh!

johnml1135 commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

johnml1135 commented Aug 19, 2026 • edited by ddaspit Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Quick summary

Where to look

Deliberately not included

Validation

Uh oh!

codecov-commenter commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

johnml1135 commented Sep 15, 2026

Uh oh!

johnml1135 commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

johnml1135 commented Aug 19, 2026 •

edited by ddaspit

Loading

codecov-commenter commented Aug 20, 2026 •

edited

Loading