Add allMatches to Traverse function to improve performance

[jtmaxwell3]

I discovered when parsing wemulujaʼjawype in the Aweti project that a single call to Traverse in DeterministicFsaTraversalMethod could take 4 seconds. It was processing more than 200,000 traversals, even though the input only had 47 characters and the fsa only had 4 states. This was because the DeterministicFsaTraversalInstance data structure included the registers, which encoded the match to that point. Processing 200,000 traversals was especially annoying since the caller only wanted one match.

The registers only record the match, they don't filter the traversal in any way. So it should be possible to traverse in two passes, where the first pass finds traversals that are acceptable without recording the registers and the second pass extracts the registers for each of the successful traversals. But it turns out that the code path that was causing the performance issue only takes the first match. So I optimized for the case when the caller only wanted one match. This can be done in one pass.

To fix the performance issue, I added an allMatches parameter to Traverse, and changed the code to ignore traversals that arrived at a previously processed <State, AnnotationIndex> tuple. Since the only difference between the new traversal and the previously processed traversal is in the registers, we can just ignore the new traversal. This means that in the worst case it only takes O(N * |States|) to find a traversal. I tried returning immediately when I found the first match, but that produces the shortest match and many of the MatcherTests failed. Letting it run to the end finds matches of all lengths. Otherwise, the choice of match doesn't seem to matter.

With this change in place, parsing wemulujaʼjawype went from 83 seconds to 3 seconds and parsing 185 Aweti words went from 4 minutes to 1 minute.

---

This change is 

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.40376% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.15%. Comparing base (a4b2974) to head (8a51794).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
.../FiniteState/NondeterministicFsaTraversalMethod.cs	42.50%	23 Missing ⚠️
src/SIL.Machine/FiniteState/TraversalMethodBase.cs	85.06%	18 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #511      +/-   ##
==========================================
+ Coverage   74.09%   74.15%   +0.05%     
==========================================
  Files         456      456              
  Lines       38107    38307     +200     
  Branches     5221     5268      +47     
==========================================
+ Hits        28237    28406     +169     
- Misses       8712     8740      +28     
- Partials     1158     1161       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnml1135]

The optimization is real and worth having, but the (State, AnnotationIndex) key drops traversal state that filters later arcs, so Match() returns different answers than master — including outright lost matches.

The mechanism you found reproduces, and is worse than you reported. It isn't the state count: TraversalMethodBase.Advance (src/SIL.Machine/FiniteState/TraversalMethodBase.cs:275-292) forks an instance for every Optional annotation at the next offset, so instances grow exponentially in the number of optional annotations and independently of |States|. On a synthetic 2-state FSA with 20 optional annotations I measured 2,097,150 instances popped (~2^(k+1)); at k≥24 it blows past a 5,000,000 budget. Your fix bounds it at ≤ N·|States| — measured ratio 0.5–1.0×, turning never-returns cases into instant. That part holds up completely.

The problem. A differential fuzz — 20,000 random pattern/input pairs, fixed seed, identical generator compiled against both master and this branch, comparing success, range, every group range and every variable binding — found:

cases=20000  divergences=112  allMatchesFirstDiff=0

AllMatches().First() is identical in all 20,000 cases, which pins the divergence to exactly the allMatches=false path this PR changes. Two classes:

1. Variable bindings (108 cases). TraversalInstance.VariableBindings is not just a record of the match — CheckInputMatch → Input.Matches unifies against it, so which branch survives decides which later arcs can match. Matcher.Compile (src/SIL.Machine/Matching/Matcher.cs:89) only determinizes when a pattern has no variables, so every variable-bearing pattern — i.e. every alpha-variable phonological rule — runs NondeterministicFsaTraversalMethod, where the new key is blindest. Minimal case:

pattern: high=$v0+      (anchored both ends, 5 annotations)
master : success=True;  range=[0,5); v0=-
this PR: success=False; range=<null>; v0=<unbound>

2. Capture registers (4 cases), on the deterministic path. All four are variable-free, verified to compile with IsDeterministic=True, and all have alternation plus capture groups. They don't merely change captures — they shorten the match range:

pattern: (g0(back=back+)|g1(high=high+back=back+))
master : range=[0,4); g1=[0,4)
this PR: range=[0,1); g0=[0,1)

So "deterministic method only" would not be a sufficient guard; group-bearing patterns diverge there too.

b348f48f does not affect this: re-running the same fuzz against da352c7a produced a byte-identical candidate output file (same SHA-256), same 112 divergences, same gate hit counts. That commit fixes instance-release bookkeeping after the skip decision, not the key itself. Both committed suites stay green on this branch (826/0/3 and 98/0/0), so the existing tests cannot see this.

A narrowing that keeps the win. I prototyped and tested one:

// DeterministicFsaTraversalMethod
bool dedupeByState = !allMatches && Fst.GroupNames.Count() <= 1;

with NondeterministicFsaTraversalMethod reverted to master (keeping the allMatches parameter only to satisfy ITraversalMethod). Results: fuzz divergences 112 → 0; the group-free microbenchmark sweep is cell-for-cell identical to this PR across all 18 cells, so the speedup is fully preserved where patterns have no capture groups; grouped patterns fall back to master's behaviour exactly; suites unchanged. (GroupNames.Count() should be hoisted out of Traverse — it is O(1) via the ICollection fast path but is evaluated once per call.)

Where this could go further. The rewrite-rule environment matchers look like the best target. RewriteRuleSpec (src/SIL.Machine.Morphology.HermitCrab/PhonologicalRules/RewriteRuleSpec.cs:83-107) reads only leftEnvMatch.Success and leftEnvMatch.VariableBindings from those matches — never the range, never a group. So for environment matching the registers are pure overhead, exactly as your description says, and the sound dedup key there is (State, AnnotationIndex, VariableBindings), with bindings droppable entirely when IgnoreVariables is set. That would extend the bound to the variable-bearing patterns this narrowing has to give up, without any capture-reconstruction pass.

One caveat on the measurements. I could not reproduce the 83s → 3s end to end. On the Aweti grammar I have (aweti-hc.xml re-exported 2026-09-16 from Jun-26 data), that word does not complete on either build — both grow to a 40 GB heap cap in ~5 minutes. Instrumenting it showed zero DeterministicFsaTraversalMethod calls out of 7.3M, i.e. the run dies during analysis and never reaches synthesis where that method is used, so we are clearly not parsing the same grammar you were. Across five grammars (Sena, Amharic, Indonesian, Mbugwe, Aweti; sampled words, parity clean) I measured 0.90×–1.01×. If you can share the exact grammar export you measured, I can re-run the comparison on it.

Happy to open a PR with the narrowed gate, the differential fuzz harness, and a regression test for the lost-match case if that is useful. Separately, the fuzz also found 19 cases where Match() throws NullReferenceException on master (e.g. back=$v0*, anchored to start) — pre-existing and unrelated to this PR; I will file that on its own.

🤖 Analysis performed with Claude Code

[johnml1135]

Rebuilt on the stack you named — master + filter-final-templates-in-analysis + change-add-to-priority-union — and the Aweti word still does not complete here. But the attempt turned up something you'll want to know about #491 independently.

All five of our grammars are partial, so #491's prune is inert by default on every one of them. Morpher.IsPartial is true for each, so AlwaysEnforceFinalTemplates is the only way the prune engages:

grammar	partial morphemes
Aweti	3 (2 stems, 1 affix)
Sena	27
Mbugwe	1
Amharic	1
Indonesian	4

That is presumably not true of whatever you measured on, and it would explain why the stack did nothing for us.

Forcing the guard on does change the behaviour, just not enough. With AlwaysEnforceFinalTemplates = true, the failure mode flips from memory-bound to time-bound: without it the word hits a 40 GB heap cap in ~250 s; with it both arms plateau around 16.7 GB and are still running at 480 s. Real, reproducible, and in the direction you would want — but the word does not finish either way, so there is still at least one more factor.

Measurements, all on Aweti source-list index 182, maxDegreeOfParallelism: 1, 40 GB cap:

arm	guard	result
master + #491 + #494	off	killed, 248 s, 40.17 GB
+ #511 (da352c7)	off	killed, 248 s, 40.16 GB
+ narrowed #511	off	killed, 238 s, 40.08 GB
master + #491 + #494	on	killed at 480 s, 16.64 GB
+ #511	on	killed at 480 s, 16.73 GB

None produced a single parse, so there is nothing to compare for either speed or parity, and I can't yet tell you whether the narrowed gate would cost you your win.

Three things would let me reproduce it rather than keep guessing:

1. The exact aweti-hc.xml you measured on. Mine was re-exported 2026-09-16 from the Jun-26 .fwdata; if yours came out of FLEx earlier or with different export settings it may be a materially different grammar. This is my main suspicion.
2. Whether AlwaysEnforceFinalTemplates was set, given the table above.
3. How you invoked the parse — through FLEx's parser, or Morpher.ParseWord directly, and at what parallelism.

Worth noting for its own sake: when I instrumented the word on plain master, DeterministicFsaTraversalMethod was never invoked at all — 7.3M traversal calls, all nondeterministic. The run dies in analysis and never reaches synthesis, which is where the deterministic method gets used. So on my export I am not even reaching the code path your PR optimizes, which is consistent with everything above.

🤖 Analysis performed with Claude Code