fix: correction validation scoring — joint log-probability at the word boundary#42
Open
hiimdec wants to merge 1 commit into
Open
fix: correction validation scoring — joint log-probability at the word boundary#42hiimdec wants to merge 1 commit into
hiimdec wants to merge 1 commit into
Conversation
…d boundary
The autocorrect lane detected typos but suppressed nearly every fix even in
ideal context (definately after "I " stayed lowModelMargin). Instrumenting
the scorer against a real GGUF found three compounding scoring defects, not a
thresholds problem — which explains the escalating threshold surgery in
ADR-113/114/115:
1. the anchor kept its trailing space, so candidates tokenized standalone
(a split BPE rarely produces after a space token), flooring every score;
2. per-token MEAN log-probability favoured multi-token strings to the point
of ranking the misspelling above its correction (mean: definately -6.56
beats definitely -8.09; joint sums: definitely wins by ~5 nats);
3. the -6.0 absolute floor asked "is this a likely continuation?" — the
wrong question for a correction — so every candidate died there.
Fixes: trailing whitespace moves onto the scored word (ADR-017's analogue);
comparisons use joint sums; the floor and much-better slack collapse into one
advantage-over-original gate (default 0 — any model preference for the user's
own string vetoes, strengthening deliberate-spelling protection); margins
recalibrated to joint nats (2.0, relaxed to 0.5 for dictionary-flagged
edit-distance<=2 top guesses); whitespace-only suffix windows skip the join
probe; open words are corrected only when they are dead-end fragments
(mirroring the in-beam typo guard); the badge strikes only the wrong letters.
Real-model verdict for "I ": definitely conf=0.95 margin~5.4. ADR-116.
Unit tests pin the logged regression cases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The autocorrect lane (ADR-108–115) reliably detects typos but suppresses almost every fix, even in ideal prose context —
definatelytyped after "I " logsSUPPRESS(lowModelMargin). InstrumentingCorrectionValidationScoreragainst a real GGUF showed the thresholds were never the problem; the scoring was, in three compounding ways (likely why ADR-113/114/115 kept lowering thresholds without success):" definitely"(1 token) scored −8.09 mean vs" definately"(2 tokens) −6.56 — the misspelling outscored its own correction. Joint (summed) log-probs are −8.09 vs −13.12; the correction wins by ~5 nats.lowModelMarginwas just the empty-result label.Changes
minimumAdvantageOverOriginal(default 0 — any model preference for the original vetoes, which strengthens deliberate-spelling / proper-noun protection). Dictionary flagging already guarantees the replacement is a real word.minimumMargin2.0, relaxedclearCutMargin0.5 for dictionary-flagged, edit-distance ≤ 2, model-top guesses.Result (real model):
definately→definitelyat confidence 0.95, joint margin ≈ 5.4 (thin context) / 6.0 (rich); verified end-to-end on device and with unit tests pinning the logged regression cases.Logged as ADR-116 (next after 115 on this branch's base). Note: my other open PR #41 also introduces an ADR-116 — happy to renumber whichever you merge second. Independent of #41 (disjoint files apart from the decision log).
🤖 Generated with Claude Code