Skip to content

fix: correction validation scoring — joint log-probability at the word boundary#42

Open
hiimdec wants to merge 1 commit into
johnbean393:mainfrom
hiimdec:fix/correction-scoring
Open

fix: correction validation scoring — joint log-probability at the word boundary#42
hiimdec wants to merge 1 commit into
johnbean393:mainfrom
hiimdec:fix/correction-scoring

Conversation

@hiimdec

@hiimdec hiimdec commented Jul 13, 2026

Copy link
Copy Markdown

The autocorrect lane (ADR-108–115) reliably detects typos but suppresses almost every fix, even in ideal prose context — definately typed after "I " logs SUPPRESS(lowModelMargin). Instrumenting CorrectionValidationScorer against a real GGUF showed the thresholds were never the problem; the scoring was, in three compounding ways (likely why ADR-113/114/115 kept lowering thresholds without success):

  1. Trailing-space anchor — the anchor kept the space just typed, so candidate words tokenized standalone (a split BPE vocabularies essentially never produce after a space token), flooring every candidate score.
  2. Mean vs joint log-probability — per-token mean scoring systematically favours multi-token strings (their trailing pieces are near-deterministic). Measured on the model: " definitely" (1 token) scored −8.09 mean vs " definately" (2 tokens) −6.56 — the misspelling outscored its own correction. Joint (summed) log-probs are −8.09 vs −13.12; the correction wins by ~5 nats.
  3. Wrong absolute floor — the −6.0 mean floor effectively asked "is this a likely continuation?", the wrong question for a correction; every candidate died there and lowModelMargin was just the empty-result label.

Changes

  • Move the anchor's trailing whitespace onto the scored word (the correction-lane analogue of ADR-017's caret-boundary sanitization).
  • Compare candidates and the user's original on joint log-probability; keep the per-token mean only for the suffix-join probe (identical window text across candidates).
  • Replace the absolute floor + "original much better" slack with one gate: the replacement must beat the user's own string by minimumAdvantageOverOriginal (default 0 — any model preference for the original vetoes, which strengthens deliberate-spelling / proper-noun protection). Dictionary flagging already guarantees the replacement is a real word.
  • Recalibrate margins to joint nats: minimumMargin 2.0, relaxed clearCutMargin 0.5 for dictionary-flagged, edit-distance ≤ 2, model-top guesses.
  • Skip the suffix-join probe for whitespace-only windows (probing a bare space token is noise that vetoed end-of-sentence corrections).
  • Correct an open current word only when it is a dead-end fragment; a fragment still extensible into a dictionary word is typing-in-progress and left to completion (mirrors the in-beam typo guard's closed-word rule, ADR-015).
  • Render the correction badge with a per-letter diff (strike only the wrong letters, embolden only the changed ones).

Result (real model): definatelydefinitely at confidence 0.95, joint margin ≈ 5.4 (thin context) / 6.0 (rich); verified end-to-end on device and with unit tests pinning the logged regression cases.

Logged as ADR-116 (next after 115 on this branch's base). Note: my other open PR #41 also introduces an ADR-116 — happy to renumber whichever you merge second. Independent of #41 (disjoint files apart from the decision log).

🤖 Generated with Claude Code

…d boundary

The autocorrect lane detected typos but suppressed nearly every fix even in
ideal context (definately after "I " stayed lowModelMargin). Instrumenting
the scorer against a real GGUF found three compounding scoring defects, not a
thresholds problem — which explains the escalating threshold surgery in
ADR-113/114/115:

  1. the anchor kept its trailing space, so candidates tokenized standalone
     (a split BPE rarely produces after a space token), flooring every score;
  2. per-token MEAN log-probability favoured multi-token strings to the point
     of ranking the misspelling above its correction (mean: definately -6.56
     beats definitely -8.09; joint sums: definitely wins by ~5 nats);
  3. the -6.0 absolute floor asked "is this a likely continuation?" — the
     wrong question for a correction — so every candidate died there.

Fixes: trailing whitespace moves onto the scored word (ADR-017's analogue);
comparisons use joint sums; the floor and much-better slack collapse into one
advantage-over-original gate (default 0 — any model preference for the user's
own string vetoes, strengthening deliberate-spelling protection); margins
recalibrated to joint nats (2.0, relaxed to 0.5 for dictionary-flagged
edit-distance<=2 top guesses); whitespace-only suffix windows skip the join
probe; open words are corrected only when they are dead-end fragments
(mirroring the in-beam typo guard); the badge strikes only the wrong letters.

Real-model verdict for "I ": definitely conf=0.95 margin~5.4. ADR-116.
Unit tests pin the logged regression cases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant