Skip to content

V71: what stood in for a missing cell is counted - #180

Merged
dapiced merged 1 commit into
mainfrom
dom/modest-darwin-4oe9q0
Nov 19, 2026
Merged

dapiced merged 1 commit into
mainfrom
dom/modest-darwin-4oe9q0

Conversation

@dapiced

@dapiced dapiced commented Nov 19, 2026

Copy link
Copy Markdown
Owner

What

G3. A missing or unreadable number has always become the training median, and a missing category the most frequent one; a missing text is left with no word. It was done with no count anywhere.

On titanic deck is 77 % empty, under the 95 % that sets a column aside: it trained, three quarters of it a value nobody recorded, and neither the run, the report nor the model card said so.

The encoder now counts what it fills, per column, empty cells apart from unreadable ones.

  • train/pipeline.ts: countFilled, over the rows of a run or of a scored file.
  • train/filled.ts: the sentence, and readFilled, a validator, since a summary can arrive through a share link.
  • TrainSummary.filled: the count travels with the saved run and the share link.
  • Said under the leaderboard (FilledNotice), in the HTML report, in the model card (whose limits name the columns filled past one half), and for a scored file in BatchScorePanel.

Nothing about the filling changes and no figure of any run moves: every existing test passes untouched.

Measured first

On the six demos: titanic fills deck on 688 rows of 891 (77 %), age on 177 (20 %), embarked and embark_town on 2; mpg fills horsepower on 6 of 398; the four others fill nothing.

What a mostly filled column is worth, titanic, ten seeds, eight families:

deck Mean accuracy
as it is, filled by « C » 0.813
removed 0.818
recoded as known / unknown 0.817
« unknown » as a category of its own 0.815

Half a point at most, inside the noise. The plan promised a « measured alert threshold »: it does not exist, since no score separates a harmless share from a harmful one. The mark kept is a reading rule and is called one: past one half, most of what the models read in the column is its fill value, and its importance is that of a mostly constant column.

Defects of this wave's own making

Two of the new tests expected the wrong thing and were right to fail: n/a is a marker of a missing value, so it counts as empty and not as unreadable (the documentation now says so); and on a tie the most frequent category is the first in alphabetical order.

Not in this wave

Another way to fill a cell (model-based imputation was refused twice); a « was missing » column, which the measure did not justify; setting aside a column because it is mostly filled; the Data Studio's recipe (G10).

Docs

ML reference (new section), limits, « score a batch », in both languages; README; PLAN row V71; changelog; version 1.71.0. Seven proposals remain in PLAN.md (G4 to G10).

Checks

  • Lint, typecheck (apart from the xlsx types unreachable in my sandbox), format: clean.
  • Unit: 1350 passed locally, 5 failed, all in spreadsheet.test.ts (the xlsx tarball cannot be fetched in my sandbox; CI is the reference). 11 new tests.
  • E2E: 135 passed locally after a fresh build, 1 failed (lab.spec.ts:31, the Excel upload, same cause). New e2e/filled.spec.ts on titanic.
  • The branch was rebased on main after V70 merged (one commit, no conflict); the full suites ran before the rebase, the touched tests and the typecheck again after it.

Generated by Claude Code

A missing or unreadable number has always become the training median,
and a missing category the most frequent one, with no count anywhere.
On titanic deck is 77 % empty, under the 95 % that sets a column aside:
it trained, three quarters of it a value nobody recorded, and neither
the run, the report nor the model card said so.

The encoder now counts what it fills, per column, empty cells apart from
unreadable ones. The count travels with the run's summary and is said
under the leaderboard, in the report and in the model card; a scored
file says the same for its own rows. Nothing about the filling changes
and no figure of any run moves.

Measured first: keeping deck, dropping it or recoding it as known or
unknown gives 0.813, 0.818 and 0.817 of mean accuracy over ten seeds,
inside the noise. So the mark past one half is a reading rule, named as
one, not a threshold drawn from scores.

Docs in both languages, README, PLAN row V71, version 1.71.0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AyBsUet8WybTG986ubroar
@dapiced
dapiced marked this pull request as ready for review November 19, 2026 23:15
@dapiced
dapiced merged commit 887734b into main Nov 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants