Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
the reason the wave was built.

## V71 — What stood in for a missing cell is counted

G3.

_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way.

## V70 — Twin columns are flagged, not removed

G2.
Expand Down
6 changes: 3 additions & 3 deletions PLAN.md

Large diffs are not rendered by default.

7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ The project follows three non-negotiable principles:
| Glass box | **One row, replayed through the model.** Beside the Shapley explanation (what weighed), a replay tells what happened: the questions a decision tree asked, in the units of the file, and the training rows that share its leaf; the count of a forest's 40 trees; the five rows of the file a k-NN compared it to, with its ties flagged; the prediction a boosting builds round after round, and its error on the validation rows. The replay walks the model itself: measured on 2,560 test rows of the demos, its last step is the model's answer on every one. The replay showed the boosting's validation error turning well before its last round; early stopping was then measured on rows kept away from everything and **not built**: fewer rounds improved the probabilities, not the predictions |
| Counterfactuals | **What would have to change.** For one row of a classification, the smallest change, one column or two, that makes the inspected model answer another class: only values the training rows hold, the columns the user locks left alone, free text never edited, and a plain « nothing found » with what was tried. Each change carries two readings the measure made necessary: whether the changed row still looks like a training row (8 to 60 % of the nearest changes did not), and how many of the other trained models follow it (14 to 54 % did). A counterfactual speaks of the model, not of the world |
| Twin columns | **Two columns that say the same thing are flagged, not removed.** As soon as a file is read, a column that maps one to one onto another (`class` and `pclass`), or can be read off another (`adult_male` off `who`), says so on its card. Twins split the importance between them and let a counterfactual change one without the other: on titanic 43 % of the first counterfactuals did. Removing them moves no score, so the lab leaves the choice to the user |
| Filled cells | **What stood in for a missing cell is counted.** A missing number has always become the training median and a missing category the most frequent one, in silence. The run, the report, the model card and a scored file now say, per column, how many cells were empty, how many could not be read as a number, and what replaced them. On titanic `deck` is 77 % filled: past one half the run says the column is mostly its fill value. Keeping it, dropping it or recoding it moved no score, so the lab reports and leaves the choice to the user |
| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met |
| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) |
| Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner |
Expand Down Expand Up @@ -290,7 +291,7 @@ Wikimedia Commons), the portrait is NASA, public domain._
a doc page carries its whole article, in English until the app mounts); Lighthouse
mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
Runtime) load lazily.
- **Quality bar.** 1344 unit tests and 135 Playwright end-to-end tests across five
- **Quality bar.** 1355 unit tests and 136 Playwright end-to-end tests across five
projects — desktop, a phone viewport in English and in French, dark mode, and
Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on
Expand Down Expand Up @@ -411,8 +412,8 @@ that redoes a run outside the browser and measures its own gap (V64), a submissi
for a competition (V65), a step-by-step replay of one row through the model (V66), early
stopping for the boosting, measured and not built (V67), counterfactuals (V68), and a SQL
result that reaches the lab whole instead of its first 200 rows (V69), and twin columns
flagged before training (V70). Proposed, none scheduled: eight items from a new reading of
the repository, listed in PLAN.md (a count of filled cells, ranking on probabilities, an honest forecast band, and
flagged before training (V70), and a count of the cells the lab fills in (V71). Proposed,
none scheduled: seven items from a new reading of the repository, listed in PLAN.md (ranking on probabilities, an honest forecast band, and
others). The consented generative chat (V12) stays
postponed by product decision. No wave starts without an explicit launch command. The full plan, delivery log and design decisions live in
[PLAN.md](PLAN.md); [CHANGELOG.md](CHANGELOG.md) is extracted from it — one entry per
Expand Down
29 changes: 29 additions & 0 deletions e2e/filled.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import { expect, test } from '@playwright/test';

test.use({ locale: 'en-US' });
test.setTimeout(120_000);

// NOTE: keep this title ASCII-only: it names the test-results directory.
test('titanic: the run says which cells it filled in, and how many', async ({ page }) => {
await page.goto('/ml?demo=titanic&target=survived');
await page.getByTestId('train-button').click();
await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 90000 });

const notice = page.getByTestId('filled-notice').first();
await expect(notice).toContainText('Cells the lab filled in before training');
// deck is 77 % empty: kept, since under the 95 % that sets a column aside,
// and most of what the models read in it is its most frequent value.
const deck = notice.getByTestId('filled-deck');
await expect(deck).toHaveAttribute('data-mostly', 'true');
await expect(deck).toContainText('688 cells were empty of 891 (77 %). Filled with "C".');
const age = notice.getByTestId('filled-age');
await expect(age).toHaveAttribute('data-mostly', 'false');
await expect(age).toContainText('177 cells were empty of 891 (20 %)');

// Excluded, the column is no longer filled, and no longer counted.
await page.getByTestId('column-card-deck').getByRole('button', { name: 'Exclude' }).click();
await page.getByTestId('train-button').click();
await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 90000 });
await expect(page.getByTestId('filled-deck')).toHaveCount(0);
await expect(page.getByTestId('filled-age').first()).toBeVisible();
});
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "labml",
"private": true,
"version": "1.70.0",
"version": "1.71.0",
"license": "MIT",
"type": "module",
"engines": {
Expand Down
2 changes: 2 additions & 0 deletions src/content/docs/en/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ They are not equivalent, and mixing them would be dishonest:
| Advice from a counterfactual | It says what would make the model answer otherwise, not what would change the outcome in the world: the model learned which values go together, not what causes what |
| A counterfactual of more than two columns, or one that edits free text | A change of five columns is no longer an explanation, and a sentence has no nearest neighbour to try. When the answer sits in the text, the search finds nothing and says so |
| Removing a twin column, or seeing a fact spread over three | Two columns that map one to one, or one read off another, are flagged on their cards (V70) and left in: which one to keep is your call, and removing them moves no score. A fact carried by three columns together (`who` from `sex` and `age`) is not seen, nor is a redundancy that is only approximate |
| Another way to fill a missing cell, or a « was missing » column | The median and the most frequent category are the whole of it, and model-based imputation was refused twice. Since V71 the lab counts what it fills and says so; on titanic an indicator for `deck` changed no score, so none is created for you |
| Setting aside a column that is mostly filled | Under 95 % of empty cells a column stays in. Past one half the run says the column is mostly its fill value, and leaves the decision to you |
| Counterfactuals for a regression, a re-imported model, or with the decision threshold | « Another class » has no meaning for a value; an export carries no training rows to draw observed values from; and the change found flips the prediction the what-if panel shows, not the decision of a chosen threshold |
| SQL reading a file the way the rest of the app does | The console hands the file to DuckDB's own reader: a decimal comma, a windows-1252 file or an Excel sheet are read by the app and not by SQL, which also always sees the original file, before the recipe and before a join. What leaves the console is complete since V69; what enters it is still DuckDB's reading |
| The recipe or the contract travelling with a SQL result | The ML Lab handoff carries the rows and a file name, nothing else |
Expand Down
19 changes: 19 additions & 0 deletions src/content/docs/en/reference-ml.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,25 @@ over the classes present in the true labels or in the predictions. A class
absent from both does not count as a zero; a class that is predicted but never
true does, because those predictions are all wrong.

## What stands in for a missing cell

A model cannot read an empty cell. Before training, a missing or unreadable
number becomes the **median** of the training rows, a missing category the
**most frequent** one, and a missing text is left with no word at all. The fill
values are fitted on the training rows only.

Since V71 this is counted. Under the leaderboard, in the report and in the model
card, each filled column says how many cells were empty, how many held something
that could not be read as a number (more often a reading problem than a gap),
and what stood in for them. A scored file says the same for its own rows. The
usual markers of a missing value (`NA`, `n/a`, and the like) count as empty.

A column filled past **one half** is said to be mostly its fill value. That is a
reading rule, not a threshold drawn from scores: on titanic, `deck` is 77 %
empty, and keeping it, dropping it or recoding it as known/unknown gives 0.813,
0.818 and 0.817 of mean accuracy over ten splits, which is noise. What changes
is how to read its importance: it is that of a mostly constant column.

## The run journal

Some steps of a run are best-effort: the ensemble is optional, the helper
Expand Down
7 changes: 7 additions & 0 deletions src/content/docs/en/score-a-batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,13 @@ abstained row is a case to route to human review, not a model error.

With no rule attached to the model, these columns do not appear at all.

## Cells filled in the scored file

A scored file is filled the way the training rows were: a missing number takes
the training median, a missing category the most frequent training one. The
result says how many cells of each column that was, so a file that arrives half
empty in a column the model relies on does not pass in silence.

## Labels never seen

If your batch contains a class absent from training, LabML still predicts it
Expand Down
Loading
Loading