Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
the reason the wave was built.

## V70 — Twin columns are flagged, not removed

G2.

_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way.

## V69 — A SQL result reaches the lab whole

G1, the first of the ten proposals a new reading of the repository produced after V68 (the others are listed below).
Expand Down
28 changes: 14 additions & 14 deletions PLAN.md

Large diffs are not rendered by default.

9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ The project follows three non-negotiable principles:
| Competition path | **An unlabelled file becomes a submission.** Under a scored batch, one block writes an identifier and a prediction under the names the platform expects: a class, a probability or a value, every row kept in its order, missing and repeated identifiers counted. The lab guesses neither the identifier nor the names. The same family can be **fitted again on every labelled row**, announced as a model nobody measured: on the demos, against rows kept away from both models, it gained 5 accuracy points on the smallest text file, 0.8 on titanic, 2 to 3 % of RMSE on two regressions, nothing on two others, and lost one time in five to one in three even where it paid |
| Glass box | **One row, replayed through the model.** Beside the Shapley explanation (what weighed), a replay tells what happened: the questions a decision tree asked, in the units of the file, and the training rows that share its leaf; the count of a forest's 40 trees; the five rows of the file a k-NN compared it to, with its ties flagged; the prediction a boosting builds round after round, and its error on the validation rows. The replay walks the model itself: measured on 2,560 test rows of the demos, its last step is the model's answer on every one. The replay showed the boosting's validation error turning well before its last round; early stopping was then measured on rows kept away from everything and **not built**: fewer rounds improved the probabilities, not the predictions |
| Counterfactuals | **What would have to change.** For one row of a classification, the smallest change, one column or two, that makes the inspected model answer another class: only values the training rows hold, the columns the user locks left alone, free text never edited, and a plain « nothing found » with what was tried. Each change carries two readings the measure made necessary: whether the changed row still looks like a training row (8 to 60 % of the nearest changes did not), and how many of the other trained models follow it (14 to 54 % did). A counterfactual speaks of the model, not of the world |
| Twin columns | **Two columns that say the same thing are flagged, not removed.** As soon as a file is read, a column that maps one to one onto another (`class` and `pclass`), or can be read off another (`adult_male` off `who`), says so on its card. Twins split the importance between them and let a counterfactual change one without the other: on titanic 43 % of the first counterfactuals did. Removing them moves no score, so the lab leaves the choice to the user |
| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met |
| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) |
| Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner |
Expand Down Expand Up @@ -289,7 +290,7 @@ Wikimedia Commons), the portrait is NASA, public domain._
a doc page carries its whole article, in English until the app mounts); Lighthouse
mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
Runtime) load lazily.
- **Quality bar.** 1330 unit tests and 134 Playwright end-to-end tests across five
- **Quality bar.** 1344 unit tests and 135 Playwright end-to-end tests across five
projects — desktop, a phone viewport in English and in French, dark mode, and
Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on
Expand Down Expand Up @@ -409,9 +410,9 @@ requires of the next file what was found on a reference (V63), and a scikit-lear
that redoes a run outside the browser and measures its own gap (V64), a submission file
for a competition (V65), a step-by-step replay of one row through the model (V66), early
stopping for the boosting, measured and not built (V67), counterfactuals (V68), and a SQL
result that reaches the lab whole instead of its first 200 rows (V69). Proposed, none
scheduled: nine items from a new reading of the repository, listed in PLAN.md (duplicate
columns, a count of filled cells, ranking on probabilities, an honest forecast band, and
result that reaches the lab whole instead of its first 200 rows (V69), and twin columns
flagged before training (V70). Proposed, none scheduled: eight items from a new reading of
the repository, listed in PLAN.md (a count of filled cells, ranking on probabilities, an honest forecast band, and
others). The consented generative chat (V12) stays
postponed by product decision. No wave starts without an explicit launch command. The full plan, delivery log and design decisions live in
[PLAN.md](PLAN.md); [CHANGELOG.md](CHANGELOG.md) is extracted from it — one entry per
Expand Down
31 changes: 31 additions & 0 deletions e2e/redundancy.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import { expect, test } from '@playwright/test';

test.use({ locale: 'en-US' });

// NOTE: keep this title ASCII-only: it names the test-results directory.
test('titanic: twin columns are flagged on their cards, and excluded by the user only', async ({
page,
}) => {
await page.goto('/ml?demo=titanic&target=survived');
// class/pclass, embarked/embark_town, and adult_male read off who. The twin
// of the target, alive, is a leak and is flagged as one, not as a twin.
await expect(page.getByTestId('redundant-summary')).toContainText(
'5 columns say the same thing as another',
{ timeout: 30000 },
);
await expect(page.getByTestId('redundant-pclass')).toContainText(
'Says the same thing as "class".',
);
await expect(page.getByTestId('redundant-embark_town')).toContainText(
'Says the same thing as "embarked".',
);
await expect(page.getByTestId('redundant-adult_male')).toContainText('Can be read off "who".');
await expect(page.getByTestId('redundant-who')).toHaveCount(0);
await expect(page.getByTestId('redundant-alive')).toHaveCount(0);

// Flagged, not removed: both are still in until the user excludes one.
const card = page.getByTestId('column-card-class');
await card.getByRole('button', { name: 'Exclude' }).click();
await expect(page.getByTestId('redundant-pclass')).toHaveCount(0);
await expect(page.getByTestId('redundant-summary')).toContainText('3 columns');
});
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "labml",
"private": true,
"version": "1.69.0",
"version": "1.70.0",
"license": "MIT",
"type": "module",
"engines": {
Expand Down
12 changes: 8 additions & 4 deletions src/content/docs/en/find-a-counterfactual.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,14 @@ the cloud of training rows, and flags the ones that do not. On iris about half
the answers stay flagged whatever is tried: its four columns move together, and
changing one alone leaves the cloud.

The lab also does not know that two columns say the same thing. On titanic,
`sex`, `who` and `adult_male` describe one fact; changing `adult_male` alone
describes a passenger who cannot exist. Lock the columns that duplicate another,
or better, leave them out of the features.
Two columns that say the same thing are the other way to build a row that
cannot exist: changing `pclass` while `class` stays put. Measured on titanic
with every column in, **43 % of the first answers changed one column of such a
pair alone**. Since V70 the lab flags these twins on their cards before
training, and an answer that touches one without the other says so. Exclude one
of the two and train again: on the same rows, none of the answers did it any
more. What the flag does not see is a fact spread over three columns: `sex`,
`age` and `who` together.

## Reading two: do the other models follow?

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/en/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ They are not equivalent, and mixing them would be dishonest:
| Replaying a re-imported model, or drawing a whole tree | The replay needs the training rows the run holds, which an export does not carry. It shows the path of one row, not the graph of the tree |
| Advice from a counterfactual | It says what would make the model answer otherwise, not what would change the outcome in the world: the model learned which values go together, not what causes what |
| A counterfactual of more than two columns, or one that edits free text | A change of five columns is no longer an explanation, and a sentence has no nearest neighbour to try. When the answer sits in the text, the search finds nothing and says so |
| Knowing that two columns say the same thing | Changing one of two redundant columns alone can describe a row that cannot exist. The search flags a changed row far from every training row, which does not catch this case: lock the duplicate, or leave it out of the features |
| Removing a twin column, or seeing a fact spread over three | Two columns that map one to one, or one read off another, are flagged on their cards (V70) and left in: which one to keep is your call, and removing them moves no score. A fact carried by three columns together (`who` from `sex` and `age`) is not seen, nor is a redundancy that is only approximate |
| Counterfactuals for a regression, a re-imported model, or with the decision threshold | « Another class » has no meaning for a value; an export carries no training rows to draw observed values from; and the change found flips the prediction the what-if panel shows, not the decision of a chosen threshold |
| SQL reading a file the way the rest of the app does | The console hands the file to DuckDB's own reader: a decimal comma, a windows-1252 file or an Excel sheet are read by the app and not by SQL, which also always sees the original file, before the recipe and before a join. What leaves the console is complete since V69; what enters it is still DuckDB's reading |
| The recipe or the contract travelling with a SQL result | The ML Lab handoff carries the rows and a file name, nothing else |
Expand Down
10 changes: 10 additions & 0 deletions src/content/docs/en/reference-ml.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,16 @@ triggers task detection (binary, multi-class, regression) and **leak
detection**: a column that almost perfectly mirrors the target is excluded
automatically and named.

**Twin columns** are flagged too, as soon as the file is read: two columns that
map one to one (`class` and `pclass`), or one that can be read off another
(`adult_male` off `who`). Unlike a leak, a twin is **not excluded**: its card
says so and leaves the choice to you. Twins split the importance and the
Shapley values between them, and let a counterfactual change one without the
other. Measured on titanic over ten splits, removing them moves no score (0.813
against 0.814 of mean accuracy): it is about reading the model, not about its
performance. The test is exact, on values seen often enough to judge: it found
the three pairs of titanic and nothing on the nine other demo files.

While a run trains, and while anything computed from it is running
(hyperparameter search, learning curve, robust ranking, batch scoring), the
target and the columns are **locked**. Changing them at that moment would file
Expand Down
12 changes: 8 additions & 4 deletions src/content/docs/fr/chercher-un-contrefactuel.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,14 @@ pas. Sur iris, environ la moitié des réponses restent signalées quoi qu'on
essaie : ses quatre colonnes bougent ensemble, et en changer une seule fait sortir
du nuage.

Le labo ne sait pas non plus que deux colonnes disent la même chose. Sur titanic,
`sex`, `who` et `adult_male` décrivent un seul fait ; changer `adult_male` seul
décrit un passager qui ne peut pas exister. Verrouillez les colonnes qui en
doublent une autre, ou mieux, retirez-les des variables.
Deux colonnes qui disent la même chose sont l'autre façon de construire une
ligne qui ne peut pas exister : changer `pclass` alors que `class` ne bouge pas.
Mesuré sur titanic avec toutes les colonnes, **43 % des premières réponses
changeaient une seule colonne d'une telle paire**. Depuis V70 le labo signale
ces jumelles sur leurs cartes avant l'entraînement, et une réponse qui touche
l'une sans l'autre le dit. Excluez l'une des deux et réentraînez : sur les mêmes
lignes, plus aucune réponse ne le faisait. Ce que le signalement ne voit pas,
c'est un fait réparti sur trois colonnes : `sex`, `age` et `who` ensemble.

## Seconde lecture : les autres modèles suivent-ils ?

Expand Down
Loading
Loading