Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
the reason the wave was built.

## V66 — The glass box: one row, replayed through the model

The third and last part of F8.

_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way.

## V65 — An unlabelled file becomes a submission

The second part of F8; the glass box stays proposed below.
Expand Down
11 changes: 6 additions & 5 deletions PLAN.md

Large diffs are not rendered by default.

5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ The project follows three non-negotiable principles:
| MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts, or **up to six at once** read against the oldest of the selection; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining |
| Python bridge | **A run, redone with scikit-learn.** One readable script carries the row numbers of each split and the preprocessing, and translates each model to its nearest estimator. It does not promise the lab's figures: it prints them next to its own, with the gap. Measured on 30 runs: the encoded matrix is equal to within 2.9e-14, the baseline, the linear regression and the naive Bayes agree to the last digit, k-NN agrees until two neighbours tie, and the hand-written families differ by up to several test rows on one split. A column name is never code: the run travels as one JSON block. |
| Competition path | **An unlabelled file becomes a submission.** Under a scored batch, one block writes an identifier and a prediction under the names the platform expects: a class, a probability or a value, every row kept in its order, missing and repeated identifiers counted. The lab guesses neither the identifier nor the names. The same family can be **fitted again on every labelled row**, announced as a model nobody measured: on the demos, against rows kept away from both models, it gained 5 accuracy points on the smallest text file, 0.8 on titanic, 2 to 3 % of RMSE on two regressions, nothing on two others, and lost one time in five to one in three even where it paid |
| Glass box | **One row, replayed through the model.** Beside the Shapley explanation (what weighed), a replay tells what happened: the questions a decision tree asked, in the units of the file, and the training rows that share its leaf; the count of a forest's 40 trees; the five rows of the file a k-NN compared it to, with its ties flagged; the prediction a boosting builds round after round, and its error on the validation rows. The replay walks the model itself: measured on 2,560 test rows of the demos, its last step is the model's answer on every one |
| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met |
| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) |
| Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner |
Expand Down Expand Up @@ -281,12 +282,12 @@ Wikimedia Commons), the portrait is NASA, public domain._
so), slow model families train on measured, announced caps scored against the same
full test set, and parsing refuses past a named 20M-cell memory budget instead of
letting the tab die, in every section and for a batch to score.
- **Performance.** Every page — the sections, the home page and each of the thirteen
- **Performance.** Every page — the sections, the home page and each of the fourteen
documentation URLs — serves a prerendered static shell (hero paints before JavaScript;
a doc page carries its whole article, in English until the app mounts); Lighthouse
mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
Runtime) load lazily.
- **Quality bar.** 1251 unit tests and 132 Playwright end-to-end tests across five
- **Quality bar.** 1294 unit tests and 133 Playwright end-to-end tests across five
projects — desktop, a phone viewport in English and in French, dark mode, and
Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on
Expand Down
69 changes: 69 additions & 0 deletions e2e/glass-box.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
import { expect, test, type Page } from '@playwright/test';

test.use({ locale: 'en-US' });
test.setTimeout(120_000);

/** Inspects one family and replays the default what-if row through it. */
async function replay(page: Page, family: RegExp) {
await page.getByRole('cell', { name: family }).click();
const prediction = page.getByTestId('what-if-prediction');
await expect(prediction).toBeVisible({ timeout: 30000 });
await page.getByTestId('replay-button').click();
const box = page.getByTestId('glass-box');
await expect(box).toBeVisible({ timeout: 30000 });
return { box, predicted: (await prediction.textContent())!.trim() };
}

// NOTE: keep this title ASCII-only: it names the test-results directory.
test('glass box: each family replays a row to the prediction shown', async ({ page }) => {
await page.goto('/ml?demo=iris&target=species');
await page.getByTestId('train-button').click();
await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60000 });

// The tree: questions in the units of the file, down to a leaf.
const tree = await replay(page, /Decision tree/);
await expect(tree.box).toHaveAttribute('data-kind', 'tree');
await expect(tree.box.locator('ol li').first()).toContainText(
/(petal|sepal)_(length|width) < [\d.]+\? (yes|no)/,
);
await expect(tree.box.getByTestId('glass-answer')).toContainText(
`The tree answers ${tree.predicted}.`,
);
await expect(tree.box.getByTestId('glass-answer')).toContainText(
'training rows end in the same leaf',
);

// k-NN: five rows of the file, and their vote.
const knn = await replay(page, /k-nearest neighbors/i);
await expect(knn.box).toHaveAttribute('data-kind', 'knn');
await expect(knn.box.getByTestId('glass-neighbours').locator('tbody tr')).toHaveCount(5);
await expect(knn.box.getByTestId('glass-answer')).toContainText(
`The model answers ${knn.predicted}.`,
);

// The forest: a count of its trees.
const forest = await replay(page, /Random forest/);
await expect(forest.box.getByTestId('glass-answer')).toContainText(`40 trees voted`);
await expect(forest.box.getByTestId('glass-answer')).toContainText(
`The forest answers ${forest.predicted}.`,
);

// The boosting: a curve that ends on the class predicted, and the validation rows.
const boosting = await replay(page, /Gradient boosting/i);
await expect(boosting.box.getByTestId('glass-answer')).toContainText(
`the share of "${boosting.predicted}" the model begins with`,
);
await expect(boosting.box.getByTestId('glass-validation')).toContainText('validation rows');

// A new row empties the replay, like the explanation: it described another row.
await page.getByTestId('what-if').locator('input[type="number"]').first().fill('7.7');
await expect(page.getByTestId('glass-box')).toHaveCount(0);
await expect(page.getByTestId('replay-button')).toBeVisible();

// A family with no steps says so instead of offering a replay.
await page.getByRole('cell', { name: /Logistic regression/ }).click();
await expect(page.getByTestId('glass-none')).toContainText('one weighted sum', {
timeout: 30000,
});
await expect(page.getByTestId('replay-button')).toHaveCount(0);
});
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "labml",
"private": true,
"version": "1.65.0",
"version": "1.66.0",
"license": "MIT",
"type": "module",
"engines": {
Expand Down
4 changes: 4 additions & 0 deletions src/content/docs/en/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ They are not equivalent, and mixing them would be dishonest:
| A score or a rank for a submission | The lab has neither the platform's metric nor its private rows. It counts the rows written and the identifiers a platform would refuse, nothing more |
| A measure of the model refitted on every row | Fitting on everything spends the test set. The score shown stays the one of the model fitted on 64 % of the rows; on the demos the refit gained from nothing to 5 points, and lost one time in five to one in three even where it paid |
| A submission from a re-imported model, a blend refitted, or a threshold applied | The submission is built from the live run. A re-imported model has no training rows to refit on, a blend picks its members on validation rows, and the label written is the model's raw prediction, as in the scored CSV |
| Replaying how a model was trained | The replay tells how the trained model predicts one row. How each tree was grown, round after round, is another story and a much longer one |
| Replaying a network, a linear model, a blend, or a forest tree by tree | A network has no steps that read as questions, a weighted sum is already read by the Shapley explanation, a blend averages other models, and forty trees are counted, not told. The panel says so for each |
| Stopping the boosting early | The replay shows the validation error round by round, and on the demos its lowest point came well before the last round on 17 runs out of 30. It is shown and sets nothing: changing the number of rounds is a training decision that needs its own measure |
| Replaying a re-imported model, or drawing a whole tree | The replay needs the training rows the run holds, which an export does not carry. It shows the path of one row, not the graph of the tree |
| A generative chat backed by a remote API (V12) | Postponed by product decision (confirmed 09/10/2026). Today's chat runs entirely in the browser; a remote API would send out the question and the column schema, never the data, and would require a consent screen |

## Dropped after measurement
Expand Down
1 change: 1 addition & 0 deletions src/content/docs/en/read-a-learning-curve.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,6 @@ beyond what was measured is a guess, and LabML does not offer one.

## Where to go next

- [Replay a prediction](/docs/rejouer-une-prediction) to read what the model did on one row.
- [Compare two runs](/docs/comparer-deux-runs) to measure a change of features.
- [The method choices](/docs/methode) on why the baseline comes first.
9 changes: 9 additions & 0 deletions src/content/docs/en/reference-ml.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,15 @@ says so when it is not the one the card is about.
The card holds figures only, never a row of the data. What you type is kept
with the run in your browser and travels in its share link.

## Replaying one row

The what-if panel can **replay** its row through the inspected model: the
questions a decision tree asked, the count of a forest's trees, the five rows a
k-NN compared it to, the prediction a boosting builds round after round. The
replay walks the model itself, so its last step is the prediction shown, and its
questions are written in the units of the file. See
[replay a prediction](/docs/rejouer-une-prediction).

## Export, import, score

The model exports to JSON with its manifest: family, pipeline, versions. It
Expand Down
89 changes: 89 additions & 0 deletions src/content/docs/en/replay-a-prediction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
---
slug: rejouer-une-prediction
kind: how-to
order: 6
title: Replay a prediction step by step
summary: Open the model on one row and read what it did, the questions a tree asked, the rows a k-NN consulted, the rounds of a boosting.
---

## The gesture

1. Train a model and click a family in the leaderboard to inspect it.
2. In **What if…**, set the values of one row. The prediction updates.
3. Click **Replay it step by step**.

:::try /ml?demo=titanic&target=survived | Train titanic and replay a passenger

**Why this prediction?** (Shapley) says what weighed. The replay says what
happened: the steps the model took for this row, in the order it took them.
The two answer different questions and are worth reading together.

## What each family tells

| Family | What the replay shows |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Decision tree | each question asked of the row, how many training rows were asked the same, and the leaf it ends in: how many training rows share it and what they are |
| Random forest | how many of its 40 trees answered each class, or the spread of their values |
| k-NN | the 5 training rows the row was compared to: their row number in your file, their distance, their target and their main columns |
| Gradient boosting | the prediction round after round, the rounds that moved it most, and the error on the validation rows as the rounds add up |

The other families have no steps to tell, and the panel says why: a linear or
logistic model is one weighted sum that the Shapley explanation already reads, a
network mixes every column at once, and the blend averages other models.

## The replay is the model

Nothing is approximated. A replay walks the structure the model itself predicts
with, and its last step is the prediction shown above it. This was measured
before the panel was built: on the six demos, five splits each, 2,560 test rows,
the replay gave the model's answer on every row for the four families.

The first measure found one gap: three rows of titanic where the forest and its
replay disagreed. On a tie between two classes the library keeps the first class
to reach the top count, in the order of the trees, not the smallest one. The
replay now does the same, and says when a vote is tied.

## Questions in the units of your file

A model sees encoded columns: a number standardized, a category spread over 0/1
columns. The replay translates what translates exactly:

- a number gets its unit back: `age < 9.5`, not `age < -1.42`;
- a 0/1 column becomes a question about the category: « sex is "male"? »;
- a word of a free-text column and a frequency rank are shown as the model sees
them, under their encoded name. On the reviews demo, whose columns are text,
nearly every question of the tree is of that kind.

## Reading the boosting

Every row starts at the same point (the share of the class in training, or the
mean), and each round adds a small correction. The panel gives the round by
which 90 % of the way was made, and tells up to three rounds in full: the
largest movers among those asking different questions, each with the number of
other rounds that asked this row exactly the same.

It also shows the error on the **validation rows** as the rounds add up. On the
demos the lowest validation error came well before the last round on 17 runs out
of 30, and on the energy demo the boosting never beat its starting point on
three splits out of five. The lab shows this and sets nothing: it always runs
its rounds to the end, and the curve is read on validation, never on the test
set.

## Ties in a k-NN

When the next row is exactly as far as the last neighbour kept, another row
order or another library would have kept the other one, and the answer can
change with it. It is common where columns hold few distinct values: 17 % of the
test rows on titanic, 39 % on energy. The panel flags it on the row concerned.

## A trace is not a cause

The path of a tree says which questions were asked of this row. It does not say
that a column makes the answer: another tree, trained on another split, would
ask other questions and often agree on the prediction.

## Where to go next

- [Read a learning curve](/docs/lire-une-courbe) to see whether more rows would help.
- [The ML Lab reference](/docs/reference-ml) for what each family is.
- [The limits](/docs/limites) for what the replay does not do.
Loading
Loading