Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
the reason the wave was built.

## V67 — Early stopping for the boosting: measured, and not built

F9, opened by V66, which had shown the boosting's validation error lowest well before its 120th round on 17 runs of 30.

_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way.

## V66 — The glass box: one row, replayed through the model

The third and last part of F8.
Expand Down
10 changes: 5 additions & 5 deletions PLAN.md

Large diffs are not rendered by default.

44 changes: 22 additions & 22 deletions README.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "labml",
"private": true,
"version": "1.66.0",
"version": "1.67.0",
"license": "MIT",
"type": "module",
"engines": {
Expand Down
80 changes: 40 additions & 40 deletions src/content/docs/en/limits.md

Large diffs are not rendered by default.

24 changes: 21 additions & 3 deletions src/content/docs/en/replay-a-prediction.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,27 @@ other rounds that asked this row exactly the same.
It also shows the error on the **validation rows** as the rounds add up. On the
demos the lowest validation error came well before the last round on 17 runs out
of 30, and on the energy demo the boosting never beat its starting point on
three splits out of five. The lab shows this and sets nothing: it always runs
its rounds to the end, and the curve is read on validation, never on the test
set.
three splits out of five.

That looks like a reason to stop earlier, so it was measured (V67), on a quarter
of the rows kept away from everything, ten splits per demo. Stopping at the
lowest point of the curve, against the 120 rounds the lab runs:

| Demo | Accuracy or RMSE on the rows kept away | Log loss |
| ------- | -------------------------------------- | ------------------------- |
| titanic | +0.4 point (better 5, worse 4) | better on 10 splits of 10 |
| fraud | no change | slightly worse |
| iris | no change | slightly worse |
| reviews | **4 points lower** (worse 6, better 2) | better on 10 splits of 10 |
| mpg | no change | no change |
| energy | RMSE 3.5 % lower (better 9, worse 1) | the same measure |

Fewer rounds give better probabilities, not better predictions: the classes
predicted are no more accurate, and on reviews they are less. The one gain is on
energy, where the boosting does no better than the mean and is never the family
the validation rows choose. So the lab keeps its 120 rounds. The curve says
where the error turned; it does not say that fewer rounds would predict better.
It is read on validation, never on the test set.

## Ties in a k-NN

Expand Down
80 changes: 40 additions & 40 deletions src/content/docs/fr/limites.md

Large diffs are not rendered by default.

26 changes: 23 additions & 3 deletions src/content/docs/fr/rejouer-une-prediction.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,29 @@ ligne.
Il montre aussi l'erreur sur les **lignes de validation** à mesure que les tours
s'ajoutent. Sur les démos, l'erreur de validation la plus basse arrivait bien
avant le dernier tour sur 17 runs sur 30, et sur la démo energy le boosting n'a
jamais battu son point de départ sur trois découpes sur cinq. Le labo le montre
et ne règle rien : il fait toujours ses tours jusqu'au bout, et la courbe se lit
sur la validation, jamais sur le jeu de test.
jamais battu son point de départ sur trois découpes sur cinq.

Cela ressemble à une raison de s'arrêter plus tôt, et cela a donc été mesuré
(V67), sur un quart des lignes tenu à l'écart de tout, dix découpes par démo.
S'arrêter au point le plus bas de la courbe, face aux 120 tours que fait le
labo :

| Démo | Exactitude ou RMSE sur les lignes tenues à l'écart | Log loss |
| ------- | -------------------------------------------------- | -------------------------------- |
| titanic | +0,4 point (mieux 5 fois, pire 4) | meilleure sur 10 découpes sur 10 |
| fraud | aucun changement | un peu moins bonne |
| iris | aucun changement | un peu moins bonne |
| reviews | **4 points de moins** (pire 6 fois, mieux 2) | meilleure sur 10 découpes sur 10 |
| mpg | aucun changement | aucun changement |
| energy | RMSE 3,5 % plus basse (mieux 9 fois, pire 1) | la même mesure |

Moins de tours donnent de meilleures probabilités, pas de meilleures
prédictions : les classes prédites ne sont pas plus justes, et sur reviews elles
le sont moins. Le seul gain est sur energy, où le boosting ne fait pas mieux que
la moyenne et n'est jamais la famille que les lignes de validation choisissent.
Le labo garde donc ses 120 tours. La courbe dit où l'erreur a tourné ; elle ne
dit pas que moins de tours prédiraient mieux. Elle se lit sur la validation,
jamais sur le jeu de test.

## Les ex æquo d'un k-NN

Expand Down
2 changes: 1 addition & 1 deletion src/features/ml/components/insights/GlassBoxView.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ describe('GlassBoxView', () => {
'On the 143 validation rows, the log loss was lowest at round 22 (0.433) and is 0.449 after round 120',
);
expect(screen.getByTestId('glass-validation')).toHaveTextContent(
'never on the test set, and it sets nothing',
'never on the test set, and it sets nothing: the lab runs its rounds to the end, and that was measured',
);
expect(screen.getAllByRole('img')).toHaveLength(2);
});
Expand Down
23 changes: 23 additions & 0 deletions src/features/ml/train/gbdt.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,29 @@
* LightGBM/XGBoost: features are bucketed into quantile bins once, trees are
* grown depth-wise on second-order gradients (XGBoost-style gain), and leaves
* take Newton values. Deterministic by construction — no randomness at all.
*
* V67: why the rounds are run to the end. V66 showed the validation error of
* the boosting lowest well before round 120 on 17 runs of 30, which reads as a
* case for early stopping. Measured before building it: six demos, ten splits
* each, a quarter of the rows kept away from everything, the 120 rounds
* against a stop at the lowest validation error (and against a stop chosen on
* rows held out inside the training rows, with or without a refit on all of
* them). On the metric the leaderboard ranks on:
*
* titanic +0.4 accuracy point (better 5, worse 4) log loss better 10/10
* fraud no change log loss slightly worse
* iris no change log loss slightly worse
* reviews -4.0 accuracy points (worse 6, better 2) log loss better 10/10
* mpg no change in RMSE
* energy RMSE 3.5 % lower (better 9, worse 1)
*
* Fewer rounds give better probabilities and no better predictions; the one
* gain is on energy, where the boosting does no better than the mean and is
* never the family the validation rows choose. Stops chosen on inner held-out
* rows did no better than the validation stop on five demos of six. And a stop chosen on
* the validation rows flatters the boosting on the rows that choose the
* family: its validation score rose above its private score by 0.9 accuracy
* point on titanic and 6 on reviews. So nothing stops early here.
*/

export interface GbdtParams {
Expand Down
3 changes: 2 additions & 1 deletion src/features/ml/train/glass-box.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@
* - the boosting's validation error was lowest well before its last round
* (more than 2 % under the final error) on 17 runs of 30, and on energy it
* never beat its starting point on three seeds of five. The replay shows
* it and sets nothing: stopping early is a training decision (F9).
* it and sets nothing. V67 measured the obvious fix, stopping at that
* round, and found it does not predict better: see `gbdt.ts`.
*
* Pure and deterministic.
*/
Expand Down
4 changes: 2 additions & 2 deletions src/locales/en.json
Original file line number Diff line number Diff line change
Expand Up @@ -758,11 +758,11 @@
},
"validation": {
"never": "On the {{rows}} validation rows, the {{metric}} was lowest before the first round ({{start}}) and is {{lastValue}} after round {{last}}: on these rows the boosting never beat its starting point.",
"early": "On the {{rows}} validation rows, the {{metric}} was lowest at round {{best}} ({{bestValue}}) and is {{lastValue}} after round {{last}}: the later rounds fitted the training rows and did not help these.",
"early": "On the {{rows}} validation rows, the {{metric}} was lowest at round {{best}} ({{bestValue}}) and is {{lastValue}} after round {{last}}: past that round this error grew again on these rows.",
"late": "On the {{rows}} validation rows, the {{metric}} went from {{start}} to {{lastValue}} after round {{last}}, its lowest being {{bestValue}} at round {{best}}: the rounds kept helping, or cost nothing."
},
"validationLabel": "The error on the validation rows, round after round",
"validationNote": "This is read on the validation rows, never on the test set, and it sets nothing: the lab always runs its rounds to the end. Measured on the demos, the lowest validation error came well before the last round on 17 runs out of 30.",
"validationNote": "Read on the validation rows, never on the test set, and it sets nothing: the lab runs its rounds to the end, and that was measured. On the demos, against rows kept away from everything, stopping at the lowest point of this curve made the predictions no more accurate on four demos of six and 4 points less accurate on reviews; it helped only on energy, where the boosting does no better than the mean. What it improves is the probabilities (log loss), not the classes predicted. The curve says where the error turned, not that fewer rounds would predict better.",
"same_one": "{{count}} other round asked this row the same questions.",
"same_other": "{{count}} other rounds asked this row the same questions."
},
Expand Down
4 changes: 2 additions & 2 deletions src/locales/fr.json
Original file line number Diff line number Diff line change
Expand Up @@ -758,11 +758,11 @@
},
"validation": {
"never": "Sur les {{rows}} lignes de validation, la {{metric}} était la plus basse avant le premier tour ({{start}}) et vaut {{lastValue}} après le tour {{last}} : sur ces lignes, le boosting n'a jamais battu son point de départ.",
"early": "Sur les {{rows}} lignes de validation, la {{metric}} était la plus basse au tour {{best}} ({{bestValue}}) et vaut {{lastValue}} après le tour {{last}} : les tours suivants ont collé aux lignes d'entraînement sans aider celles-ci.",
"early": "Sur les {{rows}} lignes de validation, la {{metric}} était la plus basse au tour {{best}} ({{bestValue}}) et vaut {{lastValue}} après le tour {{last}} : passé ce tour, cette erreur est remontée sur ces lignes.",
"late": "Sur les {{rows}} lignes de validation, la {{metric}} est passée de {{start}} à {{lastValue}} après le tour {{last}}, son plus bas étant {{bestValue}} au tour {{best}} : les tours ont continué d'aider, ou n'ont rien coûté."
},
"validationLabel": "L'erreur sur les lignes de validation, tour après tour",
"validationNote": "Ceci se lit sur les lignes de validation, jamais sur le jeu de test, et ne règle rien : le labo fait toujours ses tours jusqu'au bout. Mesuré sur les démos, l'erreur de validation la plus basse arrivait bien avant le dernier tour sur 17 runs sur 30.",
"validationNote": "Ceci se lit sur les lignes de validation, jamais sur le jeu de test, et ne règle rien : le labo fait ses tours jusqu'au bout, et c'est mesuré. Sur les démos, face à des lignes tenues à l'écart de tout, s'arrêter au point le plus bas de cette courbe n'a pas rendu les prédictions plus justes sur quatre démos sur six, et les a rendues moins justes de 4 points sur reviews ; cela n'a aidé que sur energy, où le boosting ne fait pas mieux que la moyenne. Ce qui s'améliore, ce sont les probabilités (log loss), pas les classes prédites. La courbe dit où l'erreur a tourné, pas que moins de tours prédiraient mieux.",
"same_one": "{{count}} autre tour a posé les mêmes questions à cette ligne.",
"same_other": "{{count}} autres tours ont posé les mêmes questions à cette ligne."
},
Expand Down
Loading