From 4467aeffcec362fb9424ffd0e193db31a6cee0ed Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 13 Nov 2026 19:09:02 +0000 Subject: [PATCH] docs(ml): V67 early stopping for the boosting, measured and not built V66 showed the boosting's validation error lowest well before its last round on 17 runs of 30. Before building early stopping, it was measured: six demos, ten splits each, a quarter of the rows kept away from everything, the 120 fixed rounds against a stop at the lowest validation error, a stop on rows held out inside the training rows, and that stop followed by a refit. On the metric the leaderboard ranks on, stopping early changes nothing on four demos, loses 4 accuracy points on reviews, and helps only on energy, where the boosting does no better than the mean. It improves the log loss, not the predictions. A stop chosen on the validation rows also flatters the boosting on the rows that choose the family. Nothing changes in training. The measure is recorded in gbdt.ts and PLAN.md, the glass box no longer suggests that fewer rounds would predict better, and the docs carry the table in both languages. F9 is closed. Version 1.67.0. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01AyBsUet8WybTG986ubroar --- CHANGELOG.md | 6 ++ PLAN.md | 10 +-- README.md | 44 +++++----- package-lock.json | 4 +- package.json | 2 +- src/content/docs/en/limits.md | 80 +++++++++---------- src/content/docs/en/replay-a-prediction.md | 24 +++++- src/content/docs/fr/limites.md | 80 +++++++++---------- src/content/docs/fr/rejouer-une-prediction.md | 26 +++++- .../components/insights/GlassBoxView.test.tsx | 2 +- src/features/ml/train/gbdt.ts | 23 ++++++ src/features/ml/train/glass-box.ts | 3 +- src/locales/en.json | 4 +- src/locales/fr.json | 4 +- 14 files changed, 190 insertions(+), 122 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cbb9893..f25f723 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and the reason the wave was built. +## V67 — Early stopping for the boosting: measured, and not built + +F9, opened by V66, which had shown the boosting's validation error lowest well before its 120th round on 17 runs of 30. + +_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. + ## V66 — The glass box: one row, replayed through the model The third and last part of F8. diff --git a/PLAN.md b/PLAN.md index 87eea96..93b0188 100644 --- a/PLAN.md +++ b/PLAN.md @@ -680,7 +680,7 @@ defect that produces a wrong number comes first), then extends the lab past the export: uncertainty with a guarantee, governance, and monitoring after deployment, with nothing added on a server. -V53 to V66 are delivered. Every other row is a proposal with its launch +V53 to V67 are delivered. Every other row is a proposal with its launch condition, and none starts without an explicit launch command. | Wave | Content | Why | @@ -700,14 +700,14 @@ condition, and none starts without an explicit launch command. | **V64 — delivered** | **A run, redone in Python.** The first of the three parts of F8, the two others staying proposed below. A run stayed inside the browser: its model, its predictions, its report and its card export, the training itself did not. This writes a scikit-learn script that redoes it. **What the code said before anything was written**: the lab's estimators are written by hand or come from ml.js, and `tests/golden/README.md` has held since V42 that requiring the same numbers of a tree, a forest, a boosting or a network « would be a false contract ». So the script could not promise the lab's figures, and the design starts there: carry exactly what can be carried, translate the rest, and print both figures with the gap. **Carried exactly**: the row numbers of the three splits (not a seed that should redo them), and the preprocessing, one transformer per column in the lab's order. **Translated**: each family to its nearest estimator with the lab's settings, labelled on every line it prints. **Measured, six demos and five seeds each, with scikit-learn 1.9.1, and the first run found three faults in the translation.** (1) A numeric column with missing values was encoded differently (0.36 on titanic's `age`): the lab standardizes on the values that are there and then imputes the median, the script did it the other way round and computed the mean on imputed cells. (2) The network, on a regression, gave an RMSE of 219 against the lab's 18 on energy: the lab's network learns a standardized target and converts back, and the script now wraps its estimator the same way. (3) k-NN was labelled « same algorithm » and moved by 0.39 on energy, a file with one feature column: two neighbours at the same distance are not chosen the same way by the two libraries. It has a label of its own. **After the fixes, on the 30 runs**: the matrix the script encodes from the training rows equals the lab's to within **2.9e-14**, free text included. The baseline, the linear regression and the Gaussian naive Bayes give the same test metric on every run. k-NN gives the same on four demos out of six, and differs on titanic (one row of 178 at worst) and energy (up to 1.32 of RMSE). The translated families differ by ±0.06 of accuracy on average at most and by up to 0.15 on one split, which on reviews' 48 test rows is seven rows; the lab's tree is 4.7 points better than scikit-learn's on titanic on average, its network 6 points better on iris. The whole record is committed (`tests/golden/python-bridge.md`) with the two commands that regenerate it, outside CI like the other references. **A column name is never code.** The run travels as one JSON block read by `json.loads`; the rest of the script is the same text for every run. Every apostrophe of the block is written as its JSON escape, so nothing can close the string that holds it; a test gives a column the name of a shell command and checks that Python reads the name and runs none of it. **The script is run in the suite.** `python3 -m py_compile`, then `--check`, a mode that reads the run with the standard library alone: a script that does not compile fails the tests, in CI too. It refuses a file with another number of rows or columns, since its row numbers would point at other rows. **What this wave deliberately does not do**: promise the lab's figures beyond the families where they were measured equal; export a notebook, an environment file or a project; redo the ensemble (named as not redone), the hyperparameter search, the learning curve, the group audit or the production check; read a spreadsheet, which it asks to be saved as CSV. 1224 unit tests, 131 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | | **V65 — delivered** | **An unlabelled file becomes a submission.** The second part of F8; the glass box stays proposed below. Scoring a batch gave back every original column plus the prediction, the right file to join back into a system and the wrong one for a platform that expects two columns under its own names. Under a scored batch, a block now writes an identifier and a prediction. **What the code said before anything was written**: the lab fits on 64 % of the labelled rows and keeps 16 % to choose and 20 % to measure, and the usage in a competition is to fit again on everything before predicting. That was the real question of the wave, so it was measured first. **Measured, six demos, ten splits each, a quarter of the rows kept away from both models, every family**: the same family fitted on all the remaining rows instead of 64 % of them gained **+5.0 accuracy points on reviews** (180 rows; better 51 times of 70, worse 11), **3 % of RMSE on mpg** (47 of 60, worse 13), **2 % on energy** (44 of 60, worse 16), **+0.8 point on titanic** (41 of 70, worse 20), and **nothing on fraud** (-0.04 point; better 21, worse 25) **nor on iris** (-0.1 point; better 14, worse 17). The plan expected « a small and variable gain, clearer on small files ». The second half was wrong: iris is the smallest demo and gains nothing, because 72 rows already say all there is to say. The gain follows the learning curve, not the size, and the panel now says so and points at the curve (V26). Even where the refit pays it loses one time in five to one in three, so it is **offered and never the default**, and the result line names the model that predicted: the measured one, or its sibling « nobody measured ». A refit costs 18 ms to 3.6 s per family on the demos, done on demand. **The lab guesses nothing.** The user picks the identifier column (or asks for the rows to be numbered from 1), types the two names, and chooses what a prediction is: the class by name, the probability of the second class (binary), one probability per class (three classes or more), or the value (regression). A form with no object for the task is not offered. A column the training file read as an identifier is pointed at when there is exactly one, and not chosen. **No row lost or reordered.** The file has the rows of the scored file in their order; an identifier missing or repeated is written and counted, since a platform would refuse it. The end-to-end test scores a file whose identifiers are out of order and checks the order survives. **A submission is not a score**: the block says the lab knows neither the platform's metric nor its private rows, and that a public leaderboard wears like a test set (V60), each compared submission being a choice made with it. **Defects of this wave's own making, caught before the push**: (1) the refit was first sent the configuration held by the page; the worker now keeps the one its artifacts were trained with, so a refit can never repeat another run; (2) the blend has no entry in the model zoo, so asking to refit it raised an internal error: the option is disabled for it with the reason (its members are picked on validation rows); (3) on a phone the three fields pushed the whole page sideways, caught on the screenshot review: they now shrink with their column; (4) the first end-to-end run passed nothing because it ran against the previous build. **What this wave deliberately does not do**: send anything to a platform or know its rules; promise a score or a rank; measure the refitted model, which has no test set left; refit a blend, or build a submission from a re-imported model, which has no training rows; apply the decision threshold or the multiclass policy to the label written (it is the raw prediction, as in the scored CSV); blend several models into one submission. The glass box, last part of F8. 1251 unit tests, 132 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | | **V66 — delivered** | **The glass box: one row, replayed through the model.** The third and last part of F8. For a row typed in the what-if panel the lab gave a prediction and a Shapley explanation (V28), what weighed. It did not show what happened. A replay now tells it for four families: the questions a decision tree asked and the training rows that share its leaf, the count of a forest's trees, the five rows a k-NN compared the row to, the prediction a boosting builds round after round. **What the code said before anything was written**: F8 spoke of « hand-written » models, and the tree and the forest come from ml-cart and ml-random-forest. A replay stays possible by walking their exported JSON, which is also what a model rebuilt from a helper core holds (V37). No family kept what a replay needs: the boosting only summed all its trees, the k-NN search dropped the index of its neighbours, the tree exposed no path. **One rule: the replay is the model.** Nothing is approximated, and the last step of a replay must equal `predict`. **Measured before the panel, six demos, five seeds, 2,560 test rows**: zero mismatch for the tree, the k-NN and the boosting, in binary, multiclass and regression. **The forest failed on three rows of titanic**: on a 20 to 20 vote the replay kept the smallest class, and the library keeps the first class to reach the top count, in the order of its trees. The replay does the same now, flags a tied vote, and the forest, which the plan kept only « if its count is exact », is in. **Questions in the units of the file.** A threshold on a standardized number is given its unit back, a 0/1 column becomes « is it this category? ». Measured on the tree's paths: every question translates on five demos, and 6 of 1,878 on reviews, whose columns are free text: there a question is the weight of a word, shown as the model sees it. A path is 1.9 questions long on fraud and 8 on energy on average, 10 at most. **Two findings the replay made visible.** (1) **k-NN ties**: the next row is exactly as far as the last neighbour kept on 17 % of titanic's test rows and 39 % of energy's, 0 to 1 % elsewhere. V64 had met this from the outside, as a gap with scikit-learn; the panel flags it on the row concerned. (2) **The boosting's last rounds**: read on the validation rows, the error was lowest well before round 120 (more than 2 % under the final error) on 17 runs of 30. On titanic the lowest log loss came at rounds 21 to 25 on three seeds (0.43 against 0.45 to 0.48 at the end); on reviews at round 11 on one seed (0.67 against 0.79, worse than the 0.69 it started from); on energy the boosting never beat its starting point on three seeds of five. The lab runs its rounds to the end and has no early stopping. The panel shows the curve, on validation and never on the test set, and sets nothing; the fix is proposed as F9 below, to be measured on its own. **Defects of this wave's own making, caught before the push**: (1) the forest's tie, above; (2) ml-random-forest nests its parameters under `baseModel`, which the first replay did not read; (3) the three rounds told for the boosting were always rounds 1, 2 and 3, the same tree three times, seen on the screenshot review: the rounds told are now the largest movers among those asking different questions, each with the count of other rounds that asked the same. **What this wave deliberately does not do**: replay how a model was trained; replay a network, a linear model or a blend (the panel says why for each), or a forest tree by tree; stop the boosting early or change any setting from what is seen; replay a re-imported model, which carries no training rows; draw a whole tree. A replay is shown and saved nowhere: it is absent from the saved run and from the share link. A trace is not a cause, and the panel says so. F8 is complete. 1294 unit tests, 133 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | +| **V67 — delivered** | **Early stopping for the boosting: measured, and not built.** F9, opened by V66, which had shown the boosting's validation error lowest well before its 120th round on 17 runs of 30. The obvious reading is overfitting and the obvious fix is to stop at that round. This wave measured the fix before writing it, and the measure says no. **The trap seen before measuring**: the validation rows choose the family (V35, V53). A boosting that also chooses its number of rounds on them would be scored on rows it tuned itself on, against families that tuned nothing there. So three stops were compared, not one: (A) at the lowest error on the run's validation rows; (B) at the lowest error on 15 or 20 % of the training rows held out inside the boosting, keeping that smaller model; (C) the round found by B, applied to the model fitted on all the training rows. Plus a stop by patience (10 or 20 rounds without progress) against the global minimum. **Measured on six demos, ten splits each, a quarter of the rows kept away from everything, against the 120 fixed rounds, on the metric the leaderboard ranks on.** Titanic: A gains 0.4 accuracy point, better on 5 splits and worse on 4. Fraud, iris and mpg: no change. **Reviews: A loses 4.0 accuracy points** (worse 6, better 2), B and C lose 5 to 7. Energy: RMSE 3.5 % lower with A (better 9, worse 1), where the stop is often round 0, the mean of the training rows: there the boosting does no better than the baseline, and it is the family chosen on validation on none of the ten splits. B and C did no better than A on five demos of six, and B, which gives up rows, was the worst of the three on four demos. Patience and global minimum gave the same result within noise. **What does improve is the log loss**: better on 10 splits of 10 on titanic (0.02) and on reviews (0.10). Fewer rounds give better probabilities and no better predictions, and on reviews both at once: a lower log loss and 4 points less accuracy. V66's curve is a log loss in classification, which is why it turned early while the classes predicted kept improving or held. **The trap was real**: with A the boosting's validation score rose above its score on the private rows by 0.9 accuracy point on titanic, 6 points on reviews, 0.04 of RMSE on mpg and 0.18 on energy, beyond what it already did at round 120. On the choice of the family it changed little, and never for the better: the family chosen on validation scored 0.828 on the private rows with A against 0.832 without on titanic, an RMSE of 3.034 against 3.023 on mpg, the same elsewhere. **Decision, by the rule the plan had set (« ship only what the measure justifies »)**: nothing changes in training. The boosting keeps its 120 rounds, no figure of any run moves, and no golden reference is regenerated. **What ships**: the measure, written in `gbdt.ts` where the next reader will look for early stopping; the glass box's sentence, which said the later rounds « did not help » and let the reader conclude that fewer rounds would predict better, now says what was measured; the how-to page and the limits page in both languages carry the table. F9 is closed. **Defects of this wave's own making**: V66's own wording, above, was the defect: it read a log-loss curve as a verdict on the predictions. **What this wave deliberately does not do**: stop early when the user ranks on log loss, where the measure does favour fewer rounds on two demos of four: the lab ranks on accuracy or RMSE by default, and a stop tied to the ranking metric is another design, with the validation trap to solve first; tune depth or learning rate; touch the network or the forest. 1294 unit tests, 133 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | Proposed features, ranked by value over effort. Each reuses a module that exists; none is scheduled. -| Feature | Content | Reuses | -| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | -| F7 | **Counterfactuals.** The smallest change that flips a decision: a deterministic search, columns declared immutable, values kept inside observed ranges, a named refusal when nothing plausible exists. | `insights.ts`, `shapley.ts`, `score.ts` | -| F9 | **Early stopping for the boosting, measured first.** V66 showed that on 17 runs of 30 the validation error of the boosting was lowest well before its 120th round, and that on energy it never beat its starting point on three splits of five. Stop at the best validation round, or keep 120: to decide on rows kept away from the choice, as V65 did for the refit, since the validation rows already pick the family. | `gbdt.ts`, `glass-box.ts` (`stagedValidation`) | +| Feature | Content | Reuses | +| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- | +| F7 | **Counterfactuals.** The smallest change that flips a decision: a deterministic search, columns declared immutable, values kept inside observed ranges, a named refusal when nothing plausible exists. | `insights.ts`, `shapley.ts`, `score.ts` | Not proposed, because the project already refused them: an AutoML mode, a tenth model family, model-based imputation, tabular deep learning, telemetry. diff --git a/README.md b/README.md index 6a13697..504c039 100644 --- a/README.md +++ b/README.md @@ -72,28 +72,28 @@ The project follows three non-negotiable principles: ### ML Lab — `/ml` -| Area | What it does | -| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Data in | Drag & drop CSV/Excel (parsed in a worker), demo datasets, per-column profiling, automatic task detection, smart exclusions and **target-leakage detection**; **a French export reads correctly** — encoding settled by decoding and failing (not sniffing), delimiter and decimal separator detected per column and announced with the count that proved them; **free-text columns** join the pipeline as hand-written TF-IDF (FR/EN tokenization, capped vocabulary, fitted on the training split only); the same reader and the same **20-million-cell memory budget** serve the ML Lab, the Data Studio, the data assistant and batch scoring, so a file is never refused by name in one section and left to exhaust the tab in another | -| Models | 8 classifiers / 7 regressors trained live: naive baseline, linear/logistic regression, k-NN, Gaussian Naive Bayes, decision tree, random forest, **hand-written histogram gradient boosting** (LightGBM-style) and **MLP** (seeded He init, Adam) | -| Leaderboard | Accuracy/F1/ROC-AUC/log-loss or RMSE/MAE/R², delta vs baseline, train time, inference latency p50/p95, **95% bootstrap intervals** with a paired winner-vs-baseline verdict | -| Prediction sets | **Split conformal prediction** for the inspected model: a set of classes or a value with a margin, at an 80, 90 or 95 % coverage target, with the **coverage measured on the test split shown beside the target** and its interval. The threshold is calibrated on one half of the test split and measured on the other, both ways, never on validation: validation chose the winner, and a threshold calibrated there covered 77.4 % for a target of 80 on the mpg sample. A level the split cannot carry is refused with the numbers | -| Understanding | Confusion matrix, ROC, permutation importance, partial dependence, live what-if with **exact Shapley explanations**, **signed word effects** for text columns (which words push the answer up or down), plain-language read (FR/EN, rule-generated) | -| Where it fails | **Per-segment analysis**: the test set sliced by every categorical column — including excluded ones, where proxy effects hide — worst gaps first | -| Group audit | **Does the model treat groups differently?** On one column the user names (the lab never picks it), each group gets its selection rate, recall and false-alarm rate with a Wilson interval, and every gap to the largest group carries its own interval: real, or inside the noise, which the panel refuses to read as equal treatment. Groups under 20 test rows are listed, not compared. A finding, not a verdict: no legal threshold, no correction | -| Test-set ledger | **Has this test set been used too much?** Comparing runs on one test set and keeping the best is choosing with it. The ledger counts the **distinct configurations** a test set has scored (retraining the same one counts for nothing), per file and target, in a table that survives deleted runs. Marks measured on the demos: the best of 5 runs sits 0.2 to 0.8 times the split-to-split noise above its real level, up to that noise from 10. From the first mark it offers a **fresh split** (the next seed), says it is the same rows redealt and not new data, and refuses the draw on a chronological split, which has one test set. | -| Production check | **Is this model still good on the file that just arrived?** Every scored batch gets one verdict (stable, watch, retrain) from three signals: the columns, the model's answers, and the performance when the file carries labels. The reference travels **inside the model export** (format 5: bin edges, shares and counts, never a row), so a re-imported model checks a batch months later. A drift counts only above its own **simulated noise ceiling**: the usual fixed PSI thresholds raised a strong alert on 8 to 78 % of 100-row batches in which nothing had moved, the ceiling on 1.5 to 6 %. And only a **measured drop** on labelled rows says « retrain »: a drift alone stops at « watch ». | -| Imbalance | Precision-recall curve (AP), calibration curve (Brier), **cost-priced decision threshold** with the optimal cut computed by exhaustive sweep, and a **multiclass decision policy** — one threshold per class, normalized-margin tie-break, explicit abstention instead of a silent argmax fallback, tuned on validation and frozen before the test reveal. The cost-optimal cut is **suggested on validation and measured on test**: picked on the test set it was fitted to it, and the cost shown read 33 to 60 % below the honest one on the fraud sample | -| Tuning | Seeded random search scored by stratified 3-fold cross-validation, pipeline refitted inside each fold — the test set is scored exactly once | -| More data? | **Learning curve** on demand: one model retrained on growing seeded nested fractions, 95% bootstrap band, plain verdict — still climbing (collect more rows) or flattened (work on features) — including whether an announced training cap costs accuracy | -| No target? | Seeded k-means (k chosen by silhouette) + power-iteration PCA projection, groups described in plain language; date column? **Holt-Winters forecasting** validated by rolling-origin backtest | -| MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts, or **up to six at once** read against the oldest of the selection; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining | -| Python bridge | **A run, redone with scikit-learn.** One readable script carries the row numbers of each split and the preprocessing, and translates each model to its nearest estimator. It does not promise the lab's figures: it prints them next to its own, with the gap. Measured on 30 runs: the encoded matrix is equal to within 2.9e-14, the baseline, the linear regression and the naive Bayes agree to the last digit, k-NN agrees until two neighbours tie, and the hand-written families differ by up to several test rows on one split. A column name is never code: the run travels as one JSON block. | -| Competition path | **An unlabelled file becomes a submission.** Under a scored batch, one block writes an identifier and a prediction under the names the platform expects: a class, a probability or a value, every row kept in its order, missing and repeated identifiers counted. The lab guesses neither the identifier nor the names. The same family can be **fitted again on every labelled row**, announced as a model nobody measured: on the demos, against rows kept away from both models, it gained 5 accuracy points on the smallest text file, 0.8 on titanic, 2 to 3 % of RMSE on two regressions, nothing on two others, and lost one time in five to one in three even where it paid | -| Glass box | **One row, replayed through the model.** Beside the Shapley explanation (what weighed), a replay tells what happened: the questions a decision tree asked, in the units of the file, and the training rows that share its leaf; the count of a forest's 40 trees; the five rows of the file a k-NN compared it to, with its ties flagged; the prediction a boosting builds round after round, and its error on the validation rows. The replay walks the model itself: measured on 2,560 test rows of the demos, its last step is the model's answer on every one | -| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met | -| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) | -| Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner | +| Area | What it does | +| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Data in | Drag & drop CSV/Excel (parsed in a worker), demo datasets, per-column profiling, automatic task detection, smart exclusions and **target-leakage detection**; **a French export reads correctly** — encoding settled by decoding and failing (not sniffing), delimiter and decimal separator detected per column and announced with the count that proved them; **free-text columns** join the pipeline as hand-written TF-IDF (FR/EN tokenization, capped vocabulary, fitted on the training split only); the same reader and the same **20-million-cell memory budget** serve the ML Lab, the Data Studio, the data assistant and batch scoring, so a file is never refused by name in one section and left to exhaust the tab in another | +| Models | 8 classifiers / 7 regressors trained live: naive baseline, linear/logistic regression, k-NN, Gaussian Naive Bayes, decision tree, random forest, **hand-written histogram gradient boosting** (LightGBM-style) and **MLP** (seeded He init, Adam) | +| Leaderboard | Accuracy/F1/ROC-AUC/log-loss or RMSE/MAE/R², delta vs baseline, train time, inference latency p50/p95, **95% bootstrap intervals** with a paired winner-vs-baseline verdict | +| Prediction sets | **Split conformal prediction** for the inspected model: a set of classes or a value with a margin, at an 80, 90 or 95 % coverage target, with the **coverage measured on the test split shown beside the target** and its interval. The threshold is calibrated on one half of the test split and measured on the other, both ways, never on validation: validation chose the winner, and a threshold calibrated there covered 77.4 % for a target of 80 on the mpg sample. A level the split cannot carry is refused with the numbers | +| Understanding | Confusion matrix, ROC, permutation importance, partial dependence, live what-if with **exact Shapley explanations**, **signed word effects** for text columns (which words push the answer up or down), plain-language read (FR/EN, rule-generated) | +| Where it fails | **Per-segment analysis**: the test set sliced by every categorical column — including excluded ones, where proxy effects hide — worst gaps first | +| Group audit | **Does the model treat groups differently?** On one column the user names (the lab never picks it), each group gets its selection rate, recall and false-alarm rate with a Wilson interval, and every gap to the largest group carries its own interval: real, or inside the noise, which the panel refuses to read as equal treatment. Groups under 20 test rows are listed, not compared. A finding, not a verdict: no legal threshold, no correction | +| Test-set ledger | **Has this test set been used too much?** Comparing runs on one test set and keeping the best is choosing with it. The ledger counts the **distinct configurations** a test set has scored (retraining the same one counts for nothing), per file and target, in a table that survives deleted runs. Marks measured on the demos: the best of 5 runs sits 0.2 to 0.8 times the split-to-split noise above its real level, up to that noise from 10. From the first mark it offers a **fresh split** (the next seed), says it is the same rows redealt and not new data, and refuses the draw on a chronological split, which has one test set. | +| Production check | **Is this model still good on the file that just arrived?** Every scored batch gets one verdict (stable, watch, retrain) from three signals: the columns, the model's answers, and the performance when the file carries labels. The reference travels **inside the model export** (format 5: bin edges, shares and counts, never a row), so a re-imported model checks a batch months later. A drift counts only above its own **simulated noise ceiling**: the usual fixed PSI thresholds raised a strong alert on 8 to 78 % of 100-row batches in which nothing had moved, the ceiling on 1.5 to 6 %. And only a **measured drop** on labelled rows says « retrain »: a drift alone stops at « watch ». | +| Imbalance | Precision-recall curve (AP), calibration curve (Brier), **cost-priced decision threshold** with the optimal cut computed by exhaustive sweep, and a **multiclass decision policy** — one threshold per class, normalized-margin tie-break, explicit abstention instead of a silent argmax fallback, tuned on validation and frozen before the test reveal. The cost-optimal cut is **suggested on validation and measured on test**: picked on the test set it was fitted to it, and the cost shown read 33 to 60 % below the honest one on the fraud sample | +| Tuning | Seeded random search scored by stratified 3-fold cross-validation, pipeline refitted inside each fold — the test set is scored exactly once | +| More data? | **Learning curve** on demand: one model retrained on growing seeded nested fractions, 95% bootstrap band, plain verdict — still climbing (collect more rows) or flattened (work on features) — including whether an announced training cap costs accuracy | +| No target? | Seeded k-means (k chosen by silhouette) + power-iteration PCA projection, groups described in plain language; date column? **Holt-Winters forecasting** validated by rolling-origin backtest | +| MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts, or **up to six at once** read against the oldest of the selection; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining | +| Python bridge | **A run, redone with scikit-learn.** One readable script carries the row numbers of each split and the preprocessing, and translates each model to its nearest estimator. It does not promise the lab's figures: it prints them next to its own, with the gap. Measured on 30 runs: the encoded matrix is equal to within 2.9e-14, the baseline, the linear regression and the naive Bayes agree to the last digit, k-NN agrees until two neighbours tie, and the hand-written families differ by up to several test rows on one split. A column name is never code: the run travels as one JSON block. | +| Competition path | **An unlabelled file becomes a submission.** Under a scored batch, one block writes an identifier and a prediction under the names the platform expects: a class, a probability or a value, every row kept in its order, missing and repeated identifiers counted. The lab guesses neither the identifier nor the names. The same family can be **fitted again on every labelled row**, announced as a model nobody measured: on the demos, against rows kept away from both models, it gained 5 accuracy points on the smallest text file, 0.8 on titanic, 2 to 3 % of RMSE on two regressions, nothing on two others, and lost one time in five to one in three even where it paid | +| Glass box | **One row, replayed through the model.** Beside the Shapley explanation (what weighed), a replay tells what happened: the questions a decision tree asked, in the units of the file, and the training rows that share its leaf; the count of a forest's 40 trees; the five rows of the file a k-NN compared it to, with its ties flagged; the prediction a boosting builds round after round, and its error on the validation rows. The replay walks the model itself: measured on 2,560 test rows of the demos, its last step is the model's answer on every one. The replay showed the boosting's validation error turning well before its last round; early stopping was then measured on rows kept away from everything and **not built**: fewer rounds improved the probabilities, not the predictions | +| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met | +| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) | +| Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner | **What the run gives you, past the ranking:** diff --git a/package-lock.json b/package-lock.json index 486cc62..354c3a2 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "labml", - "version": "1.66.0", + "version": "1.67.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "labml", - "version": "1.66.0", + "version": "1.67.0", "license": "MIT", "dependencies": { "@duckdb/duckdb-wasm": "1.28.0", diff --git a/package.json b/package.json index 153c874..f7e4325 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "labml", "private": true, - "version": "1.66.0", + "version": "1.67.0", "license": "MIT", "type": "module", "engines": { diff --git a/src/content/docs/en/limits.md b/src/content/docs/en/limits.md index 5a7ad95..d75c8fa 100644 --- a/src/content/docs/en/limits.md +++ b/src/content/docs/en/limits.md @@ -27,46 +27,46 @@ They are not equivalent, and mixing them would be dishonest: ## Set aside by choice -| Set aside | Why | -| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| An AutoML « we handle everything » mode | The exact opposite of a lab that shows its decisions | -| A tenth model family | Nine is plenty; a tenth improves neither honesty nor understanding | -| Tabular deep learning | High cost, no gain at this scale, and no longer hand-written | -| A spreadsheet-style cell editor | Hand edits break reproducibility — the recipe is the only record | -| Fuzzy deduplication | Guaranteed false positives on names and addresses: silently merging two real people | -| Model-based imputation | Opaque, and it fabricates plausible values — refused twice, in V39 and V40 | -| Caching models across runs | The cache key would be the whole configuration plus the data; a stale hit is a wrong leaderboard with no warning | -| Guessing a file's locale | The browser has no relationship to the file dropped into it | -| Making the absence indicator mandatory | It would add columns to every existing recipe: imposing is not announcing | -| Translating the changelog and prerendering the French docs | The changelog is an engineering log, not editorial content; without per-language routes, hreflang alternates would lie | -| Catching async errors and shipping crash reports | React boundaries cannot see callbacks or event handlers — claiming otherwise would be false; and nothing leaves the browser, telemetry included | -| A daily Dependabot and `npm audit fix --force` | A daily cadence drowns the signal; `--force` moves majors without reading the break. The remaining advisories sit in build tooling, and we say so instead of hiding it | -| Picking multiclass thresholds automatically | The lab shows its decisions instead of making them for you; and abstention is neither uncertainty calibration nor a conformal guarantee (that guarantee exists on its own panel, as prediction sets) | -| A run id in every worker message | The worker answers in the order it is written to, so the training status is enough to recognise a stale answer. The id becomes necessary the day requests can overlap | -| A « rare class » warning in the interface | A one-row class stays in training, and a target where every class has one row is refused by name; a banner would flag a problem the split already handles | -| Splitting the ML Lab store into independent slices | Every action shares one worker: slicing would have moved code without separating anything | -| Correcting a score for the runs it was compared with | The inflation depends on how close the runs really are (0.2 to 0.8 times the split-to-split noise at 5 runs across the demos), and the lab cannot know that. The test-set ledger counts and warns; the score stays the one measured | -| Fingerprinting a file's content for the ledger | The ledger recognises a file by its name and shape. Two files sharing both would share a count; hashing every file on every load was not worth that rare case | -| Counting a tuning or an analysis in the ledger | They read the test set without choosing between runs; the ledger counts what a comparison between runs can inflate | -| Saying « retrain » from a drift alone | A drift says the inputs or the answers moved, not that the model is wrong more often: on the mpg demo the batch that moved furthest had a lower error. Without labels the production check stops at « watch » | -| Weighting a drift by the importance of its column | The export does not carry importances, and a column the model barely uses can still be the first sign that the population changed. Every column of the model is read the same way | -| A history of the batches checked | Each check reads one batch against the reference. Following a trend across batches needs a store of past checks, which nothing has asked for yet | -| Changing how the PSI is computed | The bins, the smoothing of empty bins and the formula are the ones everyone uses, so a PSI read here compares with one read elsewhere. What changed in V62 is what the number is compared to: its own noise ceiling | -| A verdict on a shift of the missing rate | It is shown beside each column and left to the reader: a column going from 3 % to 0 % missing is a fact about the export, not a distribution to test | -| Hand-written rules in a data contract | A contract freezes what the lab knows how to find: seven rules, the readings, the shares. Custom bounds and expressions are a language of their own, with its parser, its errors and its tests, and nothing has asked for it yet | -| Checking a contract from the ML Lab's scoring panel | A file is checked in the Data Studio, where it already leaves for the lab. The model export keeps its own reference for the batch it scores (V61) | -| A contract that blocks | It reports and stops. A broken contract still loads, cleans and trains: the decision stays with whoever reads the clauses | -| The same figures in Python as in the lab | The exported script carries the rows of each split and the preprocessing exactly, and translates the models: the lab's estimators are written by hand. Measured on 30 runs, three families agree to the last digit and the others differ by up to several test rows on one split. The script prints both figures and the gap instead of promising one | -| A notebook, an environment file or a project as the Python export | One readable `.py` file to put where you like. A notebook format, pinned versions and a folder layout are choices that belong to whoever runs it | -| Guessing a competition's format, or sending the file to it | The lab knows no platform: you name the identifier, the columns and what a prediction is, and the file is downloaded. Nothing leaves the browser | -| A score or a rank for a submission | The lab has neither the platform's metric nor its private rows. It counts the rows written and the identifiers a platform would refuse, nothing more | -| A measure of the model refitted on every row | Fitting on everything spends the test set. The score shown stays the one of the model fitted on 64 % of the rows; on the demos the refit gained from nothing to 5 points, and lost one time in five to one in three even where it paid | -| A submission from a re-imported model, a blend refitted, or a threshold applied | The submission is built from the live run. A re-imported model has no training rows to refit on, a blend picks its members on validation rows, and the label written is the model's raw prediction, as in the scored CSV | -| Replaying how a model was trained | The replay tells how the trained model predicts one row. How each tree was grown, round after round, is another story and a much longer one | -| Replaying a network, a linear model, a blend, or a forest tree by tree | A network has no steps that read as questions, a weighted sum is already read by the Shapley explanation, a blend averages other models, and forty trees are counted, not told. The panel says so for each | -| Stopping the boosting early | The replay shows the validation error round by round, and on the demos its lowest point came well before the last round on 17 runs out of 30. It is shown and sets nothing: changing the number of rounds is a training decision that needs its own measure | -| Replaying a re-imported model, or drawing a whole tree | The replay needs the training rows the run holds, which an export does not carry. It shows the path of one row, not the graph of the tree | -| A generative chat backed by a remote API (V12) | Postponed by product decision (confirmed 09/10/2026). Today's chat runs entirely in the browser; a remote API would send out the question and the column schema, never the data, and would require a consent screen | +| Set aside | Why | +| ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| An AutoML « we handle everything » mode | The exact opposite of a lab that shows its decisions | +| A tenth model family | Nine is plenty; a tenth improves neither honesty nor understanding | +| Tabular deep learning | High cost, no gain at this scale, and no longer hand-written | +| A spreadsheet-style cell editor | Hand edits break reproducibility — the recipe is the only record | +| Fuzzy deduplication | Guaranteed false positives on names and addresses: silently merging two real people | +| Model-based imputation | Opaque, and it fabricates plausible values — refused twice, in V39 and V40 | +| Caching models across runs | The cache key would be the whole configuration plus the data; a stale hit is a wrong leaderboard with no warning | +| Guessing a file's locale | The browser has no relationship to the file dropped into it | +| Making the absence indicator mandatory | It would add columns to every existing recipe: imposing is not announcing | +| Translating the changelog and prerendering the French docs | The changelog is an engineering log, not editorial content; without per-language routes, hreflang alternates would lie | +| Catching async errors and shipping crash reports | React boundaries cannot see callbacks or event handlers — claiming otherwise would be false; and nothing leaves the browser, telemetry included | +| A daily Dependabot and `npm audit fix --force` | A daily cadence drowns the signal; `--force` moves majors without reading the break. The remaining advisories sit in build tooling, and we say so instead of hiding it | +| Picking multiclass thresholds automatically | The lab shows its decisions instead of making them for you; and abstention is neither uncertainty calibration nor a conformal guarantee (that guarantee exists on its own panel, as prediction sets) | +| A run id in every worker message | The worker answers in the order it is written to, so the training status is enough to recognise a stale answer. The id becomes necessary the day requests can overlap | +| A « rare class » warning in the interface | A one-row class stays in training, and a target where every class has one row is refused by name; a banner would flag a problem the split already handles | +| Splitting the ML Lab store into independent slices | Every action shares one worker: slicing would have moved code without separating anything | +| Correcting a score for the runs it was compared with | The inflation depends on how close the runs really are (0.2 to 0.8 times the split-to-split noise at 5 runs across the demos), and the lab cannot know that. The test-set ledger counts and warns; the score stays the one measured | +| Fingerprinting a file's content for the ledger | The ledger recognises a file by its name and shape. Two files sharing both would share a count; hashing every file on every load was not worth that rare case | +| Counting a tuning or an analysis in the ledger | They read the test set without choosing between runs; the ledger counts what a comparison between runs can inflate | +| Saying « retrain » from a drift alone | A drift says the inputs or the answers moved, not that the model is wrong more often: on the mpg demo the batch that moved furthest had a lower error. Without labels the production check stops at « watch » | +| Weighting a drift by the importance of its column | The export does not carry importances, and a column the model barely uses can still be the first sign that the population changed. Every column of the model is read the same way | +| A history of the batches checked | Each check reads one batch against the reference. Following a trend across batches needs a store of past checks, which nothing has asked for yet | +| Changing how the PSI is computed | The bins, the smoothing of empty bins and the formula are the ones everyone uses, so a PSI read here compares with one read elsewhere. What changed in V62 is what the number is compared to: its own noise ceiling | +| A verdict on a shift of the missing rate | It is shown beside each column and left to the reader: a column going from 3 % to 0 % missing is a fact about the export, not a distribution to test | +| Hand-written rules in a data contract | A contract freezes what the lab knows how to find: seven rules, the readings, the shares. Custom bounds and expressions are a language of their own, with its parser, its errors and its tests, and nothing has asked for it yet | +| Checking a contract from the ML Lab's scoring panel | A file is checked in the Data Studio, where it already leaves for the lab. The model export keeps its own reference for the batch it scores (V61) | +| A contract that blocks | It reports and stops. A broken contract still loads, cleans and trains: the decision stays with whoever reads the clauses | +| The same figures in Python as in the lab | The exported script carries the rows of each split and the preprocessing exactly, and translates the models: the lab's estimators are written by hand. Measured on 30 runs, three families agree to the last digit and the others differ by up to several test rows on one split. The script prints both figures and the gap instead of promising one | +| A notebook, an environment file or a project as the Python export | One readable `.py` file to put where you like. A notebook format, pinned versions and a folder layout are choices that belong to whoever runs it | +| Guessing a competition's format, or sending the file to it | The lab knows no platform: you name the identifier, the columns and what a prediction is, and the file is downloaded. Nothing leaves the browser | +| A score or a rank for a submission | The lab has neither the platform's metric nor its private rows. It counts the rows written and the identifiers a platform would refuse, nothing more | +| A measure of the model refitted on every row | Fitting on everything spends the test set. The score shown stays the one of the model fitted on 64 % of the rows; on the demos the refit gained from nothing to 5 points, and lost one time in five to one in three even where it paid | +| A submission from a re-imported model, a blend refitted, or a threshold applied | The submission is built from the live run. A re-imported model has no training rows to refit on, a blend picks its members on validation rows, and the label written is the model's raw prediction, as in the scored CSV | +| Replaying how a model was trained | The replay tells how the trained model predicts one row. How each tree was grown, round after round, is another story and a much longer one | +| Replaying a network, a linear model, a blend, or a forest tree by tree | A network has no steps that read as questions, a weighted sum is already read by the Shapley explanation, a blend averages other models, and forty trees are counted, not told. The panel says so for each | +| Stopping the boosting early | Measured (V67) on rows kept away from everything: stopping at the lowest validation error made predictions no more accurate on four demos of six, 4 points less accurate on reviews, and better only on energy, where the boosting does no better than the mean. It improves the probabilities (log loss), not the classes. Choosing the round on the validation rows would also flatter the boosting on the very rows that choose the family. The lab keeps its 120 rounds | +| Replaying a re-imported model, or drawing a whole tree | The replay needs the training rows the run holds, which an export does not carry. It shows the path of one row, not the graph of the tree | +| A generative chat backed by a remote API (V12) | Postponed by product decision (confirmed 09/10/2026). Today's chat runs entirely in the browser; a remote API would send out the question and the column schema, never the data, and would require a consent screen | ## Dropped after measurement diff --git a/src/content/docs/en/replay-a-prediction.md b/src/content/docs/en/replay-a-prediction.md index 261319b..4c7634c 100644 --- a/src/content/docs/en/replay-a-prediction.md +++ b/src/content/docs/en/replay-a-prediction.md @@ -65,9 +65,27 @@ other rounds that asked this row exactly the same. It also shows the error on the **validation rows** as the rounds add up. On the demos the lowest validation error came well before the last round on 17 runs out of 30, and on the energy demo the boosting never beat its starting point on -three splits out of five. The lab shows this and sets nothing: it always runs -its rounds to the end, and the curve is read on validation, never on the test -set. +three splits out of five. + +That looks like a reason to stop earlier, so it was measured (V67), on a quarter +of the rows kept away from everything, ten splits per demo. Stopping at the +lowest point of the curve, against the 120 rounds the lab runs: + +| Demo | Accuracy or RMSE on the rows kept away | Log loss | +| ------- | -------------------------------------- | ------------------------- | +| titanic | +0.4 point (better 5, worse 4) | better on 10 splits of 10 | +| fraud | no change | slightly worse | +| iris | no change | slightly worse | +| reviews | **4 points lower** (worse 6, better 2) | better on 10 splits of 10 | +| mpg | no change | no change | +| energy | RMSE 3.5 % lower (better 9, worse 1) | the same measure | + +Fewer rounds give better probabilities, not better predictions: the classes +predicted are no more accurate, and on reviews they are less. The one gain is on +energy, where the boosting does no better than the mean and is never the family +the validation rows choose. So the lab keeps its 120 rounds. The curve says +where the error turned; it does not say that fewer rounds would predict better. +It is read on validation, never on the test set. ## Ties in a k-NN diff --git a/src/content/docs/fr/limites.md b/src/content/docs/fr/limites.md index a038366..8e37acb 100644 --- a/src/content/docs/fr/limites.md +++ b/src/content/docs/fr/limites.md @@ -27,46 +27,46 @@ Elles ne se valent pas, et les mélanger serait malhonnête : ## Ce qui a été écarté par choix -| Écarté | Pourquoi | -| -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Un mode AutoML « on s'occupe de tout » | L'exact contraire d'un laboratoire qui montre ses décisions | -| Une dixième famille de modèles | Neuf suffisent ; une dixième n'améliore ni l'honnêteté ni la compréhension | -| Le deep learning tabulaire | Coût élevé, aucun gain à cette échelle, et ce ne serait plus écrit à la main | -| Un éditeur de cellules façon tableur | Les retouches à la main cassent la reproductibilité — la recette est le seul enregistrement | -| La déduplication floue | Faux positifs garantis sur des noms et des adresses : fusionner deux personnes réelles en silence | -| L'imputation par modèle | Opaque, et elle fabrique des valeurs plausibles — refusée deux fois, en V39 et en V40 | -| Cacher les modèles entre exécutions | La clé de cache serait la configuration entière plus les données ; un hit périmé, c'est un classement faux sans avertissement | -| Deviner la locale d'un fichier | Le navigateur n'a aucun rapport avec le fichier qu'on y dépose | -| Rendre l'indicateur d'absence obligatoire | Il ajouterait des colonnes à toute recette existante : imposer n'est pas annoncer | -| Traduire le changelog et prérendre la documentation française | Le changelog est un journal d'ingénierie, pas un contenu éditorial ; sans routes par langue, des alternates hreflang mentiraient | -| Capter les erreurs asynchrones et envoyer des rapports de plantage | Les boundaries React ne voient pas les callbacks ni les gestionnaires d'événements — prétendre le contraire serait faux ; et rien ne quitte le navigateur, télémétrie comprise | -| Un Dependabot quotidien et un `npm audit fix --force` | Une cadence quotidienne noie le signal ; `--force` change des majeures sans lire la rupture. Les advisories restantes touchent les outils de build, et on le dit plutôt que de le masquer | -| Choisir les seuils multiclasses automatiquement | Le laboratoire montre ses décisions au lieu de les prendre à votre place ; et l'abstention n'est ni une calibration d'incertitude ni une garantie conforme (cette garantie existe dans son propre panneau, sous forme d'ensembles de prédiction) | -| Un identifiant de run dans chaque message du worker | Le worker répond dans l'ordre où on lui écrit : l'état de l'entraînement suffit à reconnaître une réponse périmée. L'identifiant deviendra nécessaire le jour où des requêtes pourront se chevaucher | -| Un avertissement « classe rare » dans l'interface | Une classe d'une seule ligne reste à l'entraînement, et une cible où chaque classe n'a qu'une ligne est refusée par son nom ; un bandeau signalerait un problème que la découpe règle déjà | -| Découper le store du ML Lab en tranches indépendantes | Toutes ses actions partagent un seul worker : découper aurait déplacé du code sans rien séparer | -| Corriger un score selon les runs auxquels il a été comparé | L'optimisme dépend de l'écart réel entre les runs (0,2 à 0,8 fois le bruit entre deux découpes à 5 runs sur les démos), et le labo ne peut pas le connaître. Le registre du jeu de test compte et avertit ; le score reste celui qui a été mesuré | -| Prendre l'empreinte du contenu d'un fichier pour le registre | Le registre reconnaît un fichier à son nom et à sa forme. Deux fichiers qui partagent les deux partageraient un compte ; hacher chaque fichier à chaque chargement ne valait pas ce cas rare | -| Compter un réglage ou une analyse dans le registre | Ils lisent le jeu de test sans choisir entre des runs ; le registre compte ce qu'une comparaison entre runs peut gonfler | -| Dire « réentraîner » à partir d'une dérive seule | Une dérive dit que les entrées ou les réponses ont bougé, pas que le modèle se trompe davantage : sur la démo mpg, le lot qui avait le plus bougé avait une erreur plus faible. Sans étiquettes, la vérification en production s'arrête à « surveiller » | -| Pondérer une dérive par l'importance de sa colonne | L'export ne porte pas les importances, et une colonne que le modèle utilise peu peut être le premier signe que la population a changé. Toutes les colonnes du modèle sont lues de la même façon | -| Un historique des lots vérifiés | Chaque vérification lit un lot contre la référence. Suivre une tendance d'un lot à l'autre demande de garder les vérifications passées, ce que rien n'a encore demandé | -| Changer le calcul du PSI | Les tranches, le lissage des tranches vides et la formule sont ceux de tout le monde : un PSI lu ici se compare à un PSI lu ailleurs. Ce que V62 a changé, c'est ce à quoi on compare le chiffre : son propre plafond de bruit | -| Un verdict sur un déplacement du taux de manquants | Il est affiché à côté de chaque colonne et laissé au lecteur : une colonne qui passe de 3 % à 0 % de manquants est un fait sur l'export, pas une distribution à tester | -| Des règles écrites à la main dans un contrat de données | Un contrat fige ce que le labo sait constater : sept règles, les lectures, les parts. Des bornes sur mesure et des expressions sont un langage à part entière, avec son analyseur, ses erreurs et ses tests, et rien ne l'a encore demandé | -| Vérifier un contrat depuis le panneau de scoring du ML Lab | Un fichier se vérifie dans le Data Studio, d'où il part déjà vers le lab. L'export de modèle garde sa propre référence pour le lot qu'il score (V61) | -| Un contrat qui bloque | Il signale et s'arrête. Un contrat rompu se charge, se nettoie et s'entraîne quand même : la décision reste à qui lit les clauses | -| Les mêmes chiffres en Python que dans le labo | Le script exporté reprend à l'identique les lignes de chaque découpe et le prétraitement, et traduit les modèles : les estimateurs du labo sont écrits à la main. Mesuré sur 30 exécutions, trois familles concordent au dernier chiffre et les autres s'écartent jusqu'à plusieurs lignes de test sur une découpe. Le script affiche les deux chiffres et l'écart au lieu d'en promettre un | -| Un notebook, un fichier d'environnement ou un projet comme export Python | Un seul fichier `.py` lisible, à poser où l'on veut. Un format de notebook, des versions épinglées et une arborescence sont des choix qui reviennent à qui l'exécute | -| Deviner le format d'une compétition, ou lui envoyer le fichier | Le labo ne connaît aucune plateforme : vous nommez l'identifiant, les colonnes et ce qu'est une prédiction, et le fichier se télécharge. Rien ne quitte le navigateur | -| Un score ou un rang pour une soumission | Le labo n'a ni la métrique de la plateforme ni ses lignes privées. Il compte les lignes écrites et les identifiants qu'une plateforme refuserait, rien de plus | -| Une mesure du modèle réentraîné sur toutes les lignes | S'entraîner sur tout consomme le jeu de test. Le score affiché reste celui du modèle entraîné sur 64 % des lignes ; sur les démos le réentraînement a gagné de rien à 5 points, et perdu une fois sur cinq à une fois sur trois même là où il rapportait | -| Une soumission depuis un modèle réimporté, un mélange réentraîné, ou un seuil appliqué | La soumission se construit depuis le run en cours. Un modèle réimporté n'a pas de lignes d'entraînement, un mélange choisit ses membres sur les lignes de validation, et la classe écrite est la prédiction brute du modèle, comme dans le CSV scoré | -| Rejouer la façon dont un modèle a été entraîné | Le rejeu raconte comment le modèle entraîné prédit une ligne. Comment chaque arbre a été construit, tour après tour, est une autre histoire, bien plus longue | -| Rejouer un réseau, un modèle linéaire, un mélange, ou une forêt arbre par arbre | Un réseau n'a pas d'étapes qui se lisent comme des questions, une somme pondérée est déjà lue par l'explication de Shapley, un mélange fait la moyenne d'autres modèles, et quarante arbres se comptent, ils ne se racontent pas. Le panneau le dit pour chacun | -| Arrêter le boosting plus tôt | Le rejeu montre l'erreur de validation tour par tour, et sur les démos son point le plus bas arrivait bien avant le dernier tour sur 17 runs sur 30. C'est montré et cela ne règle rien : changer le nombre de tours est une décision d'entraînement qui demande sa propre mesure | -| Rejouer un modèle réimporté, ou dessiner un arbre entier | Le rejeu a besoin des lignes d'entraînement que le run détient, et qu'un export ne porte pas. Il montre le chemin d'une ligne, pas le graphe de l'arbre | -| Un chat génératif branché sur une API distante (V12) | Reporté par décision produit (confirmé le 09/10/2026). Le chat actuel tourne entièrement dans le navigateur ; une API distante ferait sortir la question et le schéma des colonnes, jamais les données, et exigerait un écran de consentement | +| Écarté | Pourquoi | +| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Un mode AutoML « on s'occupe de tout » | L'exact contraire d'un laboratoire qui montre ses décisions | +| Une dixième famille de modèles | Neuf suffisent ; une dixième n'améliore ni l'honnêteté ni la compréhension | +| Le deep learning tabulaire | Coût élevé, aucun gain à cette échelle, et ce ne serait plus écrit à la main | +| Un éditeur de cellules façon tableur | Les retouches à la main cassent la reproductibilité — la recette est le seul enregistrement | +| La déduplication floue | Faux positifs garantis sur des noms et des adresses : fusionner deux personnes réelles en silence | +| L'imputation par modèle | Opaque, et elle fabrique des valeurs plausibles — refusée deux fois, en V39 et en V40 | +| Cacher les modèles entre exécutions | La clé de cache serait la configuration entière plus les données ; un hit périmé, c'est un classement faux sans avertissement | +| Deviner la locale d'un fichier | Le navigateur n'a aucun rapport avec le fichier qu'on y dépose | +| Rendre l'indicateur d'absence obligatoire | Il ajouterait des colonnes à toute recette existante : imposer n'est pas annoncer | +| Traduire le changelog et prérendre la documentation française | Le changelog est un journal d'ingénierie, pas un contenu éditorial ; sans routes par langue, des alternates hreflang mentiraient | +| Capter les erreurs asynchrones et envoyer des rapports de plantage | Les boundaries React ne voient pas les callbacks ni les gestionnaires d'événements — prétendre le contraire serait faux ; et rien ne quitte le navigateur, télémétrie comprise | +| Un Dependabot quotidien et un `npm audit fix --force` | Une cadence quotidienne noie le signal ; `--force` change des majeures sans lire la rupture. Les advisories restantes touchent les outils de build, et on le dit plutôt que de le masquer | +| Choisir les seuils multiclasses automatiquement | Le laboratoire montre ses décisions au lieu de les prendre à votre place ; et l'abstention n'est ni une calibration d'incertitude ni une garantie conforme (cette garantie existe dans son propre panneau, sous forme d'ensembles de prédiction) | +| Un identifiant de run dans chaque message du worker | Le worker répond dans l'ordre où on lui écrit : l'état de l'entraînement suffit à reconnaître une réponse périmée. L'identifiant deviendra nécessaire le jour où des requêtes pourront se chevaucher | +| Un avertissement « classe rare » dans l'interface | Une classe d'une seule ligne reste à l'entraînement, et une cible où chaque classe n'a qu'une ligne est refusée par son nom ; un bandeau signalerait un problème que la découpe règle déjà | +| Découper le store du ML Lab en tranches indépendantes | Toutes ses actions partagent un seul worker : découper aurait déplacé du code sans rien séparer | +| Corriger un score selon les runs auxquels il a été comparé | L'optimisme dépend de l'écart réel entre les runs (0,2 à 0,8 fois le bruit entre deux découpes à 5 runs sur les démos), et le labo ne peut pas le connaître. Le registre du jeu de test compte et avertit ; le score reste celui qui a été mesuré | +| Prendre l'empreinte du contenu d'un fichier pour le registre | Le registre reconnaît un fichier à son nom et à sa forme. Deux fichiers qui partagent les deux partageraient un compte ; hacher chaque fichier à chaque chargement ne valait pas ce cas rare | +| Compter un réglage ou une analyse dans le registre | Ils lisent le jeu de test sans choisir entre des runs ; le registre compte ce qu'une comparaison entre runs peut gonfler | +| Dire « réentraîner » à partir d'une dérive seule | Une dérive dit que les entrées ou les réponses ont bougé, pas que le modèle se trompe davantage : sur la démo mpg, le lot qui avait le plus bougé avait une erreur plus faible. Sans étiquettes, la vérification en production s'arrête à « surveiller » | +| Pondérer une dérive par l'importance de sa colonne | L'export ne porte pas les importances, et une colonne que le modèle utilise peu peut être le premier signe que la population a changé. Toutes les colonnes du modèle sont lues de la même façon | +| Un historique des lots vérifiés | Chaque vérification lit un lot contre la référence. Suivre une tendance d'un lot à l'autre demande de garder les vérifications passées, ce que rien n'a encore demandé | +| Changer le calcul du PSI | Les tranches, le lissage des tranches vides et la formule sont ceux de tout le monde : un PSI lu ici se compare à un PSI lu ailleurs. Ce que V62 a changé, c'est ce à quoi on compare le chiffre : son propre plafond de bruit | +| Un verdict sur un déplacement du taux de manquants | Il est affiché à côté de chaque colonne et laissé au lecteur : une colonne qui passe de 3 % à 0 % de manquants est un fait sur l'export, pas une distribution à tester | +| Des règles écrites à la main dans un contrat de données | Un contrat fige ce que le labo sait constater : sept règles, les lectures, les parts. Des bornes sur mesure et des expressions sont un langage à part entière, avec son analyseur, ses erreurs et ses tests, et rien ne l'a encore demandé | +| Vérifier un contrat depuis le panneau de scoring du ML Lab | Un fichier se vérifie dans le Data Studio, d'où il part déjà vers le lab. L'export de modèle garde sa propre référence pour le lot qu'il score (V61) | +| Un contrat qui bloque | Il signale et s'arrête. Un contrat rompu se charge, se nettoie et s'entraîne quand même : la décision reste à qui lit les clauses | +| Les mêmes chiffres en Python que dans le labo | Le script exporté reprend à l'identique les lignes de chaque découpe et le prétraitement, et traduit les modèles : les estimateurs du labo sont écrits à la main. Mesuré sur 30 exécutions, trois familles concordent au dernier chiffre et les autres s'écartent jusqu'à plusieurs lignes de test sur une découpe. Le script affiche les deux chiffres et l'écart au lieu d'en promettre un | +| Un notebook, un fichier d'environnement ou un projet comme export Python | Un seul fichier `.py` lisible, à poser où l'on veut. Un format de notebook, des versions épinglées et une arborescence sont des choix qui reviennent à qui l'exécute | +| Deviner le format d'une compétition, ou lui envoyer le fichier | Le labo ne connaît aucune plateforme : vous nommez l'identifiant, les colonnes et ce qu'est une prédiction, et le fichier se télécharge. Rien ne quitte le navigateur | +| Un score ou un rang pour une soumission | Le labo n'a ni la métrique de la plateforme ni ses lignes privées. Il compte les lignes écrites et les identifiants qu'une plateforme refuserait, rien de plus | +| Une mesure du modèle réentraîné sur toutes les lignes | S'entraîner sur tout consomme le jeu de test. Le score affiché reste celui du modèle entraîné sur 64 % des lignes ; sur les démos le réentraînement a gagné de rien à 5 points, et perdu une fois sur cinq à une fois sur trois même là où il rapportait | +| Une soumission depuis un modèle réimporté, un mélange réentraîné, ou un seuil appliqué | La soumission se construit depuis le run en cours. Un modèle réimporté n'a pas de lignes d'entraînement, un mélange choisit ses membres sur les lignes de validation, et la classe écrite est la prédiction brute du modèle, comme dans le CSV scoré | +| Rejouer la façon dont un modèle a été entraîné | Le rejeu raconte comment le modèle entraîné prédit une ligne. Comment chaque arbre a été construit, tour après tour, est une autre histoire, bien plus longue | +| Rejouer un réseau, un modèle linéaire, un mélange, ou une forêt arbre par arbre | Un réseau n'a pas d'étapes qui se lisent comme des questions, une somme pondérée est déjà lue par l'explication de Shapley, un mélange fait la moyenne d'autres modèles, et quarante arbres se comptent, ils ne se racontent pas. Le panneau le dit pour chacun | +| Arrêter le boosting plus tôt | Mesuré (V67) sur des lignes tenues à l'écart de tout : s'arrêter à l'erreur de validation la plus basse n'a pas rendu les prédictions plus justes sur quatre démos sur six, les a rendues moins justes de 4 points sur reviews, et meilleures seulement sur energy, où le boosting ne fait pas mieux que la moyenne. Cela améliore les probabilités (log loss), pas les classes. Choisir le tour sur les lignes de validation flatterait en plus le boosting sur les lignes mêmes qui choisissent la famille. Le labo garde ses 120 tours | +| Rejouer un modèle réimporté, ou dessiner un arbre entier | Le rejeu a besoin des lignes d'entraînement que le run détient, et qu'un export ne porte pas. Il montre le chemin d'une ligne, pas le graphe de l'arbre | +| Un chat génératif branché sur une API distante (V12) | Reporté par décision produit (confirmé le 09/10/2026). Le chat actuel tourne entièrement dans le navigateur ; une API distante ferait sortir la question et le schéma des colonnes, jamais les données, et exigerait un écran de consentement | ## Ce qui a été abandonné après mesure diff --git a/src/content/docs/fr/rejouer-une-prediction.md b/src/content/docs/fr/rejouer-une-prediction.md index 53ad68c..08cea4c 100644 --- a/src/content/docs/fr/rejouer-une-prediction.md +++ b/src/content/docs/fr/rejouer-une-prediction.md @@ -71,9 +71,29 @@ ligne. Il montre aussi l'erreur sur les **lignes de validation** à mesure que les tours s'ajoutent. Sur les démos, l'erreur de validation la plus basse arrivait bien avant le dernier tour sur 17 runs sur 30, et sur la démo energy le boosting n'a -jamais battu son point de départ sur trois découpes sur cinq. Le labo le montre -et ne règle rien : il fait toujours ses tours jusqu'au bout, et la courbe se lit -sur la validation, jamais sur le jeu de test. +jamais battu son point de départ sur trois découpes sur cinq. + +Cela ressemble à une raison de s'arrêter plus tôt, et cela a donc été mesuré +(V67), sur un quart des lignes tenu à l'écart de tout, dix découpes par démo. +S'arrêter au point le plus bas de la courbe, face aux 120 tours que fait le +labo : + +| Démo | Exactitude ou RMSE sur les lignes tenues à l'écart | Log loss | +| ------- | -------------------------------------------------- | -------------------------------- | +| titanic | +0,4 point (mieux 5 fois, pire 4) | meilleure sur 10 découpes sur 10 | +| fraud | aucun changement | un peu moins bonne | +| iris | aucun changement | un peu moins bonne | +| reviews | **4 points de moins** (pire 6 fois, mieux 2) | meilleure sur 10 découpes sur 10 | +| mpg | aucun changement | aucun changement | +| energy | RMSE 3,5 % plus basse (mieux 9 fois, pire 1) | la même mesure | + +Moins de tours donnent de meilleures probabilités, pas de meilleures +prédictions : les classes prédites ne sont pas plus justes, et sur reviews elles +le sont moins. Le seul gain est sur energy, où le boosting ne fait pas mieux que +la moyenne et n'est jamais la famille que les lignes de validation choisissent. +Le labo garde donc ses 120 tours. La courbe dit où l'erreur a tourné ; elle ne +dit pas que moins de tours prédiraient mieux. Elle se lit sur la validation, +jamais sur le jeu de test. ## Les ex æquo d'un k-NN diff --git a/src/features/ml/components/insights/GlassBoxView.test.tsx b/src/features/ml/components/insights/GlassBoxView.test.tsx index d9a629c..daf67d2 100644 --- a/src/features/ml/components/insights/GlassBoxView.test.tsx +++ b/src/features/ml/components/insights/GlassBoxView.test.tsx @@ -224,7 +224,7 @@ describe('GlassBoxView', () => { 'On the 143 validation rows, the log loss was lowest at round 22 (0.433) and is 0.449 after round 120', ); expect(screen.getByTestId('glass-validation')).toHaveTextContent( - 'never on the test set, and it sets nothing', + 'never on the test set, and it sets nothing: the lab runs its rounds to the end, and that was measured', ); expect(screen.getAllByRole('img')).toHaveLength(2); }); diff --git a/src/features/ml/train/gbdt.ts b/src/features/ml/train/gbdt.ts index 0aa6242..923d1d3 100644 --- a/src/features/ml/train/gbdt.ts +++ b/src/features/ml/train/gbdt.ts @@ -3,6 +3,29 @@ * LightGBM/XGBoost: features are bucketed into quantile bins once, trees are * grown depth-wise on second-order gradients (XGBoost-style gain), and leaves * take Newton values. Deterministic by construction — no randomness at all. + * + * V67: why the rounds are run to the end. V66 showed the validation error of + * the boosting lowest well before round 120 on 17 runs of 30, which reads as a + * case for early stopping. Measured before building it: six demos, ten splits + * each, a quarter of the rows kept away from everything, the 120 rounds + * against a stop at the lowest validation error (and against a stop chosen on + * rows held out inside the training rows, with or without a refit on all of + * them). On the metric the leaderboard ranks on: + * + * titanic +0.4 accuracy point (better 5, worse 4) log loss better 10/10 + * fraud no change log loss slightly worse + * iris no change log loss slightly worse + * reviews -4.0 accuracy points (worse 6, better 2) log loss better 10/10 + * mpg no change in RMSE + * energy RMSE 3.5 % lower (better 9, worse 1) + * + * Fewer rounds give better probabilities and no better predictions; the one + * gain is on energy, where the boosting does no better than the mean and is + * never the family the validation rows choose. Stops chosen on inner held-out + * rows did no better than the validation stop on five demos of six. And a stop chosen on + * the validation rows flatters the boosting on the rows that choose the + * family: its validation score rose above its private score by 0.9 accuracy + * point on titanic and 6 on reviews. So nothing stops early here. */ export interface GbdtParams { diff --git a/src/features/ml/train/glass-box.ts b/src/features/ml/train/glass-box.ts index 015ebe6..f95867b 100644 --- a/src/features/ml/train/glass-box.ts +++ b/src/features/ml/train/glass-box.ts @@ -32,7 +32,8 @@ * - the boosting's validation error was lowest well before its last round * (more than 2 % under the final error) on 17 runs of 30, and on energy it * never beat its starting point on three seeds of five. The replay shows - * it and sets nothing: stopping early is a training decision (F9). + * it and sets nothing. V67 measured the obvious fix, stopping at that + * round, and found it does not predict better: see `gbdt.ts`. * * Pure and deterministic. */ diff --git a/src/locales/en.json b/src/locales/en.json index 26def76..328ae49 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -758,11 +758,11 @@ }, "validation": { "never": "On the {{rows}} validation rows, the {{metric}} was lowest before the first round ({{start}}) and is {{lastValue}} after round {{last}}: on these rows the boosting never beat its starting point.", - "early": "On the {{rows}} validation rows, the {{metric}} was lowest at round {{best}} ({{bestValue}}) and is {{lastValue}} after round {{last}}: the later rounds fitted the training rows and did not help these.", + "early": "On the {{rows}} validation rows, the {{metric}} was lowest at round {{best}} ({{bestValue}}) and is {{lastValue}} after round {{last}}: past that round this error grew again on these rows.", "late": "On the {{rows}} validation rows, the {{metric}} went from {{start}} to {{lastValue}} after round {{last}}, its lowest being {{bestValue}} at round {{best}}: the rounds kept helping, or cost nothing." }, "validationLabel": "The error on the validation rows, round after round", - "validationNote": "This is read on the validation rows, never on the test set, and it sets nothing: the lab always runs its rounds to the end. Measured on the demos, the lowest validation error came well before the last round on 17 runs out of 30.", + "validationNote": "Read on the validation rows, never on the test set, and it sets nothing: the lab runs its rounds to the end, and that was measured. On the demos, against rows kept away from everything, stopping at the lowest point of this curve made the predictions no more accurate on four demos of six and 4 points less accurate on reviews; it helped only on energy, where the boosting does no better than the mean. What it improves is the probabilities (log loss), not the classes predicted. The curve says where the error turned, not that fewer rounds would predict better.", "same_one": "{{count}} other round asked this row the same questions.", "same_other": "{{count}} other rounds asked this row the same questions." }, diff --git a/src/locales/fr.json b/src/locales/fr.json index cf3e85a..2e358ab 100644 --- a/src/locales/fr.json +++ b/src/locales/fr.json @@ -758,11 +758,11 @@ }, "validation": { "never": "Sur les {{rows}} lignes de validation, la {{metric}} était la plus basse avant le premier tour ({{start}}) et vaut {{lastValue}} après le tour {{last}} : sur ces lignes, le boosting n'a jamais battu son point de départ.", - "early": "Sur les {{rows}} lignes de validation, la {{metric}} était la plus basse au tour {{best}} ({{bestValue}}) et vaut {{lastValue}} après le tour {{last}} : les tours suivants ont collé aux lignes d'entraînement sans aider celles-ci.", + "early": "Sur les {{rows}} lignes de validation, la {{metric}} était la plus basse au tour {{best}} ({{bestValue}}) et vaut {{lastValue}} après le tour {{last}} : passé ce tour, cette erreur est remontée sur ces lignes.", "late": "Sur les {{rows}} lignes de validation, la {{metric}} est passée de {{start}} à {{lastValue}} après le tour {{last}}, son plus bas étant {{bestValue}} au tour {{best}} : les tours ont continué d'aider, ou n'ont rien coûté." }, "validationLabel": "L'erreur sur les lignes de validation, tour après tour", - "validationNote": "Ceci se lit sur les lignes de validation, jamais sur le jeu de test, et ne règle rien : le labo fait toujours ses tours jusqu'au bout. Mesuré sur les démos, l'erreur de validation la plus basse arrivait bien avant le dernier tour sur 17 runs sur 30.", + "validationNote": "Ceci se lit sur les lignes de validation, jamais sur le jeu de test, et ne règle rien : le labo fait ses tours jusqu'au bout, et c'est mesuré. Sur les démos, face à des lignes tenues à l'écart de tout, s'arrêter au point le plus bas de cette courbe n'a pas rendu les prédictions plus justes sur quatre démos sur six, et les a rendues moins justes de 4 points sur reviews ; cela n'a aidé que sur energy, où le boosting ne fait pas mieux que la moyenne. Ce qui s'améliore, ce sont les probabilités (log loss), pas les classes prédites. La courbe dit où l'erreur a tourné, pas que moins de tours prédiraient mieux.", "same_one": "{{count}} autre tour a posé les mêmes questions à cette ligne.", "same_other": "{{count}} autres tours ont posé les mêmes questions à cette ligne." },