diff --git a/CHANGELOG.md b/CHANGELOG.md index b443241..841f926 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and the reason the wave was built. +## V65 — An unlabelled file becomes a submission + +The second part of F8; the glass box stays proposed below. + +_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. + ## V64 — A run, redone in Python The first of the three parts of F8, the two others staying proposed below. diff --git a/PLAN.md b/PLAN.md index b37762d..fb35569 100644 --- a/PLAN.md +++ b/PLAN.md @@ -680,7 +680,7 @@ defect that produces a wrong number comes first), then extends the lab past the export: uncertainty with a guarantee, governance, and monitoring after deployment, with nothing added on a server. -V53 to V64 are delivered. Every other row is a proposal with its launch +V53 to V65 are delivered. Every other row is a proposal with its launch condition, and none starts without an explicit launch command. | Wave | Content | Why | @@ -698,14 +698,15 @@ condition, and none starts without an explicit launch command. | **V62 — delivered** | **A noise ceiling under the Data Studio's drift report.** Not one of the features of the cap: a defect V61 measured and left open. The drift panel (V11) and the profile comparison (V40) graded a PSI with the thresholds everyone uses, 0.1 moderate and 0.25 strong, whatever the size of the files. Over B bins the PSI of two samples of one population averages about (B − 1)(1/n + 1/n_ref), 0.11 for ten bins and 100 rows against 500: the thresholds read the sample size. **Measured first, on two halves of one file, where every alert is false** (Fisher-Yates shuffle, no replacement, 200 reports per case). Share of reports flagging at least one column with the fixed thresholds: **100 %** on the Data Studio's own demo (cafe-sales, 118 rows), 95 % on mpg (398), 52 % on titanic (891), 21 % on fraud (640); with 30 or 60 new rows against the rest, 99 to 100 % on all four. **The rule keeps the three levels and their names; a level above « stable » now requires the PSI to be above its noise ceiling**, the value it stays under, 95 reports in 100, when the two files are two draws of the same data at their real sizes. The level is shared between the columns of a report. A PSI of 0.1 or more that its ceiling explains is shown as « inside the noise », with the ceiling beside it, not as a drift and not as a bare « stable ». **The first version of the rule failed its own measure, on the demo it was written for.** V61 simulated both sides from the reference's shares. On cafe-sales that flagged **48 to 64 %** of no-drift reports, against 2 to 9 % on the three other files. Two causes, one root: a category the reference happens to lack has a share of zero there, so the simulation never produces it, and one stray spelling in the new half (« paris », « LYON ») reads as a strong drift; a notes column with one value on each side had a ceiling of zero. Under the hypothesis being tested the two files are one population, and its best estimate is their pool: both sides are now drawn from the **pooled shares**. Measured again: cafe-sales **3 to 9.5 %**, titanic 0.5 to 4 %, fraud 5 to 8 %, mpg 4.5 to 8.5 %. The production check takes the same correction; its false alerts, re-measured on the V61 bench, are 1.5 to 6 % where they were 1.5 to 6.5 %, and its tests pass unchanged. **And it still sees a real move.** The June demo keeps its planted drifts: `unit_price` at a PSI of 7.1 against a ceiling of 0.6 and `product` at 3.7 against 0.3 are strong, `city` (0.303 against 0.300) stays strong by a hair, `quantity` and `currency` stay stable. Nothing changes for a large file, whose ceiling falls under 0.1. **What is built.** `data/quality/noise.ts` now holds `psi`, `noiseCeiling` (moved from the production check, unchanged) and `pooledShares`; the data layer no longer depends on the ML export for it. `buildDriftReport` and `compareToProfile` grade through `gradeAgainstNoise`; a ceiling is simulated only from a PSI of 0.1, since under it the column is stable either way, so a report on 120 columns takes about 270 ms. `compareToProfile` keeps a `raw` mode for a caller that reads its own ceilings. The panel shows the ceiling under the PSI and counts the columns set aside as noise; its note states the rule instead of the « conventional reading ». **What this wave deliberately does not do**: change how the PSI is computed (bins, smoothing, formula), so that a PSI read here still compares with one read elsewhere; touch the schema diff or the new and vanished categories, which are facts and not estimates; give a verdict on a shift of the missing rate; add an interface to the reference profile. 1175 unit tests, 129 e2e. | Approved 13/11/2026. The second wave in a row whose first rule failed its own measurement before shipping; both times the bench was the only thing that could have said so. | | **V63 — delivered** | **A data contract.** Listed as F5 below until it shipped. The quality report asks whether a file looks clean; a contract asks whether it is still the file that was agreed on. The proposal read « export the validity rules, the consistency rules and the profile as one contract »; reading the code showed what that sentence hid. The seven rules are not declarations: each decides whether it applies, from the column's name and from at least 80 % of its values being plausible (V40), which is right for an unknown file, where a column named `age` holding 20 000 is a duration. **The wave opened by measuring the price of that guard.** One column of a demo degraded by steps of impossible values: at 5, 10 and 15 % the rule flags every one of them; **at 20 % it stops applying and the report says nothing**, and nothing again at 25, 30 and 50 %. On titanic, 134 ages of 999 are reported and 178 are not; the same holds for negative quantities and prices on cafe-sales. Only the future-date rule, which has no plausibility guard, keeps speaking. A file that gets much worse goes quiet. **So a contract freezes what was found on a reference** (the columns, how much is missing, which rules applied and how often they failed, the bins and shares of each distribution) **and requires it of the next file, clause by clause**, applying each rule without its guard. Five kinds of clause, three outcomes: kept, warning, broken. Two rules: a fact is not an estimate, and moving is not breaking. **The second measurement refused two parts of the first design**, on a contract drawn from one half of a file and checked on the other half, where nothing changed (Fisher-Yates, no replacement, 200 checks per file). **(1) « A type that changed is a fact » was wrong.** On cafe-sales 84.5 % of checks came back broken, 157 of them on the type clause: `notes` was categorical in one half and text in the other in 100 checks of 200, `order_id` a number or an identifier in 57. The type the lab infers rests on how many distinct values a file happens to hold. What does not waver is how values read, as numbers, dates or words, and the clause now tests that; the inferred type stays in the contract for the reader. **(2) The missing-values clause broke 6 % of titanic's checks**, each column being tested alone at 95 %; the estimates of one check now share one level, as the drift report does. Measured again: no broken clause on fraud, mpg and energy, 2 % on titanic, drift warnings on 0.5 to 7.5 % of checks. On cafe-sales 53.5 % stay broken, and rightly: the file holds one negative price, and the half without it yields a contract the other half breaks. **How a clause is decided.** A column that is gone, values that no longer read as numbers, an impossible value: broken, with no threshold. A rule clean at the reference is broken by its first failure; a rule that already failed there, only by a measurably higher rate (Newcombe's interval). A share of missing values is broken only when measurably higher, never when lower. A distribution above its noise ceiling (V62) is a warning: the data moved without becoming invalid. A rule that finds nothing it can read warns and claims no failure. A column the contract does not describe is noted. **What is built.** `validity.ts` and `consistency.ts` now separate each rule's predicate (`applyRule`, `applyConsistency`) from its guard (`applicableRules`, `applicableConsistency`); `checkColumn` and `checkConsistency` are those two composed, and their tests pass unchanged. `data/quality/contract.ts`, pure: `buildContract`, `parseContract`, `checkContract`. The contract holds names, bounds, shares and counts, never a row. A file that is not a whole contract is refused and nothing is checked: a clause silently dropped would be a check silently skipped. A panel in the Data Studio exports the contract of the loaded file and reads the loaded file against one; the clauses that did not hold are spelled out by family, broken first, with counts and the first failing rows, and the ones that held are counted. **What this wave deliberately does not do**: let the user write rules (custom bounds and expressions are a language of their own); check a contract from the ML Lab's scoring panel, the model export keeping its own reference (V61); block, repair or version anything. 1204 unit tests, 130 e2e. | Approved 13/11/2026. The fourth wave in a row whose design was corrected by its own measurement before a line of the feature shipped. | | **V64 — delivered** | **A run, redone in Python.** The first of the three parts of F8, the two others staying proposed below. A run stayed inside the browser: its model, its predictions, its report and its card export, the training itself did not. This writes a scikit-learn script that redoes it. **What the code said before anything was written**: the lab's estimators are written by hand or come from ml.js, and `tests/golden/README.md` has held since V42 that requiring the same numbers of a tree, a forest, a boosting or a network « would be a false contract ». So the script could not promise the lab's figures, and the design starts there: carry exactly what can be carried, translate the rest, and print both figures with the gap. **Carried exactly**: the row numbers of the three splits (not a seed that should redo them), and the preprocessing, one transformer per column in the lab's order. **Translated**: each family to its nearest estimator with the lab's settings, labelled on every line it prints. **Measured, six demos and five seeds each, with scikit-learn 1.9.1, and the first run found three faults in the translation.** (1) A numeric column with missing values was encoded differently (0.36 on titanic's `age`): the lab standardizes on the values that are there and then imputes the median, the script did it the other way round and computed the mean on imputed cells. (2) The network, on a regression, gave an RMSE of 219 against the lab's 18 on energy: the lab's network learns a standardized target and converts back, and the script now wraps its estimator the same way. (3) k-NN was labelled « same algorithm » and moved by 0.39 on energy, a file with one feature column: two neighbours at the same distance are not chosen the same way by the two libraries. It has a label of its own. **After the fixes, on the 30 runs**: the matrix the script encodes from the training rows equals the lab's to within **2.9e-14**, free text included. The baseline, the linear regression and the Gaussian naive Bayes give the same test metric on every run. k-NN gives the same on four demos out of six, and differs on titanic (one row of 178 at worst) and energy (up to 1.32 of RMSE). The translated families differ by ±0.06 of accuracy on average at most and by up to 0.15 on one split, which on reviews' 48 test rows is seven rows; the lab's tree is 4.7 points better than scikit-learn's on titanic on average, its network 6 points better on iris. The whole record is committed (`tests/golden/python-bridge.md`) with the two commands that regenerate it, outside CI like the other references. **A column name is never code.** The run travels as one JSON block read by `json.loads`; the rest of the script is the same text for every run. Every apostrophe of the block is written as its JSON escape, so nothing can close the string that holds it; a test gives a column the name of a shell command and checks that Python reads the name and runs none of it. **The script is run in the suite.** `python3 -m py_compile`, then `--check`, a mode that reads the run with the standard library alone: a script that does not compile fails the tests, in CI too. It refuses a file with another number of rows or columns, since its row numbers would point at other rows. **What this wave deliberately does not do**: promise the lab's figures beyond the families where they were measured equal; export a notebook, an environment file or a project; redo the ensemble (named as not redone), the hyperparameter search, the learning curve, the group audit or the production check; read a spreadsheet, which it asks to be saved as CSV. 1224 unit tests, 131 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | +| **V65 — delivered** | **An unlabelled file becomes a submission.** The second part of F8; the glass box stays proposed below. Scoring a batch gave back every original column plus the prediction, the right file to join back into a system and the wrong one for a platform that expects two columns under its own names. Under a scored batch, a block now writes an identifier and a prediction. **What the code said before anything was written**: the lab fits on 64 % of the labelled rows and keeps 16 % to choose and 20 % to measure, and the usage in a competition is to fit again on everything before predicting. That was the real question of the wave, so it was measured first. **Measured, six demos, ten splits each, a quarter of the rows kept away from both models, every family**: the same family fitted on all the remaining rows instead of 64 % of them gained **+5.0 accuracy points on reviews** (180 rows; better 51 times of 70, worse 11), **3 % of RMSE on mpg** (47 of 60, worse 13), **2 % on energy** (44 of 60, worse 16), **+0.8 point on titanic** (41 of 70, worse 20), and **nothing on fraud** (-0.04 point; better 21, worse 25) **nor on iris** (-0.1 point; better 14, worse 17). The plan expected « a small and variable gain, clearer on small files ». The second half was wrong: iris is the smallest demo and gains nothing, because 72 rows already say all there is to say. The gain follows the learning curve, not the size, and the panel now says so and points at the curve (V26). Even where the refit pays it loses one time in five to one in three, so it is **offered and never the default**, and the result line names the model that predicted: the measured one, or its sibling « nobody measured ». A refit costs 18 ms to 3.6 s per family on the demos, done on demand. **The lab guesses nothing.** The user picks the identifier column (or asks for the rows to be numbered from 1), types the two names, and chooses what a prediction is: the class by name, the probability of the second class (binary), one probability per class (three classes or more), or the value (regression). A form with no object for the task is not offered. A column the training file read as an identifier is pointed at when there is exactly one, and not chosen. **No row lost or reordered.** The file has the rows of the scored file in their order; an identifier missing or repeated is written and counted, since a platform would refuse it. The end-to-end test scores a file whose identifiers are out of order and checks the order survives. **A submission is not a score**: the block says the lab knows neither the platform's metric nor its private rows, and that a public leaderboard wears like a test set (V60), each compared submission being a choice made with it. **Defects of this wave's own making, caught before the push**: (1) the refit was first sent the configuration held by the page; the worker now keeps the one its artifacts were trained with, so a refit can never repeat another run; (2) the blend has no entry in the model zoo, so asking to refit it raised an internal error: the option is disabled for it with the reason (its members are picked on validation rows); (3) on a phone the three fields pushed the whole page sideways, caught on the screenshot review: they now shrink with their column; (4) the first end-to-end run passed nothing because it ran against the previous build. **What this wave deliberately does not do**: send anything to a platform or know its rules; promise a score or a rank; measure the refitted model, which has no test set left; refit a blend, or build a submission from a re-imported model, which has no training rows; apply the decision threshold or the multiclass policy to the label written (it is the raw prediction, as in the scored CSV); blend several models into one submission. The glass box, last part of F8. 1251 unit tests, 132 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | Proposed features, ranked by value over effort. Each reuses a module that exists; none is scheduled. -| Feature | Content | Reuses | -| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | -| F7 | **Counterfactuals.** The smallest change that flips a decision: a deterministic search, columns declared immutable, values kept inside observed ranges, a named refusal when nothing plausible exists. | `insights.ts`, `shapley.ts`, `score.ts` | -| F8 | **Glass box and a competition path.** Two parts remain of the three first listed here; the bridge to Python shipped as V64. Replay a hand-written model step by step on the user's data (boosting rounds, tree splits, neighbours); a competition path that scores an unlabeled file into a submission. | `gbdt.ts`, `models.ts`, `score.ts` | +| Feature | Content | Reuses | +| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | +| F7 | **Counterfactuals.** The smallest change that flips a decision: a deterministic search, columns declared immutable, values kept inside observed ranges, a named refusal when nothing plausible exists. | `insights.ts`, `shapley.ts`, `score.ts` | +| F8 | **Glass box.** One part remains of the three first listed here; the bridge to Python shipped as V64 and the competition path as V65. Replay a hand-written model step by step on the user's data (boosting rounds, tree splits, neighbours). | `gbdt.ts`, `models.ts` | Not proposed, because the project already refused them: an AutoML mode, a tenth model family, model-based imputation, tabular deep learning, telemetry. diff --git a/README.md b/README.md index f4a81fe..4010a0b 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,7 @@ The project follows three non-negotiable principles: | No target? | Seeded k-means (k chosen by silhouette) + power-iteration PCA projection, groups described in plain language; date column? **Holt-Winters forecasting** validated by rolling-origin backtest | | MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts, or **up to six at once** read against the oldest of the selection; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining | | Python bridge | **A run, redone with scikit-learn.** One readable script carries the row numbers of each split and the preprocessing, and translates each model to its nearest estimator. It does not promise the lab's figures: it prints them next to its own, with the gap. Measured on 30 runs: the encoded matrix is equal to within 2.9e-14, the baseline, the linear regression and the naive Bayes agree to the last digit, k-NN agrees until two neighbours tie, and the hand-written families differ by up to several test rows on one split. A column name is never code: the run travels as one JSON block. | +| Competition path | **An unlabelled file becomes a submission.** Under a scored batch, one block writes an identifier and a prediction under the names the platform expects: a class, a probability or a value, every row kept in its order, missing and repeated identifiers counted. The lab guesses neither the identifier nor the names. The same family can be **fitted again on every labelled row**, announced as a model nobody measured: on the demos, against rows kept away from both models, it gained 5 accuracy points on the smallest text file, 0.8 on titanic, 2 to 3 % of RMSE on two regressions, nothing on two others, and lost one time in five to one in three even where it paid | | Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met | | Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) | | Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner | @@ -285,7 +286,7 @@ Wikimedia Commons), the portrait is NASA, public domain._ a doc page carries its whole article, in English until the app mounts); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 1224 unit tests and 131 Playwright end-to-end tests across five +- **Quality bar.** 1251 unit tests and 132 Playwright end-to-end tests across five projects — desktop, a phone viewport in English and in French, dark mode, and Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on diff --git a/e2e/submission.spec.ts b/e2e/submission.spec.ts new file mode 100644 index 0000000..259dbe4 --- /dev/null +++ b/e2e/submission.spec.ts @@ -0,0 +1,85 @@ +import { readFileSync, writeFileSync } from 'node:fs'; +import { expect, test } from '@playwright/test'; + +test.use({ locale: 'en-US' }); +test.setTimeout(120_000); + +// NOTE: keep this title ASCII-only: it names the test-results directory, and +// Chromium's setInputFiles silently drops files whose path is non-ASCII. +test('submission: an unlabelled file becomes an identifier and a prediction', async ({ + page, +}, testInfo) => { + await page.goto('/ml?demo=fraud&target=status'); + await page.getByTestId('train-button').click(); + await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60000 }); + // A family with probabilities, and one that can be fitted again. + await page.getByRole('cell', { name: /Logistic regression/ }).click(); + await expect(page.getByRole('button', { name: 'Model (JSON)' })).toBeEnabled(); + + // What a competition hands out: an identifier, the features, no label. The + // identifiers are not in order, so a sorted file would be caught. + const fraud = (await (await page.request.get('/datasets/fraud.csv')).text()).trim().split('\n'); + const ids = Array.from({ length: 40 }, (_, i) => `tx-${(i * 17) % 40}`); + const unlabelled = [ + `tx_id,${fraud[0].split(',').slice(0, 5).join(',')}`, + ...fraud.slice(1, 41).map((line, i) => `${ids[i]},${line.split(',').slice(0, 5).join(',')}`), + ].join('\n'); + const batchPath = testInfo.outputPath('holdout.csv'); + writeFileSync(batchPath, unlabelled); + await page.getByTestId('batch-score').locator('input[type="file"]').setInputFiles(batchPath); + const panel = page.getByTestId('submission-panel'); + await expect(panel).toBeVisible({ timeout: 30000 }); + + // Nothing is chosen for the user: no identifier, no file. + await expect(panel.getByTestId('submission-build')).toBeDisabled(); + await panel.getByTestId('submission-id-column').selectOption('tx_id'); + await expect(panel.getByTestId('submission-id-name')).toHaveValue('tx_id'); + await panel.getByTestId('submission-id-name').fill('TransactionId'); + await panel.getByTestId('submission-prediction-name').fill('is_fraud'); + await panel.getByTestId('submission-build').click(); + + const result = panel.getByTestId('submission-result'); + await expect(result).toHaveAttribute('data-model', 'measured', { timeout: 30000 }); + await expect(result.getByTestId('submission-summary')).toContainText( + '40 rows, the same as the scored file and in its order. Columns: TransactionId, is_fraud.', + ); + await expect(result).toContainText('A submission is not a score'); + + const labelDownload = page.waitForEvent('download'); + await result.getByTestId('submission-download').click(); + const labelFile = await labelDownload; + expect(labelFile.suggestedFilename()).toBe('holdout-submission.csv'); + const labelPath = testInfo.outputPath('labels.csv'); + await labelFile.saveAs(labelPath); + const labels = readFileSync(labelPath, 'utf8').trim().split('\n'); + expect(labels[0]).toBe('TransactionId,is_fraud'); + expect(labels).toHaveLength(41); + expect(labels.slice(1).map((line) => line.split(',')[0])).toEqual(ids); + for (const line of labels.slice(1)) expect(['clean', 'fraud']).toContain(line.split(',')[1]); + + // The same file as probabilities, from the family fitted again on every + // labelled row: announced as a model nobody measured. + await panel.getByTestId('submission-form-probability').check(); + await panel.getByTestId('submission-model-refit').check(); + await panel.getByTestId('submission-build').click(); + await expect(result).toHaveAttribute('data-model', 'refit', { timeout: 60000 }); + await expect(result.getByTestId('submission-model')).toContainText( + 'fitted again on the 640 labelled rows. Nobody measured this model', + ); + const probaDownload = page.waitForEvent('download'); + await result.getByTestId('submission-download').click(); + const probaPath = testInfo.outputPath('probabilities.csv'); + await (await probaDownload).saveAs(probaPath); + const probabilities = readFileSync(probaPath, 'utf8').trim().split('\n'); + expect(probabilities).toHaveLength(41); + expect(probabilities.slice(1).map((line) => line.split(',')[0])).toEqual(ids); + for (const line of probabilities.slice(1)) { + const value = Number(line.split(',')[1]); + expect(value).toBeGreaterThanOrEqual(0); + expect(value).toBeLessThanOrEqual(1); + } + + // Another model inspected: the submission of the previous one is gone. + await page.getByRole('cell', { name: /Naive Bayes/ }).click(); + await expect(page.getByTestId('submission-panel')).toHaveCount(0); +}); diff --git a/package-lock.json b/package-lock.json index c925465..1a22052 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "labml", - "version": "1.64.0", + "version": "1.65.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "labml", - "version": "1.64.0", + "version": "1.65.0", "license": "MIT", "dependencies": { "@duckdb/duckdb-wasm": "1.28.0", diff --git a/package.json b/package.json index 198eb4d..8a10ad6 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "labml", "private": true, - "version": "1.64.0", + "version": "1.65.0", "license": "MIT", "type": "module", "engines": { diff --git a/src/content/docs/en/limits.md b/src/content/docs/en/limits.md index 654c3c1..e45ab46 100644 --- a/src/content/docs/en/limits.md +++ b/src/content/docs/en/limits.md @@ -27,38 +27,42 @@ They are not equivalent, and mixing them would be dishonest: ## Set aside by choice -| Set aside | Why | -| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| An AutoML « we handle everything » mode | The exact opposite of a lab that shows its decisions | -| A tenth model family | Nine is plenty; a tenth improves neither honesty nor understanding | -| Tabular deep learning | High cost, no gain at this scale, and no longer hand-written | -| A spreadsheet-style cell editor | Hand edits break reproducibility — the recipe is the only record | -| Fuzzy deduplication | Guaranteed false positives on names and addresses: silently merging two real people | -| Model-based imputation | Opaque, and it fabricates plausible values — refused twice, in V39 and V40 | -| Caching models across runs | The cache key would be the whole configuration plus the data; a stale hit is a wrong leaderboard with no warning | -| Guessing a file's locale | The browser has no relationship to the file dropped into it | -| Making the absence indicator mandatory | It would add columns to every existing recipe: imposing is not announcing | -| Translating the changelog and prerendering the French docs | The changelog is an engineering log, not editorial content; without per-language routes, hreflang alternates would lie | -| Catching async errors and shipping crash reports | React boundaries cannot see callbacks or event handlers — claiming otherwise would be false; and nothing leaves the browser, telemetry included | -| A daily Dependabot and `npm audit fix --force` | A daily cadence drowns the signal; `--force` moves majors without reading the break. The remaining advisories sit in build tooling, and we say so instead of hiding it | -| Picking multiclass thresholds automatically | The lab shows its decisions instead of making them for you; and abstention is neither uncertainty calibration nor a conformal guarantee (that guarantee exists on its own panel, as prediction sets) | -| A run id in every worker message | The worker answers in the order it is written to, so the training status is enough to recognise a stale answer. The id becomes necessary the day requests can overlap | -| A « rare class » warning in the interface | A one-row class stays in training, and a target where every class has one row is refused by name; a banner would flag a problem the split already handles | -| Splitting the ML Lab store into independent slices | Every action shares one worker: slicing would have moved code without separating anything | -| Correcting a score for the runs it was compared with | The inflation depends on how close the runs really are (0.2 to 0.8 times the split-to-split noise at 5 runs across the demos), and the lab cannot know that. The test-set ledger counts and warns; the score stays the one measured | -| Fingerprinting a file's content for the ledger | The ledger recognises a file by its name and shape. Two files sharing both would share a count; hashing every file on every load was not worth that rare case | -| Counting a tuning or an analysis in the ledger | They read the test set without choosing between runs; the ledger counts what a comparison between runs can inflate | -| Saying « retrain » from a drift alone | A drift says the inputs or the answers moved, not that the model is wrong more often: on the mpg demo the batch that moved furthest had a lower error. Without labels the production check stops at « watch » | -| Weighting a drift by the importance of its column | The export does not carry importances, and a column the model barely uses can still be the first sign that the population changed. Every column of the model is read the same way | -| A history of the batches checked | Each check reads one batch against the reference. Following a trend across batches needs a store of past checks, which nothing has asked for yet | -| Changing how the PSI is computed | The bins, the smoothing of empty bins and the formula are the ones everyone uses, so a PSI read here compares with one read elsewhere. What changed in V62 is what the number is compared to: its own noise ceiling | -| A verdict on a shift of the missing rate | It is shown beside each column and left to the reader: a column going from 3 % to 0 % missing is a fact about the export, not a distribution to test | -| Hand-written rules in a data contract | A contract freezes what the lab knows how to find: seven rules, the readings, the shares. Custom bounds and expressions are a language of their own, with its parser, its errors and its tests, and nothing has asked for it yet | -| Checking a contract from the ML Lab's scoring panel | A file is checked in the Data Studio, where it already leaves for the lab. The model export keeps its own reference for the batch it scores (V61) | -| A contract that blocks | It reports and stops. A broken contract still loads, cleans and trains: the decision stays with whoever reads the clauses | -| The same figures in Python as in the lab | The exported script carries the rows of each split and the preprocessing exactly, and translates the models: the lab's estimators are written by hand. Measured on 30 runs, three families agree to the last digit and the others differ by up to several test rows on one split. The script prints both figures and the gap instead of promising one | -| A notebook, an environment file or a project as the Python export | One readable `.py` file to put where you like. A notebook format, pinned versions and a folder layout are choices that belong to whoever runs it | -| A generative chat backed by a remote API (V12) | Postponed by product decision (confirmed 09/10/2026). Today's chat runs entirely in the browser; a remote API would send out the question and the column schema, never the data, and would require a consent screen | +| Set aside | Why | +| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| An AutoML « we handle everything » mode | The exact opposite of a lab that shows its decisions | +| A tenth model family | Nine is plenty; a tenth improves neither honesty nor understanding | +| Tabular deep learning | High cost, no gain at this scale, and no longer hand-written | +| A spreadsheet-style cell editor | Hand edits break reproducibility — the recipe is the only record | +| Fuzzy deduplication | Guaranteed false positives on names and addresses: silently merging two real people | +| Model-based imputation | Opaque, and it fabricates plausible values — refused twice, in V39 and V40 | +| Caching models across runs | The cache key would be the whole configuration plus the data; a stale hit is a wrong leaderboard with no warning | +| Guessing a file's locale | The browser has no relationship to the file dropped into it | +| Making the absence indicator mandatory | It would add columns to every existing recipe: imposing is not announcing | +| Translating the changelog and prerendering the French docs | The changelog is an engineering log, not editorial content; without per-language routes, hreflang alternates would lie | +| Catching async errors and shipping crash reports | React boundaries cannot see callbacks or event handlers — claiming otherwise would be false; and nothing leaves the browser, telemetry included | +| A daily Dependabot and `npm audit fix --force` | A daily cadence drowns the signal; `--force` moves majors without reading the break. The remaining advisories sit in build tooling, and we say so instead of hiding it | +| Picking multiclass thresholds automatically | The lab shows its decisions instead of making them for you; and abstention is neither uncertainty calibration nor a conformal guarantee (that guarantee exists on its own panel, as prediction sets) | +| A run id in every worker message | The worker answers in the order it is written to, so the training status is enough to recognise a stale answer. The id becomes necessary the day requests can overlap | +| A « rare class » warning in the interface | A one-row class stays in training, and a target where every class has one row is refused by name; a banner would flag a problem the split already handles | +| Splitting the ML Lab store into independent slices | Every action shares one worker: slicing would have moved code without separating anything | +| Correcting a score for the runs it was compared with | The inflation depends on how close the runs really are (0.2 to 0.8 times the split-to-split noise at 5 runs across the demos), and the lab cannot know that. The test-set ledger counts and warns; the score stays the one measured | +| Fingerprinting a file's content for the ledger | The ledger recognises a file by its name and shape. Two files sharing both would share a count; hashing every file on every load was not worth that rare case | +| Counting a tuning or an analysis in the ledger | They read the test set without choosing between runs; the ledger counts what a comparison between runs can inflate | +| Saying « retrain » from a drift alone | A drift says the inputs or the answers moved, not that the model is wrong more often: on the mpg demo the batch that moved furthest had a lower error. Without labels the production check stops at « watch » | +| Weighting a drift by the importance of its column | The export does not carry importances, and a column the model barely uses can still be the first sign that the population changed. Every column of the model is read the same way | +| A history of the batches checked | Each check reads one batch against the reference. Following a trend across batches needs a store of past checks, which nothing has asked for yet | +| Changing how the PSI is computed | The bins, the smoothing of empty bins and the formula are the ones everyone uses, so a PSI read here compares with one read elsewhere. What changed in V62 is what the number is compared to: its own noise ceiling | +| A verdict on a shift of the missing rate | It is shown beside each column and left to the reader: a column going from 3 % to 0 % missing is a fact about the export, not a distribution to test | +| Hand-written rules in a data contract | A contract freezes what the lab knows how to find: seven rules, the readings, the shares. Custom bounds and expressions are a language of their own, with its parser, its errors and its tests, and nothing has asked for it yet | +| Checking a contract from the ML Lab's scoring panel | A file is checked in the Data Studio, where it already leaves for the lab. The model export keeps its own reference for the batch it scores (V61) | +| A contract that blocks | It reports and stops. A broken contract still loads, cleans and trains: the decision stays with whoever reads the clauses | +| The same figures in Python as in the lab | The exported script carries the rows of each split and the preprocessing exactly, and translates the models: the lab's estimators are written by hand. Measured on 30 runs, three families agree to the last digit and the others differ by up to several test rows on one split. The script prints both figures and the gap instead of promising one | +| A notebook, an environment file or a project as the Python export | One readable `.py` file to put where you like. A notebook format, pinned versions and a folder layout are choices that belong to whoever runs it | +| Guessing a competition's format, or sending the file to it | The lab knows no platform: you name the identifier, the columns and what a prediction is, and the file is downloaded. Nothing leaves the browser | +| A score or a rank for a submission | The lab has neither the platform's metric nor its private rows. It counts the rows written and the identifiers a platform would refuse, nothing more | +| A measure of the model refitted on every row | Fitting on everything spends the test set. The score shown stays the one of the model fitted on 64 % of the rows; on the demos the refit gained from nothing to 5 points, and lost one time in five to one in three even where it paid | +| A submission from a re-imported model, a blend refitted, or a threshold applied | The submission is built from the live run. A re-imported model has no training rows to refit on, a blend picks its members on validation rows, and the label written is the model's raw prediction, as in the scored CSV | +| A generative chat backed by a remote API (V12) | Postponed by product decision (confirmed 09/10/2026). Today's chat runs entirely in the browser; a remote API would send out the question and the column schema, never the data, and would require a consent screen | ## Dropped after measurement diff --git a/src/content/docs/en/reference-ml.md b/src/content/docs/en/reference-ml.md index 8f26ed5..ee9c285 100644 --- a/src/content/docs/en/reference-ml.md +++ b/src/content/docs/en/reference-ml.md @@ -293,6 +293,12 @@ the columns, the answers, and the performance when the file carries labels. A drift counts only above its own noise ceiling, and only a measured drop says « retrain ». See [score a batch](/docs/scorer-un-lot). +A scored file with no label can leave as a **submission file** for a +competition: an identifier and a prediction under the names you give, every row +kept in its order. The measured model predicts by default; the same family +fitted again on every labelled row is offered, and announced as a model nobody +measured. See [score a batch](/docs/scorer-un-lot). + A share link carries the run's metrics and charts in the URL fragment, never the data. A cut or damaged link never yields a half-read page: if it does not decode, or if part of the run it carries does not have the shape LabML writes, diff --git a/src/content/docs/en/refusals.md b/src/content/docs/en/refusals.md index b830ee7..0f2f8c6 100644 --- a/src/content/docs/en/refusals.md +++ b/src/content/docs/en/refusals.md @@ -122,6 +122,8 @@ These exist, but a visitor should never see one: they are invariants checked during execution. If one appears, it is a bug. `model-not-found`, `no-references`, `no-model`, `no-run`, `bad-policy`, +`no-batch`, `id-not-found`, `name-empty`, `name-clash`, `form-unavailable`, +`rows-mismatch`, `no-join`, `no-data`, `no-manifest`, `not-ready`, `canvas-2d`, `grammar-too-long`, `grammar-atom-long`, `grammar-option-long`, `grammar-too-many-options`. diff --git a/src/content/docs/en/score-a-batch.md b/src/content/docs/en/score-a-batch.md index cb60012..78421e3 100644 --- a/src/content/docs/en/score-a-batch.md +++ b/src/content/docs/en/score-a-batch.md @@ -70,6 +70,59 @@ A model re-imported from a file carries its reference with it (format 5). An older export has none: the panel says so, and checks the batch on its labels alone. +## A submission file for a competition + +A competition hands out a file with no label and expects one back with two +columns under its own names, `PassengerId,Survived`. The scored CSV carries +every original column: the right file to join back into a system, the wrong one +for a platform. Under the scored batch, the **Submission file** block reduces it +to an identifier and a prediction. + +The lab knows neither the platform nor its rules, so you name everything. + +| You choose | What it is | +| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| The identifier column | one column of the scored file, or « number the rows from 1 ». The lab points at a column that reads as an identifier, and does not choose it for you | +| The two column names | exactly what the platform expects | +| What a prediction is | the class by name, the probability of the second class (binary), one probability per class (three classes or more), or the value (regression) | +| Which model predicts | the measured model, or the same family fitted again on every labelled row | + +The file keeps **every row of the scored file, in its order**. A row whose +identifier is empty or repeated is written and counted, never dropped: a +platform would refuse it, and you should know before it does. + +### Fitting again on every row + +A run fits on 64 % of the labelled rows and keeps the rest to choose (16 %) and +to measure (20 %). Once that is done, a prediction for unlabelled rows can use +all of them. The price is that the refitted model has no test set left: its +score is the one of its sibling fitted on fewer rows. + +Whether it pays was measured before the option was built, on the six demos, ten +splits each, with a quarter of the rows kept away from both models: + +| Demo | Labelled rows | Mean gain on the rows kept away | Better / worse | +| ------- | ------------- | ------------------------------- | -------------- | +| reviews | 180 | +5.0 accuracy points | 51 / 11 of 70 | +| mpg | 298 | RMSE down 3 % | 47 / 13 of 60 | +| energy | 180 | RMSE down 2 % | 44 / 16 of 60 | +| titanic | 668 | +0.8 accuracy point | 41 / 20 of 70 | +| fraud | 480 | none | 21 / 25 of 70 | +| iris | 112 | none | 14 / 17 of 70 | + +The gain follows the [learning curve](/docs/lire-une-courbe), not the size of +the file: iris is the smallest demo and gains nothing, because its curve is +already flat. So the refit is offered and never the default, and the block says +which model predicted the file. The blend cannot be refitted: its members are +picked on the validation rows. + +### What a submission is not + +It is not a score. The lab does not know the platform's metric nor its private +rows, and promises no rank. And a public leaderboard wears like a test set: +each submission you compare to keep the best one is a choice made with it, the +same effect the [test-set ledger](/docs/reference-ml) counts inside the lab. + ## The decision columns If the scored model carries a multiclass decision rule — frozen in the lab, then diff --git a/src/content/docs/fr/limites.md b/src/content/docs/fr/limites.md index 39674f4..9b2770c 100644 --- a/src/content/docs/fr/limites.md +++ b/src/content/docs/fr/limites.md @@ -27,38 +27,42 @@ Elles ne se valent pas, et les mélanger serait malhonnête : ## Ce qui a été écarté par choix -| Écarté | Pourquoi | -| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Un mode AutoML « on s'occupe de tout » | L'exact contraire d'un laboratoire qui montre ses décisions | -| Une dixième famille de modèles | Neuf suffisent ; une dixième n'améliore ni l'honnêteté ni la compréhension | -| Le deep learning tabulaire | Coût élevé, aucun gain à cette échelle, et ce ne serait plus écrit à la main | -| Un éditeur de cellules façon tableur | Les retouches à la main cassent la reproductibilité — la recette est le seul enregistrement | -| La déduplication floue | Faux positifs garantis sur des noms et des adresses : fusionner deux personnes réelles en silence | -| L'imputation par modèle | Opaque, et elle fabrique des valeurs plausibles — refusée deux fois, en V39 et en V40 | -| Cacher les modèles entre exécutions | La clé de cache serait la configuration entière plus les données ; un hit périmé, c'est un classement faux sans avertissement | -| Deviner la locale d'un fichier | Le navigateur n'a aucun rapport avec le fichier qu'on y dépose | -| Rendre l'indicateur d'absence obligatoire | Il ajouterait des colonnes à toute recette existante : imposer n'est pas annoncer | -| Traduire le changelog et prérendre la documentation française | Le changelog est un journal d'ingénierie, pas un contenu éditorial ; sans routes par langue, des alternates hreflang mentiraient | -| Capter les erreurs asynchrones et envoyer des rapports de plantage | Les boundaries React ne voient pas les callbacks ni les gestionnaires d'événements — prétendre le contraire serait faux ; et rien ne quitte le navigateur, télémétrie comprise | -| Un Dependabot quotidien et un `npm audit fix --force` | Une cadence quotidienne noie le signal ; `--force` change des majeures sans lire la rupture. Les advisories restantes touchent les outils de build, et on le dit plutôt que de le masquer | -| Choisir les seuils multiclasses automatiquement | Le laboratoire montre ses décisions au lieu de les prendre à votre place ; et l'abstention n'est ni une calibration d'incertitude ni une garantie conforme (cette garantie existe dans son propre panneau, sous forme d'ensembles de prédiction) | -| Un identifiant de run dans chaque message du worker | Le worker répond dans l'ordre où on lui écrit : l'état de l'entraînement suffit à reconnaître une réponse périmée. L'identifiant deviendra nécessaire le jour où des requêtes pourront se chevaucher | -| Un avertissement « classe rare » dans l'interface | Une classe d'une seule ligne reste à l'entraînement, et une cible où chaque classe n'a qu'une ligne est refusée par son nom ; un bandeau signalerait un problème que la découpe règle déjà | -| Découper le store du ML Lab en tranches indépendantes | Toutes ses actions partagent un seul worker : découper aurait déplacé du code sans rien séparer | -| Corriger un score selon les runs auxquels il a été comparé | L'optimisme dépend de l'écart réel entre les runs (0,2 à 0,8 fois le bruit entre deux découpes à 5 runs sur les démos), et le labo ne peut pas le connaître. Le registre du jeu de test compte et avertit ; le score reste celui qui a été mesuré | -| Prendre l'empreinte du contenu d'un fichier pour le registre | Le registre reconnaît un fichier à son nom et à sa forme. Deux fichiers qui partagent les deux partageraient un compte ; hacher chaque fichier à chaque chargement ne valait pas ce cas rare | -| Compter un réglage ou une analyse dans le registre | Ils lisent le jeu de test sans choisir entre des runs ; le registre compte ce qu'une comparaison entre runs peut gonfler | -| Dire « réentraîner » à partir d'une dérive seule | Une dérive dit que les entrées ou les réponses ont bougé, pas que le modèle se trompe davantage : sur la démo mpg, le lot qui avait le plus bougé avait une erreur plus faible. Sans étiquettes, la vérification en production s'arrête à « surveiller » | -| Pondérer une dérive par l'importance de sa colonne | L'export ne porte pas les importances, et une colonne que le modèle utilise peu peut être le premier signe que la population a changé. Toutes les colonnes du modèle sont lues de la même façon | -| Un historique des lots vérifiés | Chaque vérification lit un lot contre la référence. Suivre une tendance d'un lot à l'autre demande de garder les vérifications passées, ce que rien n'a encore demandé | -| Changer le calcul du PSI | Les tranches, le lissage des tranches vides et la formule sont ceux de tout le monde : un PSI lu ici se compare à un PSI lu ailleurs. Ce que V62 a changé, c'est ce à quoi on compare le chiffre : son propre plafond de bruit | -| Un verdict sur un déplacement du taux de manquants | Il est affiché à côté de chaque colonne et laissé au lecteur : une colonne qui passe de 3 % à 0 % de manquants est un fait sur l'export, pas une distribution à tester | -| Des règles écrites à la main dans un contrat de données | Un contrat fige ce que le labo sait constater : sept règles, les lectures, les parts. Des bornes sur mesure et des expressions sont un langage à part entière, avec son analyseur, ses erreurs et ses tests, et rien ne l'a encore demandé | -| Vérifier un contrat depuis le panneau de scoring du ML Lab | Un fichier se vérifie dans le Data Studio, d'où il part déjà vers le lab. L'export de modèle garde sa propre référence pour le lot qu'il score (V61) | -| Un contrat qui bloque | Il signale et s'arrête. Un contrat rompu se charge, se nettoie et s'entraîne quand même : la décision reste à qui lit les clauses | -| Les mêmes chiffres en Python que dans le labo | Le script exporté reprend à l'identique les lignes de chaque découpe et le prétraitement, et traduit les modèles : les estimateurs du labo sont écrits à la main. Mesuré sur 30 exécutions, trois familles concordent au dernier chiffre et les autres s'écartent jusqu'à plusieurs lignes de test sur une découpe. Le script affiche les deux chiffres et l'écart au lieu d'en promettre un | -| Un notebook, un fichier d'environnement ou un projet comme export Python | Un seul fichier `.py` lisible, à poser où l'on veut. Un format de notebook, des versions épinglées et une arborescence sont des choix qui reviennent à qui l'exécute | -| Un chat génératif branché sur une API distante (V12) | Reporté par décision produit (confirmé le 09/10/2026). Le chat actuel tourne entièrement dans le navigateur ; une API distante ferait sortir la question et le schéma des colonnes, jamais les données, et exigerait un écran de consentement | +| Écarté | Pourquoi | +| -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Un mode AutoML « on s'occupe de tout » | L'exact contraire d'un laboratoire qui montre ses décisions | +| Une dixième famille de modèles | Neuf suffisent ; une dixième n'améliore ni l'honnêteté ni la compréhension | +| Le deep learning tabulaire | Coût élevé, aucun gain à cette échelle, et ce ne serait plus écrit à la main | +| Un éditeur de cellules façon tableur | Les retouches à la main cassent la reproductibilité — la recette est le seul enregistrement | +| La déduplication floue | Faux positifs garantis sur des noms et des adresses : fusionner deux personnes réelles en silence | +| L'imputation par modèle | Opaque, et elle fabrique des valeurs plausibles — refusée deux fois, en V39 et en V40 | +| Cacher les modèles entre exécutions | La clé de cache serait la configuration entière plus les données ; un hit périmé, c'est un classement faux sans avertissement | +| Deviner la locale d'un fichier | Le navigateur n'a aucun rapport avec le fichier qu'on y dépose | +| Rendre l'indicateur d'absence obligatoire | Il ajouterait des colonnes à toute recette existante : imposer n'est pas annoncer | +| Traduire le changelog et prérendre la documentation française | Le changelog est un journal d'ingénierie, pas un contenu éditorial ; sans routes par langue, des alternates hreflang mentiraient | +| Capter les erreurs asynchrones et envoyer des rapports de plantage | Les boundaries React ne voient pas les callbacks ni les gestionnaires d'événements — prétendre le contraire serait faux ; et rien ne quitte le navigateur, télémétrie comprise | +| Un Dependabot quotidien et un `npm audit fix --force` | Une cadence quotidienne noie le signal ; `--force` change des majeures sans lire la rupture. Les advisories restantes touchent les outils de build, et on le dit plutôt que de le masquer | +| Choisir les seuils multiclasses automatiquement | Le laboratoire montre ses décisions au lieu de les prendre à votre place ; et l'abstention n'est ni une calibration d'incertitude ni une garantie conforme (cette garantie existe dans son propre panneau, sous forme d'ensembles de prédiction) | +| Un identifiant de run dans chaque message du worker | Le worker répond dans l'ordre où on lui écrit : l'état de l'entraînement suffit à reconnaître une réponse périmée. L'identifiant deviendra nécessaire le jour où des requêtes pourront se chevaucher | +| Un avertissement « classe rare » dans l'interface | Une classe d'une seule ligne reste à l'entraînement, et une cible où chaque classe n'a qu'une ligne est refusée par son nom ; un bandeau signalerait un problème que la découpe règle déjà | +| Découper le store du ML Lab en tranches indépendantes | Toutes ses actions partagent un seul worker : découper aurait déplacé du code sans rien séparer | +| Corriger un score selon les runs auxquels il a été comparé | L'optimisme dépend de l'écart réel entre les runs (0,2 à 0,8 fois le bruit entre deux découpes à 5 runs sur les démos), et le labo ne peut pas le connaître. Le registre du jeu de test compte et avertit ; le score reste celui qui a été mesuré | +| Prendre l'empreinte du contenu d'un fichier pour le registre | Le registre reconnaît un fichier à son nom et à sa forme. Deux fichiers qui partagent les deux partageraient un compte ; hacher chaque fichier à chaque chargement ne valait pas ce cas rare | +| Compter un réglage ou une analyse dans le registre | Ils lisent le jeu de test sans choisir entre des runs ; le registre compte ce qu'une comparaison entre runs peut gonfler | +| Dire « réentraîner » à partir d'une dérive seule | Une dérive dit que les entrées ou les réponses ont bougé, pas que le modèle se trompe davantage : sur la démo mpg, le lot qui avait le plus bougé avait une erreur plus faible. Sans étiquettes, la vérification en production s'arrête à « surveiller » | +| Pondérer une dérive par l'importance de sa colonne | L'export ne porte pas les importances, et une colonne que le modèle utilise peu peut être le premier signe que la population a changé. Toutes les colonnes du modèle sont lues de la même façon | +| Un historique des lots vérifiés | Chaque vérification lit un lot contre la référence. Suivre une tendance d'un lot à l'autre demande de garder les vérifications passées, ce que rien n'a encore demandé | +| Changer le calcul du PSI | Les tranches, le lissage des tranches vides et la formule sont ceux de tout le monde : un PSI lu ici se compare à un PSI lu ailleurs. Ce que V62 a changé, c'est ce à quoi on compare le chiffre : son propre plafond de bruit | +| Un verdict sur un déplacement du taux de manquants | Il est affiché à côté de chaque colonne et laissé au lecteur : une colonne qui passe de 3 % à 0 % de manquants est un fait sur l'export, pas une distribution à tester | +| Des règles écrites à la main dans un contrat de données | Un contrat fige ce que le labo sait constater : sept règles, les lectures, les parts. Des bornes sur mesure et des expressions sont un langage à part entière, avec son analyseur, ses erreurs et ses tests, et rien ne l'a encore demandé | +| Vérifier un contrat depuis le panneau de scoring du ML Lab | Un fichier se vérifie dans le Data Studio, d'où il part déjà vers le lab. L'export de modèle garde sa propre référence pour le lot qu'il score (V61) | +| Un contrat qui bloque | Il signale et s'arrête. Un contrat rompu se charge, se nettoie et s'entraîne quand même : la décision reste à qui lit les clauses | +| Les mêmes chiffres en Python que dans le labo | Le script exporté reprend à l'identique les lignes de chaque découpe et le prétraitement, et traduit les modèles : les estimateurs du labo sont écrits à la main. Mesuré sur 30 exécutions, trois familles concordent au dernier chiffre et les autres s'écartent jusqu'à plusieurs lignes de test sur une découpe. Le script affiche les deux chiffres et l'écart au lieu d'en promettre un | +| Un notebook, un fichier d'environnement ou un projet comme export Python | Un seul fichier `.py` lisible, à poser où l'on veut. Un format de notebook, des versions épinglées et une arborescence sont des choix qui reviennent à qui l'exécute | +| Deviner le format d'une compétition, ou lui envoyer le fichier | Le labo ne connaît aucune plateforme : vous nommez l'identifiant, les colonnes et ce qu'est une prédiction, et le fichier se télécharge. Rien ne quitte le navigateur | +| Un score ou un rang pour une soumission | Le labo n'a ni la métrique de la plateforme ni ses lignes privées. Il compte les lignes écrites et les identifiants qu'une plateforme refuserait, rien de plus | +| Une mesure du modèle réentraîné sur toutes les lignes | S'entraîner sur tout consomme le jeu de test. Le score affiché reste celui du modèle entraîné sur 64 % des lignes ; sur les démos le réentraînement a gagné de rien à 5 points, et perdu une fois sur cinq à une fois sur trois même là où il rapportait | +| Une soumission depuis un modèle réimporté, un mélange réentraîné, ou un seuil appliqué | La soumission se construit depuis le run en cours. Un modèle réimporté n'a pas de lignes d'entraînement, un mélange choisit ses membres sur les lignes de validation, et la classe écrite est la prédiction brute du modèle, comme dans le CSV scoré | +| Un chat génératif branché sur une API distante (V12) | Reporté par décision produit (confirmé le 09/10/2026). Le chat actuel tourne entièrement dans le navigateur ; une API distante ferait sortir la question et le schéma des colonnes, jamais les données, et exigerait un écran de consentement | ## Ce qui a été abandonné après mesure diff --git a/src/content/docs/fr/reference-ml.md b/src/content/docs/fr/reference-ml.md index dea788b..28b5646 100644 --- a/src/content/docs/fr/reference-ml.md +++ b/src/content/docs/fr/reference-ml.md @@ -322,6 +322,13 @@ réponses, et la performance quand le fichier porte des étiquettes. Une dérive ne compte qu'au-dessus de son propre plafond de bruit, et seule une baisse mesurée dit « réentraîner ». Voir [scorer un lot](/docs/scorer-un-lot). +Un fichier scoré sans étiquette peut repartir en **fichier de soumission** pour +une compétition : un identifiant et une prédiction sous les noms que vous +donnez, toutes les lignes gardées dans leur ordre. Le modèle mesuré prédit par +défaut ; la même famille réentraînée sur toutes les lignes étiquetées est +proposée, et annoncée comme un modèle que personne n'a mesuré. Voir +[scorer un lot](/docs/scorer-un-lot). + Un lien de partage transporte les métriques et les graphiques du run dans le fragment d'URL, jamais les données. Un lien coupé ou abîmé ne donne jamais une page à moitié lue : s'il ne se décode pas, ou si une partie du run qu'il porte diff --git a/src/content/docs/fr/refus.md b/src/content/docs/fr/refus.md index 32b2b22..5ec59c2 100644 --- a/src/content/docs/fr/refus.md +++ b/src/content/docs/fr/refus.md @@ -123,6 +123,8 @@ Ceux-ci existent, mais un visiteur ne devrait jamais les voir : ce sont des invariantes vérifiées pendant l'exécution. Si l'un apparaît, c'est un bug. `model-not-found`, `no-references`, `no-model`, `no-run`, `bad-policy`, +`no-batch`, `id-not-found`, `name-empty`, `name-clash`, `form-unavailable`, +`rows-mismatch`, `no-join`, `no-data`, `no-manifest`, `not-ready`, `canvas-2d`, `grammar-too-long`, `grammar-atom-long`, `grammar-option-long`, `grammar-too-many-options`. diff --git a/src/content/docs/fr/scorer-un-lot.md b/src/content/docs/fr/scorer-un-lot.md index 335dd0a..bd0159e 100644 --- a/src/content/docs/fr/scorer-un-lot.md +++ b/src/content/docs/fr/scorer-un-lot.md @@ -74,6 +74,62 @@ Un modèle réimporté depuis un fichier porte sa référence avec lui (format 5 Un export plus ancien n'en a pas : le panneau le dit, et vérifie le lot sur ses seules étiquettes. +## Un fichier de soumission pour une compétition + +Une compétition fournit un fichier sans étiquette et en attend un en retour, +avec deux colonnes sous ses propres noms, `PassengerId,Survived`. Le CSV scoré +reprend toutes les colonnes d'origine : le bon fichier pour une jointure dans +un système, le mauvais pour une plateforme. Sous le lot scoré, le bloc +**Fichier de soumission** le réduit à un identifiant et une prédiction. + +Le labo ne connaît ni la plateforme ni ses règles : c'est vous qui nommez tout. + +| Vous choisissez | Ce que c'est | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| La colonne d'identifiant | une colonne du fichier scoré, ou « numéroter les lignes à partir de 1 ». Le labo signale une colonne qui se lit comme un identifiant, sans la choisir à votre place | +| Les deux noms de colonnes | exactement ce que la plateforme attend | +| Ce qu'est une prédiction | la classe par son nom, la probabilité de la seconde classe (binaire), une probabilité par classe (trois classes ou plus), ou la valeur (régression) | +| Quel modèle prédit | le modèle mesuré, ou la même famille réentraînée sur toutes les lignes étiquetées | + +Le fichier garde **toutes les lignes du fichier scoré, dans leur ordre**. Une +ligne dont l'identifiant est vide ou répété est écrite et comptée, jamais +supprimée : une plateforme la refuserait, et mieux vaut le savoir avant elle. + +### Réentraîner sur toutes les lignes + +Un run s'entraîne sur 64 % des lignes étiquetées et garde le reste pour choisir +(16 %) et pour mesurer (20 %). Une fois ce travail fait, une prédiction sur des +lignes sans étiquette peut se servir de toutes. Le prix : le modèle réentraîné +n'a plus de jeu de test. Son score est celui de son frère entraîné sur moins de +lignes. + +Ce que cela rapporte a été mesuré avant de construire l'option, sur les six +démos, dix découpes chacune, avec un quart des lignes tenu à l'écart des deux +modèles : + +| Démo | Lignes étiquetées | Gain moyen sur les lignes tenues à l'écart | Mieux / pire | +| ------- | ----------------- | ------------------------------------------ | ------------- | +| reviews | 180 | +5,0 points d'exactitude | 51 / 11 de 70 | +| mpg | 298 | RMSE en baisse de 3 % | 47 / 13 de 60 | +| energy | 180 | RMSE en baisse de 2 % | 44 / 16 de 60 | +| titanic | 668 | +0,8 point d'exactitude | 41 / 20 de 70 | +| fraud | 480 | aucun | 21 / 25 de 70 | +| iris | 112 | aucun | 14 / 17 de 70 | + +Le gain suit la [courbe d'apprentissage](/docs/lire-une-courbe), pas la taille +du fichier : iris est la plus petite démo et ne gagne rien, parce que sa courbe +est déjà plate. Le réentraînement est donc proposé, jamais par défaut, et le +bloc dit quel modèle a prédit le fichier. Le mélange ne se réentraîne pas : ses +membres sont choisis sur les lignes de validation. + +### Ce qu'une soumission n'est pas + +Ce n'est pas un score. Le labo ne connaît ni la métrique de la plateforme ni ses +lignes privées, et ne promet aucun rang. Et un classement public s'use comme un +jeu de test : chaque soumission comparée pour garder la meilleure est un choix +fait avec lui, le même effet que le [registre du jeu de test](/docs/reference-ml) +compte à l'intérieur du labo. + ## Les colonnes de décision Si le modèle scoré porte une règle de décision multiclasse — figée dans le diff --git a/src/features/docs/refusals.ts b/src/features/docs/refusals.ts index 178cade..f329598 100644 --- a/src/features/docs/refusals.ts +++ b/src/features/docs/refusals.ts @@ -51,6 +51,13 @@ export const REFUSALS: Refusal[] = [ { code: 'no-model', audience: 'internal', area: 'ml' }, { code: 'no-run', audience: 'internal', area: 'ml' }, { code: 'bad-policy', audience: 'internal', area: 'ml' }, + // V65: the submission panel only sends what these guard against. + { code: 'no-batch', audience: 'internal', area: 'ml' }, + { code: 'id-not-found', audience: 'internal', area: 'ml' }, + { code: 'name-empty', audience: 'internal', area: 'ml' }, + { code: 'name-clash', audience: 'internal', area: 'ml' }, + { code: 'form-unavailable', audience: 'internal', area: 'ml' }, + { code: 'rows-mismatch', audience: 'internal', area: 'ml' }, // --- Importing a model: five named reasons instead of one « invalid file » --- { code: 'invalid-json', audience: 'visitor', area: 'import' }, diff --git a/src/features/ml/components/BatchScorePanel.tsx b/src/features/ml/components/BatchScorePanel.tsx index 898fa1c..2805120 100644 --- a/src/features/ml/components/BatchScorePanel.tsx +++ b/src/features/ml/components/BatchScorePanel.tsx @@ -4,6 +4,7 @@ import { useTranslation } from 'react-i18next'; import { Badge } from '@/components/ui/badge'; import { Button } from '@/components/ui/button'; import { Eyebrow } from '@/components/ui/eyebrow'; +import { SubmissionPanel } from '@/features/ml/components/SubmissionPanel'; import { ProductionCheckView } from '@/features/ml/components/ProductionCheckView'; import { useLabStore } from '@/features/ml/lab-store'; import { METRIC_ROWS, metricDelta } from '@/features/ml/train/score-view'; @@ -195,6 +196,8 @@ export function BatchScorePanel() { {result.check && } + +

{t('ml.lab.batch.note')}

)} diff --git a/src/features/ml/components/SubmissionPanel.test.tsx b/src/features/ml/components/SubmissionPanel.test.tsx new file mode 100644 index 0000000..3e81d5c --- /dev/null +++ b/src/features/ml/components/SubmissionPanel.test.tsx @@ -0,0 +1,217 @@ +import { fireEvent, render, screen } from '@testing-library/react'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; +import { SubmissionPanel } from '@/features/ml/components/SubmissionPanel'; +import { useLabStore } from '@/features/ml/lab-store'; +import type { ColumnProfile } from '@/features/ml/data/types'; +import type { BatchScore } from '@/features/ml/train/score'; +import type { Submission } from '@/features/ml/train/submission'; +import i18n from '@/lib/i18n'; + +// V65: the panel lets the user name everything, proposes without choosing, +// and says which model predicted the file. + +const batch = (over: Partial = {}) => + ({ + fileName: 'test.csv', + model: 'forest', + rowCount: 418, + header: ['PassengerId', 'Pclass', 'Sex'], + hasProbabilities: true, + classCount: 2, + isClassification: true, + positiveClass: '1', + ...over, + }) as BatchScore; + +const submission = (over: Partial = {}): Submission => ({ + csv: 'PassengerId,Survived\n892,0\n', + header: ['PassengerId', 'Survived'], + rows: 418, + missingIds: 0, + duplicateIds: 0, + counts: [ + { label: '0', count: 266 }, + { label: '1', count: 152 }, + ], + preview: ['PassengerId,Survived', '892,0'], + fileName: 'test-submission.csv', + model: 'forest', + form: 'label', + refit: null, + ...over, +}); + +const build = vi.fn(); + +describe('SubmissionPanel', () => { + beforeEach(async () => { + await i18n.changeLanguage('en'); + build.mockClear(); + useLabStore.setState({ + batchStatus: 'done', + batchResult: batch(), + target: 'Survived', + profiles: [{ name: 'PassengerId', type: 'id' } as ColumnProfile], + submissionStatus: 'idle', + submission: null, + submissionError: null, + buildSubmission: build, + }); + }); + + it('says nothing without a scored file', () => { + useLabStore.setState({ batchResult: null }); + const { container } = render(); + expect(container).toBeEmptyDOMElement(); + }); + + it('proposes the identifier without choosing it, and waits for the choice', () => { + render(); + expect(screen.getByTestId('submission-suggestion')).toHaveTextContent( + '"PassengerId" reads as an identifier', + ); + expect(screen.getByTestId('submission-id-column')).toHaveValue(''); + expect(screen.getByTestId('submission-build')).toBeDisabled(); + expect(screen.getByTestId('submission-prediction-name')).toHaveValue('Survived'); + }); + + it('sends what the user named, with the measured model by default', () => { + render(); + fireEvent.change(screen.getByTestId('submission-id-column'), { + target: { value: 'PassengerId' }, + }); + expect(screen.getByTestId('submission-id-name')).toHaveValue('PassengerId'); + expect(screen.getByTestId('submission-model-measured')).toBeChecked(); + fireEvent.click(screen.getByTestId('submission-build')); + expect(build).toHaveBeenCalledWith({ + idColumn: 'PassengerId', + idName: 'PassengerId', + predictionName: 'Survived', + form: 'label', + refit: false, + }); + }); + + it('numbers the rows on request, under a name the user types, and can ask for the refit', () => { + render(); + const select = screen.getByTestId('submission-id-column') as HTMLSelectElement; + const rowNumber = [...select.options].find((option) => option.text.startsWith('no column'))!; + fireEvent.change(select, { target: { value: rowNumber.value } }); + expect(screen.getByTestId('submission-build')).toBeDisabled(); + fireEvent.change(screen.getByTestId('submission-id-name'), { target: { value: 'id' } }); + fireEvent.click(screen.getByTestId('submission-form-probability')); + fireEvent.click(screen.getByTestId('submission-model-refit')); + fireEvent.click(screen.getByTestId('submission-build')); + expect(build).toHaveBeenCalledWith({ + idColumn: null, + idName: 'id', + predictionName: 'Survived', + form: 'probability', + refit: true, + }); + }); + + it('refuses two columns under one name', () => { + render(); + fireEvent.change(screen.getByTestId('submission-id-column'), { + target: { value: 'PassengerId' }, + }); + fireEvent.change(screen.getByTestId('submission-prediction-name'), { + target: { value: 'PassengerId' }, + }); + expect(screen.getByTestId('submission-clash')).toBeInTheDocument(); + expect(screen.getByTestId('submission-build')).toBeDisabled(); + }); + + it('offers only the forms of the task', () => { + useLabStore.setState({ batchResult: batch({ classCount: 3, positiveClass: undefined }) }); + const { unmount } = render(); + expect(screen.getByTestId('submission-form-probabilities')).toBeInTheDocument(); + expect(screen.queryByTestId('submission-form-probability')).toBeNull(); + fireEvent.click(screen.getByTestId('submission-form-probabilities')); + expect(screen.queryByTestId('submission-prediction-name')).toBeNull(); + unmount(); + + useLabStore.setState({ + batchResult: batch({ isClassification: false, classCount: 0, hasProbabilities: false }), + }); + render(); + expect(screen.getByTestId('submission-form-value')).toBeChecked(); + expect(screen.queryByTestId('submission-form-label')).toBeNull(); + }); + + it('says the measured model predicted, and that a submission is not a score', () => { + useLabStore.setState({ submissionStatus: 'done', submission: submission() }); + render(); + expect(screen.getByTestId('submission-result')).toHaveAttribute('data-model', 'measured'); + expect(screen.getByTestId('submission-summary')).toHaveTextContent( + '418 rows, the same as the scored file and in its order. Columns: PassengerId, Survived.', + ); + expect(screen.getByTestId('submission-model')).toHaveTextContent( + 'the model the test set measured', + ); + expect(screen.getByTestId('submission-counts')).toHaveTextContent('0: 266 · 1: 152'); + expect(screen.queryByTestId('submission-ids')).toBeNull(); + expect(screen.getByTestId('submission-result')).toHaveTextContent( + 'A submission is not a score', + ); + }); + + it('announces a refitted model as one nobody measured, and its cap', () => { + useLabStore.setState({ + submissionStatus: 'done', + submission: submission({ refit: { rows: 891, fittedRows: 891 } }), + }); + const { unmount } = render(); + expect(screen.getByTestId('submission-result')).toHaveAttribute('data-model', 'refit'); + expect(screen.getByTestId('submission-model')).toHaveTextContent( + 'fitted again on the 891 labelled rows. Nobody measured this model', + ); + unmount(); + useLabStore.setState({ + submission: submission({ refit: { rows: 9000, fittedRows: 5000 } }), + }); + render(); + expect(screen.getByTestId('submission-model')).toHaveTextContent( + 'on 5,000 of the 9,000 labelled rows, its announced cap', + ); + }); + + it('does not offer to refit a blend', () => { + useLabStore.setState({ batchResult: batch({ model: 'ensemble' }) }); + render(); + expect(screen.getByTestId('submission-model-refit')).toBeDisabled(); + expect(screen.getByTestId('submission-refit-note')).toHaveTextContent( + 'The blend cannot be fitted again', + ); + }); + + it('counts the identifiers a platform would refuse', () => { + useLabStore.setState({ + submissionStatus: 'done', + submission: submission({ missingIds: 2, duplicateIds: 1 }), + }); + render(); + expect(screen.getByTestId('submission-ids')).toHaveTextContent( + '2 rows have no identifier: they are written with an empty one, and a platform will likely refuse them. 1 row repeats an identifier already used.', + ); + }); + + it('speaks French', async () => { + await i18n.changeLanguage('fr'); + useLabStore.setState({ + submissionStatus: 'done', + submission: submission({ refit: { rows: 891, fittedRows: 891 } }), + }); + render(); + expect(screen.getByTestId('submission-suggestion')).toHaveTextContent( + '« PassengerId » se lit comme un identifiant', + ); + expect(screen.getByTestId('submission-model')).toHaveTextContent( + 'Personne n’a mesuré ce modèle'.replace('’', "'"), + ); + expect(screen.getByTestId('submission-result')).toHaveTextContent( + "Une soumission n'est pas un score", + ); + }); +}); diff --git a/src/features/ml/components/SubmissionPanel.tsx b/src/features/ml/components/SubmissionPanel.tsx new file mode 100644 index 0000000..ae5b739 --- /dev/null +++ b/src/features/ml/components/SubmissionPanel.tsx @@ -0,0 +1,296 @@ +import { Loader2, Send } from 'lucide-react'; +import { useState } from 'react'; +import { useTranslation } from 'react-i18next'; +import { Button } from '@/components/ui/button'; +import { Eyebrow } from '@/components/ui/eyebrow'; +import { useLabStore } from '@/features/ml/lab-store'; +import { formsFor, type SubmissionForm } from '@/features/ml/train/submission'; +import { downloadFile } from '@/lib/download'; + +/** The select value that numbers the rows instead of reading a column. */ +const ROW_NUMBER = '\u0000row-number'; + +const fieldClass = + 'w-full min-w-0 rounded-lg border border-line bg-surface px-2 py-1 font-mono text-xs focus:border-accent focus:outline-none'; + +/** + * V65: the last scored file as a file a competition platform accepts. The + * user names everything; the lab only keeps the rows, in their order, and + * says which model predicted them. + */ +export function SubmissionPanel() { + const { t, i18n } = useTranslation(); + const batch = useLabStore((s) => s.batchResult); + const target = useLabStore((s) => s.target); + const profiles = useLabStore((s) => s.profiles); + const status = useLabStore((s) => s.submissionStatus); + const submission = useLabStore((s) => s.submission); + const error = useLabStore((s) => s.submissionError); + const build = useLabStore((s) => s.buildSubmission); + + const [idColumn, setIdColumn] = useState(''); + const [idName, setIdName] = useState(''); + const [predictionName, setPredictionName] = useState(target ?? ''); + const [form, setForm] = useState(null); + const [refit, setRefit] = useState(false); + + if (!batch?.header || batch.isClassification === undefined) return null; + const lang = i18n.resolvedLanguage ?? 'en'; + const forms = formsFor( + batch.isClassification, + batch.classCount ?? 0, + batch.hasProbabilities ?? false, + ); + const chosenForm = form !== null && forms.includes(form) ? form : forms[0]; + const needsName = chosenForm !== 'probabilities'; + + // The lab proposes, and does not choose: a single column of the scored file + // that the training file read as an identifier. + const idLike = batch.header.filter((name) => + profiles.some((profile) => profile.name === name && profile.type === 'id'), + ); + const suggestion = idLike.length === 1 ? idLike[0] : null; + + const trimmedId = idName.trim(); + const trimmedPrediction = predictionName.trim(); + const clash = needsName && trimmedId !== '' && trimmedId === trimmedPrediction; + const ready = + idColumn !== '' && trimmedId !== '' && (!needsName || trimmedPrediction !== '') && !clash; + const num = (value: number, digits: number) => + value.toLocaleString(lang, { maximumFractionDigits: digits }); + const positive = batch.positiveClass; + // A blend is built from models picked on the validation rows: fitted on + // everything, it would have nothing left to pick its members with. + const canRefit = batch.model !== 'ensemble'; + + return ( +
+
+
+
+

{t('ml.lab.submission.hint')}

+
+ +
+ + + {needsName && ( + + )} +
+ {suggestion !== null && idColumn === '' && ( +

+ {t('ml.lab.submission.suggestion', { column: suggestion })} +

+ )} + {clash && ( +

+ {t('ml.lab.submission.clash')} +

+ )} + +
+ {t('ml.lab.submission.formTitle')} + {forms.map((option) => ( + + ))} +
+ +
+ {t('ml.lab.submission.modelTitle')} + + +

+ {t(canRefit ? 'ml.lab.submission.refitNote' : 'ml.lab.submission.refitUnavailable')} +

+
+ +
+ + {status === 'building' && ( + + + )} +
+ + {status === 'error' && ( +

+ {t('ml.lab.submission.error', { detail: error ?? '' })} +

+ )} + + {status === 'done' && submission && ( +
+

+ {t('ml.lab.submission.summary', { + count: submission.rows, + rows: submission.rows.toLocaleString(lang), + columns: submission.header.join(', '), + })} +

+

+ {submission.refit + ? t( + submission.refit.fittedRows < submission.refit.rows + ? 'ml.lab.submission.byRefitCapped' + : 'ml.lab.submission.byRefit', + { + model: t(`ml.lab.models.${submission.model}`), + rows: submission.refit.rows.toLocaleString(lang), + fitted: submission.refit.fittedRows.toLocaleString(lang), + }, + ) + : t('ml.lab.submission.byMeasured', { + model: t(`ml.lab.models.${submission.model}`), + })} +

+ {(submission.missingIds > 0 || submission.duplicateIds > 0) && ( +

+ {[ + submission.missingIds > 0 && + t('ml.lab.submission.missingIds', { count: submission.missingIds }), + submission.duplicateIds > 0 && + t('ml.lab.submission.duplicateIds', { count: submission.duplicateIds }), + ] + .filter(Boolean) + .join(' ')} +

+ )} + {submission.counts && ( +

+ {submission.counts + .slice(0, 6) + .map((item) => `${item.label}: ${item.count.toLocaleString(lang)}`) + .join(' · ')} + {submission.counts.length > 6 ? ' · …' : ''} +

+ )} + {submission.range && ( +

+ {t('ml.lab.submission.range', { + min: num(submission.range.min, 4), + max: num(submission.range.max, 4), + mean: num(submission.range.mean, 4), + })} +

+ )} +
+            {submission.preview.join('\n')}
+            {submission.rows + 1 > submission.preview.length ? '\n…' : ''}
+          
+
+ +
+

{t('ml.lab.submission.note')}

+
+ )} +
+ ); +} diff --git a/src/features/ml/data/parse.worker.ts b/src/features/ml/data/parse.worker.ts index 087e51a..6f644fe 100644 --- a/src/features/ml/data/parse.worker.ts +++ b/src/features/ml/data/parse.worker.ts @@ -15,7 +15,9 @@ import { buildPredictionsCsv, referenceOf, serializeModel } from '@/features/ml/ import { explainPrediction } from '@/features/ml/train/shapley'; import { buildPythonScript } from '@/features/ml/train/python-export'; import { deserializeModel, type ImportedModel } from '@/features/ml/train/deserialize'; -import { scoreBatch, scoreRows } from '@/features/ml/train/score'; +import { predictRows, scoreBatch, scoreRows } from '@/features/ml/train/score'; +import { buildSubmission } from '@/features/ml/train/submission'; +import { refitOnAllRows } from '@/features/ml/train/refit'; import { analyzeSegments } from '@/features/ml/train/segments'; import { analyzeThresholds } from '@/features/ml/train/threshold-analysis'; import { analyzeConformal } from '@/features/ml/train/conformal-analysis'; @@ -26,7 +28,7 @@ import { import { analyzeUncertainty, type ModelLosses } from '@/features/ml/train/uncertainty'; import { detectTaskType, runTraining, type TrainArtifacts } from '@/features/ml/train/trainer'; import type { Cell, ColumnProfile, ParseResultPayload } from '@/features/ml/data/types'; -import type { ModelKey } from '@/features/ml/train/types'; +import type { ModelKey, TrainConfig } from '@/features/ml/train/types'; import { readTable } from '@/features/ml/data/ingest'; import type { ReadFormat } from '@/features/ml/data/locale'; import type { WorkerRequest, WorkerResponse } from '@/features/ml/worker-protocol'; @@ -52,6 +54,10 @@ let lastFeatureColumns: string[] = []; let datasetName = ''; /** v22: a model rebuilt from an export — fully independent of any run. */ let imported: ImportedModel | null = null; +/** V65: the last file the live run scored, kept for its submission. Never leaves the worker. */ +let lastBatch: { name: string; header: string[]; cols: Cell[][]; model: ModelKey } | null = null; +/** V65: the configuration the artifacts were trained with: a refit repeats it. */ +let lastConfig: TrainConfig | null = null; /** V38: how the current file was actually read — announced, never assumed. */ let readFormat: ReadFormat | null = null; @@ -87,6 +93,8 @@ function resetState() { columns = []; rowCount = 0; artifacts = null; + lastBatch = null; + lastConfig = null; readFormat = null; } @@ -130,25 +138,88 @@ async function handleScoreBatch( try { if (!artifacts || !lastTarget) throw new Error('no-run'); const batch = await parseBatch(source); + lastBatch = null; + const payload = scoreBatch( + artifacts, + model, + lastTarget, + name, + batch.header, + batch.cols, + decisionPolicy, + // V61: a live run builds its reference on the spot, from its own rows. + { + reference: referenceOf(artifacts, model, columnsAsMap()), + testRows: artifacts.testY.length, + }, + ); + // Kept only once the file has been scored: a refused file leaves no submission. + lastBatch = { name, header: batch.header, cols: batch.cols, model }; + post({ kind: 'batch-scored', payload }); + } catch (error) { + post({ kind: 'batch-error', message: error instanceof Error ? error.message : String(error) }); + } +} + +/** + * V65: the last scored file as an identifier and a prediction. The measured + * model predicts unless `refit` asks for its sibling fitted on every labelled + * row, which is used for this file and kept nowhere. + */ +function handleSubmission(request: Extract) { + try { + if (!artifacts || !lastBatch) throw new Error('no-batch'); + const batch = lastBatch; + const measured = artifacts.models.get(batch.model); + if (!measured) throw new Error('model-not-found'); + const idAt = request.idColumn === null ? -1 : batch.header.indexOf(request.idColumn); + if (request.idColumn !== null && idAt < 0) throw new Error('id-not-found'); + + let refit: ReturnType = null; + if (request.refit) { + if (!lastConfig) throw new Error('no-run'); + const profiles: ColumnProfile[] = header.map((column, i) => + profileColumn(column, columns[i]), + ); + refit = refitOnAllRows(columnsAsMap(), profiles, lastConfig, batch.model); + if (!refit) throw new Error('model-not-found'); + } + const pipeline = refit?.pipeline ?? artifacts.pipeline; + const { predictions, probabilities } = predictRows( + { + model: refit?.model ?? measured, + specs: pipeline.specs, + transformRow: pipeline.transformRow, + isClassification: artifacts.isClassification, + }, + batch.header, + batch.cols, + ); + const file = buildSubmission({ + ids: idAt < 0 ? null : batch.cols[idAt], + idName: request.idName, + predictionName: request.predictionName, + form: request.form, + predictions, + probabilities, + classes: artifacts.classes, + isClassification: artifacts.isClassification, + }); post({ - kind: 'batch-scored', - payload: scoreBatch( - artifacts, - model, - lastTarget, - name, - batch.header, - batch.cols, - decisionPolicy, - // V61: a live run builds its reference on the spot, from its own rows. - { - reference: referenceOf(artifacts, model, columnsAsMap()), - testRows: artifacts.testY.length, - }, - ), + kind: 'submission-built', + payload: { + ...file, + fileName: `${batch.name.replace(/\.[a-z0-9]+$/i, '')}-submission.csv`, + model: batch.model, + form: request.form, + refit: refit ? { rows: refit.rows, fittedRows: refit.fittedRows } : null, + }, }); } catch (error) { - post({ kind: 'batch-error', message: error instanceof Error ? error.message : String(error) }); + post({ + kind: 'submission-error', + message: error instanceof Error ? error.message : String(error), + }); } } @@ -219,6 +290,8 @@ self.onmessage = async (event: MessageEvent) => { } else if (request.kind === 'train') { cancelTraining = false; artifacts = null; + lastBatch = null; + lastConfig = null; lastTarget = request.config.target; const profiles: ColumnProfile[] = header.map((column, i) => profileColumn(column, columns[i]), @@ -257,6 +330,7 @@ self.onmessage = async (event: MessageEvent) => { ); if (outcome) { artifacts = outcome.artifacts; + lastConfig = request.config; lastFeatureColumns = outcome.summary.featureColumns; post({ kind: 'train-complete', @@ -527,6 +601,8 @@ self.onmessage = async (event: MessageEvent) => { message: error instanceof Error ? error.message : String(error), }); } + } else if (request.kind === 'build-submission') { + handleSubmission(request); } else if (request.kind === 'score-imported-file') { await handleScoreImported(request.file, request.file.name); } else if (request.kind === 'score-imported-url') { diff --git a/src/features/ml/lab-store.ts b/src/features/ml/lab-store.ts index f4ddc9b..695ff9e 100644 --- a/src/features/ml/lab-store.ts +++ b/src/features/ml/lab-store.ts @@ -20,6 +20,13 @@ import { fileKey, freshSeed, latestSeed, seedDealsTheTest } from '@/features/ml/ import { thresholdArtifact } from '@/features/ml/store/artifacts'; import { createResponseHandlers } from '@/features/ml/store/responses'; +/** V65: a submission belongs to one scored file and one model. */ +const noSubmission = { + submissionStatus: 'idle' as const, + submission: null, + submissionError: null, +}; + export { TEST_RATIO, TRAIN_SEED, @@ -301,6 +308,7 @@ export const useLabStore = create((set, get) => { batchStatus: 'idle', batchResult: null, batchError: null, + ...noSubmission, thresholdAnalysis: null, thresholdChoice: { threshold: 0.5, costFp: 1, costFn: 1 }, multiclassDecision: null, @@ -506,7 +514,7 @@ export const useLabStore = create((set, get) => { const state = get(); if (state.trainStatus !== 'done' || !state.insights || state.batchStatus === 'scoring') return; - set({ batchStatus: 'scoring', batchResult: null, batchError: null }); + set({ batchStatus: 'scoring', batchResult: null, batchError: null, ...noSubmission }); const decisionPolicy = activeDecisionPolicy(state); send({ kind: 'score-batch-file', @@ -561,7 +569,7 @@ export const useLabStore = create((set, get) => { const state = get(); if (state.trainStatus !== 'done' || !state.insights || state.batchStatus === 'scoring') return; - set({ batchStatus: 'scoring', batchResult: null, batchError: null }); + set({ batchStatus: 'scoring', batchResult: null, batchError: null, ...noSubmission }); const decisionPolicy = activeDecisionPolicy(state); send({ kind: 'score-batch-url', @@ -589,6 +597,13 @@ export const useLabStore = create((set, get) => { send({ kind: 'export-predictions', model: state.insights.model }); }, + buildSubmission(request) { + const state = get(); + if (state.batchStatus !== 'done' || state.submissionStatus === 'building') return; + set({ submissionStatus: 'building', submission: null, submissionError: null }); + send({ kind: 'build-submission', ...request }); + }, + exportPython() { const state = get(); if (state.trainStatus !== 'done' || !state.summary) return; diff --git a/src/features/ml/store/fence.ts b/src/features/ml/store/fence.ts index 2db507d..b584045 100644 --- a/src/features/ml/store/fence.ts +++ b/src/features/ml/store/fence.ts @@ -28,6 +28,8 @@ const RUN_DERIVED = new Set([ 'robust-cancelled', 'batch-scored', 'batch-error', + 'submission-built', + 'submission-error', 'threshold-result', 'multiclass-decision-result', 'multiclass-decision-tested', diff --git a/src/features/ml/store/responses.ts b/src/features/ml/store/responses.ts index 4d040f2..a75de55 100644 --- a/src/features/ml/store/responses.ts +++ b/src/features/ml/store/responses.ts @@ -245,13 +245,27 @@ export function createResponseHandlers({ attachArtifact({ forecast: message.payload }); }, 'batch-scored': (message) => { - set({ batchStatus: 'done', batchResult: message.payload, batchError: null }); + set({ + batchStatus: 'done', + batchResult: message.payload, + batchError: null, + submissionStatus: 'idle', + submission: null, + submissionError: null, + }); // The record keeps the numbers, never the row-level CSV. const artifact = { ...message.payload } as Partial; delete artifact.csv; delete artifact.preview; + delete artifact.header; attachArtifact({ batchScore: artifact as Omit }); }, + 'submission-built': (message) => { + set({ submissionStatus: 'done', submission: message.payload, submissionError: null }); + }, + 'submission-error': (message) => { + set({ submissionStatus: 'error', submission: null, submissionError: message.message }); + }, 'batch-error': (message) => { set({ batchStatus: 'error', batchResult: null, batchError: message.message }); }, diff --git a/src/features/ml/store/state.ts b/src/features/ml/store/state.ts index 8031562..f8f4d4f 100644 --- a/src/features/ml/store/state.ts +++ b/src/features/ml/store/state.ts @@ -12,6 +12,7 @@ import type { TaskInfo, } from '@/features/ml/data/types'; import type { BatchScore } from '@/features/ml/train/score'; +import type { Submission, SubmissionForm } from '@/features/ml/train/submission'; import type { ImportedManifest } from '@/features/ml/train/deserialize'; import type { SegmentAnalysis } from '@/features/ml/train/segments'; import type { ThresholdAnalysis } from '@/features/ml/train/threshold-analysis'; @@ -102,6 +103,10 @@ export interface LabState { batchStatus: 'idle' | 'scoring' | 'done' | 'error'; batchResult: BatchScore | null; batchError: string | null; + /** V65: the submission built from the last scored file. */ + submissionStatus: 'idle' | 'building' | 'done' | 'error'; + submission: Submission | null; + submissionError: string | null; /** Binary + probabilistic models only — null otherwise. */ thresholdAnalysis: ThresholdAnalysis | null; thresholdChoice: { threshold: number; costFp: number; costFn: number }; @@ -185,6 +190,14 @@ export interface LabState { exportPredictions: () => void; /** V64: the run as a scikit-learn script. */ exportPython: () => void; + /** V65: `idColumn` null numbers the rows from 1. */ + buildSubmission: (request: { + idColumn: string | null; + idName: string; + predictionName: string; + form: SubmissionForm; + refit: boolean; + }) => void; clearExportedFile: () => void; reset: () => void; } @@ -213,6 +226,9 @@ export const initialTraining = { batchStatus: 'idle' as const, batchResult: null, batchError: null, + submissionStatus: 'idle' as const, + submission: null as Submission | null, + submissionError: null as string | null, thresholdAnalysis: null as ThresholdAnalysis | null, thresholdChoice: { threshold: 0.5, costFp: 1, costFn: 1 }, multiclassDecision: null as LabState['multiclassDecision'], diff --git a/src/features/ml/train/refit.test.ts b/src/features/ml/train/refit.test.ts new file mode 100644 index 0000000..dab22e7 --- /dev/null +++ b/src/features/ml/train/refit.test.ts @@ -0,0 +1,87 @@ +import { describe, expect, it } from 'vitest'; +import { profileColumn } from '@/features/ml/data/profile'; +import { refitOnAllRows } from '@/features/ml/train/refit'; +import { prepareData } from '@/features/ml/train/trainer'; +import type { Cell } from '@/features/ml/data/types'; +import type { TrainConfig } from '@/features/ml/train/types'; + +// V65: the family fitted again on every labelled row. It repeats the run's +// configuration, sees the rows the run held out, and is deterministic. + +const N = 200; +const size = Array.from({ length: N }, (_, i) => String((i * 37) % 101)); +// A category only the last rows carry: some splits never show it to training. +const kind = Array.from({ length: N }, (_, i) => (i >= N - 3 ? 'rare' : i % 2 === 0 ? 'u' : 'v')); +const label = size.map((value) => (Number(value) > 50 ? 'yes' : 'no')); +const price = size.map((value, i) => String(Number(value) * 2 + (i % 5))); + +function setup(data: Record) { + const columns = new Map(Object.entries(data)); + const profiles = Object.entries(data).map(([name, values]) => profileColumn(name, values)); + return { columns, profiles }; +} + +const classification: TrainConfig = { + target: 'label', + features: ['size', 'kind'], + seed: 42, + testRatio: 0.2, +}; + +describe('refitOnAllRows', () => { + const { columns, profiles } = setup({ size, kind, label }); + + it('fits on every labelled row, the held-out ones included', () => { + const prepared = prepareData(columns, profiles, classification); + const refit = refitOnAllRows(columns, profiles, classification, 'tree')!; + expect(refit.rows).toBe(N); + expect(refit.fittedRows).toBe(N); + expect(prepared.train.length).toBeLessThan(N); + expect(refit.classes).toEqual(prepared.classes); + expect(refit.isClassification).toBe(true); + }); + + it('leaves out the rows with no label, as the run does', () => { + const holes = label.map((value, i) => (i % 10 === 0 ? '' : value)); + const data = setup({ size, kind, label: holes }); + const refit = refitOnAllRows(data.columns, data.profiles, classification, 'tree')!; + expect(refit.rows).toBe(N - N / 10); + }); + + it('gives the same model twice', () => { + const first = refitOnAllRows(columns, profiles, classification, 'forest')!; + const second = refitOnAllRows(columns, profiles, classification, 'forest')!; + const rows = Array.from({ length: N }, (_, i) => i); + expect(second.model.predict(second.pipeline.transform(rows))).toEqual( + first.model.predict(first.pipeline.transform(rows)), + ); + }); + + it('learns the rule from all the rows', () => { + const refit = refitOnAllRows(columns, profiles, classification, 'tree')!; + const rows = Array.from({ length: N }, (_, i) => i); + const predicted = refit.model.predict(refit.pipeline.transform(rows)); + const right = predicted.filter((value, i) => refit.classes[value] === label[i]).length; + expect(right / N).toBeGreaterThan(0.95); + }); + + it('keeps the shape the scorer expects: the same source columns', () => { + const refit = refitOnAllRows(columns, profiles, classification, 'logistic')!; + expect([...new Set(refit.pipeline.specs.map((spec) => spec.name))].sort()).toEqual([ + 'kind', + 'size', + ]); + const encoded = refit.pipeline.transformRow({ size: '12', kind: 'never seen' }); + expect(encoded.every((value) => Number.isFinite(value))).toBe(true); + }); + + it('fits a regression too, and refuses a family the task does not have', () => { + const data = setup({ size, kind, price }); + const regression: TrainConfig = { ...classification, target: 'price' }; + const refit = refitOnAllRows(data.columns, data.profiles, regression, 'linear')!; + expect(refit.isClassification).toBe(false); + expect(refit.rows).toBe(N); + expect(refitOnAllRows(data.columns, data.profiles, regression, 'logistic')).toBeNull(); + expect(refitOnAllRows(columns, profiles, classification, 'linear')).toBeNull(); + }); +}); diff --git a/src/features/ml/train/refit.ts b/src/features/ml/train/refit.ts new file mode 100644 index 0000000..0bdf98d --- /dev/null +++ b/src/features/ml/train/refit.ts @@ -0,0 +1,77 @@ +/** + * V65: one family, fitted again on every labelled row. + * + * A run fits on 64 % of the rows and keeps 16 % to choose and 20 % to + * measure. Once the choice is made and the measure taken, those 36 % have + * done their work, and a prediction for rows nobody has labelled can use + * them too. The price is stated wherever this model is used: it has no test + * set left. Its score is the one of the model fitted on fewer rows. + * + * Nothing else changes: the same columns, the same seed, the same announced + * caps (V25), the same class weighting. The preprocessing is fitted again on + * all the rows, so a category or a word that only the held-out rows carried + * now has its own column. + * + * Pure and deterministic. + */ +import { balancedWeights } from '@/features/ml/train/class-weight'; +import { MODEL_TRAIN_CAPS, modelZoo, type TrainedModel } from '@/features/ml/train/models'; +import { fitPipeline, type FittedPipeline } from '@/features/ml/train/pipeline'; +import { nestedSampleOrder } from '@/features/ml/train/random'; +import { prepareData } from '@/features/ml/train/trainer'; +import type { Cell, ColumnProfile } from '@/features/ml/data/types'; +import type { ModelKey, TrainConfig } from '@/features/ml/train/types'; + +export interface Refit { + model: TrainedModel; + pipeline: FittedPipeline; + /** Labelled rows the run had: train, validation and test together. */ + rows: number; + /** Rows the family was fitted on: fewer than `rows` when its cap applies. */ + fittedRows: number; + classes: string[]; + isClassification: boolean; +} + +/** Null when the run does not train this family (a regression has no logistic). */ +export function refitOnAllRows( + columns: Map, + profiles: ColumnProfile[], + config: TrainConfig, + key: ModelKey, +): Refit | null { + const prepared = prepareData(columns, profiles, config); + const { isClassification, classes, encode } = prepared; + const definition = modelZoo(isClassification ? 'classification' : 'regression').find( + (candidate) => candidate.key === key, + ); + if (!definition) return null; + + // Original order: a model must not depend on which split a row came from. + const rows = [...prepared.train, ...prepared.validation, ...prepared.test].sort((a, b) => a - b); + const pipeline = fitPipeline(columns, profiles, prepared.featureColumns, rows); + let x = pipeline.transform(rows); + let y = rows.map(encode); + + const cap = MODEL_TRAIN_CAPS[key]; + if (cap !== undefined && rows.length > cap) { + const labels = isClassification ? y.map((label) => classes[label] ?? null) : null; + const keep = nestedSampleOrder(rows.length, labels, config.seed) + .slice(0, cap) + .sort((a, b) => a - b); + x = keep.map((position) => x[position]); + y = keep.map((position) => y[position]); + } + + const weights = + config.classWeighting === 'balanced' && isClassification + ? balancedWeights(y, classes.length) + : undefined; + const model = definition.train(x, y, { + task: isClassification ? 'classification' : 'regression', + classCount: classes.length, + seed: config.seed, + ...(weights !== undefined && { classWeights: weights }), + }); + return { model, pipeline, rows: rows.length, fittedRows: y.length, classes, isClassification }; +} diff --git a/src/features/ml/train/score.ts b/src/features/ml/train/score.ts index 0a3a0d0..2e2f5ee 100644 --- a/src/features/ml/train/score.ts +++ b/src/features/ml/train/score.ts @@ -63,6 +63,14 @@ export interface BatchScore { }[]; /** Full predictions: every original column + predicted (+ probabilities). */ csv: string; + /** V65: the columns of the scored file, for the submission to pick its identifier. */ + header?: string[]; + /** V65: whether the model gives probabilities, which decides the forms offered. */ + hasProbabilities?: boolean; + classCount?: number; + /** V65: the class a binary probability speaks of: the second one. */ + positiveClass?: string; + isClassification?: boolean; } /** @@ -125,20 +133,24 @@ export function scoreBatch( ); } -/** Scores a parsed file with any RowScorer; `referenceMetrics` fills the - * honest comparison column (held-out test of the live or exporting run). */ -export function scoreRows( - scorer: RowScorer, - modelKey: ModelKey, - referenceMetrics: MetricMap, - target: string, - fileName: string, +/** + * The rows of a parsed file through a scorer: the encoded matrix, the + * predictions, and the probabilities when the family gives them. Shared by + * the batch score and the submission (V65), so both read a file the same way. + */ +export function predictRows( + scorer: Pick, header: string[], columns: Cell[][], - production?: ProductionContext, -): BatchScore { - const { model, classes, isClassification } = scorer; - +): { + index: Map; + required: string[]; + rowCount: number; + X: number[][]; + predictions: number[]; + probabilities: number[][] | null; +} { + const { model, isClassification } = scorer; const index = new Map(header.map((name, i) => [name, i])); const missing = requiredColumnsOf(scorer.specs).filter((name) => !index.has(name)); if (missing.length > 0) throw new Error(`missing-columns:${missing.join(', ')}`); @@ -162,6 +174,28 @@ export function scoreRows( const predictions = both?.labels ?? model.predict(X); const probabilities = isClassification && model.predictProba ? (both?.proba ?? model.predictProba(X)) : null; + return { index, required, rowCount, X, predictions, probabilities }; +} + +/** Scores a parsed file with any RowScorer; `referenceMetrics` fills the + * honest comparison column (held-out test of the live or exporting run). */ +export function scoreRows( + scorer: RowScorer, + modelKey: ModelKey, + referenceMetrics: MetricMap, + target: string, + fileName: string, + header: string[], + columns: Cell[][], + production?: ProductionContext, +): BatchScore { + const { model, classes, isClassification } = scorer; + + const { index, required, rowCount, X, predictions, probabilities } = predictRows( + scorer, + header, + columns, + ); const label = (value: number): string => isClassification ? (classes[value] ?? String(value)) : String(value); @@ -306,5 +340,10 @@ export function scoreRows( ...(check ? { check } : {}), preview, csv: lines.join('\n'), + header, + hasProbabilities: probabilities !== null, + classCount: classes.length, + ...(isClassification && classes.length === 2 && { positiveClass: classes[1] }), + isClassification, }; } diff --git a/src/features/ml/train/submission.test.ts b/src/features/ml/train/submission.test.ts new file mode 100644 index 0000000..de87b29 --- /dev/null +++ b/src/features/ml/train/submission.test.ts @@ -0,0 +1,157 @@ +import { describe, expect, it } from 'vitest'; +import { buildSubmission, formsFor, type SubmissionInput } from '@/features/ml/train/submission'; + +// V65: a submission has the rows of the scored file, in their order, under +// the names the user gave, and counts what a platform would refuse. + +function input(over: Partial = {}): SubmissionInput { + return { + ids: ['892', '893', '894', '895'], + idName: 'PassengerId', + predictionName: 'Survived', + form: 'label', + predictions: [0, 1, 1, 0], + probabilities: [ + [0.9, 0.1], + [0.25, 0.75], + [0.4, 0.6], + [0.8123456789, 0.1876543211], + ], + classes: ['no', 'yes'], + isClassification: true, + ...over, + }; +} + +describe('formsFor', () => { + it('offers only the forms the task has an object for', () => { + expect(formsFor(false, 0, false)).toEqual(['value']); + expect(formsFor(true, 2, true)).toEqual(['label', 'probability']); + expect(formsFor(true, 3, true)).toEqual(['label', 'probabilities']); + expect(formsFor(true, 3, false)).toEqual(['label']); + }); +}); + +describe('buildSubmission', () => { + it('writes the identifier and the class name, row for row and in order', () => { + const file = buildSubmission(input()); + expect(file.csv).toBe('PassengerId,Survived\n892,no\n893,yes\n894,yes\n895,no\n'); + expect(file.rows).toBe(4); + expect(file.header).toEqual(['PassengerId', 'Survived']); + expect(file.counts).toEqual([ + { label: 'no', count: 2 }, + { label: 'yes', count: 2 }, + ]); + expect(file.missingIds).toBe(0); + expect(file.duplicateIds).toBe(0); + }); + + it('writes the probability of the second class, to six decimals', () => { + const file = buildSubmission(input({ form: 'probability' })); + expect(file.csv.split('\n').slice(1, 5)).toEqual([ + '892,0.1', + '893,0.75', + '894,0.6', + '895,0.187654', + ]); + expect(file.range?.min).toBeCloseTo(0.1); + expect(file.range?.max).toBeCloseTo(0.75); + expect(file.counts).toBeUndefined(); + }); + + it('writes one column per class in multiclass, named after the classes', () => { + const file = buildSubmission( + input({ + form: 'probabilities', + ids: ['a', 'b'], + predictions: [2, 0], + probabilities: [ + [0.1, 0.2, 0.7], + [0.5, 0.3, 0.2], + ], + classes: ['setosa', 'versicolor', 'virginica'], + predictionName: '', + }), + ); + expect(file.csv).toBe( + 'PassengerId,setosa,versicolor,virginica\na,0.1,0.2,0.7\nb,0.5,0.3,0.2\n', + ); + }); + + it('writes the value of a regression untouched', () => { + const file = buildSubmission( + input({ + form: 'value', + isClassification: false, + classes: [], + probabilities: null, + predictions: [23.456789012, 18, 31.5, 20], + predictionName: 'mpg', + }), + ); + expect(file.csv.split('\n')[1]).toBe('892,23.456789012'); + expect(file.range).toEqual({ min: 18, max: 31.5, mean: (23.456789012 + 18 + 31.5 + 20) / 4 }); + }); + + it('numbers the rows from 1 when no column identifies them', () => { + const file = buildSubmission(input({ ids: null, idName: 'id' })); + expect(file.csv.split('\n').slice(0, 3)).toEqual(['id,Survived', '1,no', '2,yes']); + }); + + it('keeps a row whose identifier is missing or repeated, and counts it', () => { + const file = buildSubmission(input({ ids: ['892', null, ' 892 ', ''] })); + expect(file.rows).toBe(4); + expect(file.csv.split('\n').slice(1, 5)).toEqual(['892,no', ',yes', '892,yes', ',no']); + expect(file.missingIds).toBe(2); + expect(file.duplicateIds).toBe(1); + }); + + it('escapes names and values that carry a comma or a quote', () => { + const file = buildSubmission( + input({ + ids: ['a,1', 'b"2', 'c', 'd'], + idName: 'id, of row', + predictionName: 'the "answer"', + classes: ['no, never', 'yes'], + }), + ); + const lines = file.csv.split('\n'); + expect(lines[0]).toBe('"id, of row","the ""answer"""'); + expect(lines[1]).toBe('"a,1","no, never"'); + expect(lines[2]).toBe('"b""2",yes'); + }); + + it('refuses by name what would not make a readable file', () => { + expect(() => buildSubmission(input({ idName: ' ' }))).toThrow('name-empty'); + expect(() => buildSubmission(input({ predictionName: '' }))).toThrow('name-empty'); + expect(() => buildSubmission(input({ predictionName: ' PassengerId ' }))).toThrow('name-clash'); + expect(() => buildSubmission(input({ form: 'value' }))).toThrow('form-unavailable'); + expect(() => buildSubmission(input({ form: 'probability', probabilities: null }))).toThrow( + 'form-unavailable', + ); + expect(() => buildSubmission(input({ ids: ['1'] }))).toThrow('rows-mismatch'); + expect(() => + buildSubmission( + input({ + form: 'probabilities', + classes: ['a', 'PassengerId', 'c'], + probabilities: [[1, 0, 0]], + predictions: [0], + ids: ['1'], + }), + ), + ).toThrow('name-clash'); + }); + + it('previews the header and the first lines only', () => { + const file = buildSubmission( + input({ + ids: null, + predictions: Array.from({ length: 30 }, () => 1), + probabilities: Array.from({ length: 30 }, () => [0.2, 0.8]), + }), + ); + expect(file.preview).toHaveLength(5); + expect(file.preview[0]).toBe('PassengerId,Survived'); + }); +}); diff --git a/src/features/ml/train/submission.ts b/src/features/ml/train/submission.ts new file mode 100644 index 0000000..ddd51ca --- /dev/null +++ b/src/features/ml/train/submission.ts @@ -0,0 +1,188 @@ +/** + * V65: a file a competition platform accepts. + * + * Scoring a batch (V7) gives back every original column plus the prediction: + * the right file to join back into a system, the wrong one for a platform + * that expects two columns under its own names, `PassengerId,Survived`. + * + * The lab knows neither the platform nor its rules, so it guesses nothing: + * the user names the identifier column, the prediction column and what a + * prediction is (a label, a probability, a value). The lab keeps one promise, + * that the file has exactly the rows of the scored file, in their order, and + * counts what a platform would refuse: an identifier missing or repeated. + * + * Which model predicts is the other half of the wave, see `refit.ts`. + * Measured before building, on the six demos, ten splits each, with a + * quarter of the rows kept away from both models as a private set. The same + * family fitted on every labelled row instead of 64 % of them: + * + * demo labelled rows mean gain on the private set better / worse + * reviews 180 +5.0 accuracy points 51 / 11 of 70 + * mpg 298 RMSE down 3 % 47 / 13 of 60 + * energy 180 RMSE down 2 % 44 / 16 of 60 + * titanic 668 +0.8 accuracy point 41 / 20 of 70 + * fraud 480 none (-0.04 point) 21 / 25 of 70 + * iris 112 none (-0.1 point) 14 / 17 of 70 + * + * The gain follows the learning curve, not the size of the file: iris is the + * smallest demo and gains nothing, because 72 rows already say all there is + * to say. Where the curve still climbs the refit pays, and even there it + * loses one time in five to one time in three. So the refit is offered and + * never the default, and it is announced for what it is: a model nobody + * measured. + * + * Pure and deterministic. + */ +import { isMissing } from '@/features/ml/data/infer'; +import type { Cell } from '@/features/ml/data/types'; +import type { ModelKey } from '@/features/ml/train/types'; +import { csvCell } from '@/lib/csv'; + +/** + * What one prediction is, in the file: + * - `label`: the predicted class, by name; + * - `probability`: the probability of the second class (binary only); + * - `probabilities`: one column per class, named after it (three classes or more); + * - `value`: the predicted number (regression). + */ +export type SubmissionForm = 'label' | 'probability' | 'probabilities' | 'value'; + +const PREVIEW_LINES = 4; +/** Six decimals: below what any leaderboard metric can tell apart. */ +const PROBABILITY_DIGITS = 6; + +/** The forms that make sense for a task. A form with no object is not offered. */ +export function formsFor( + isClassification: boolean, + classCount: number, + hasProbabilities: boolean, +): SubmissionForm[] { + if (!isClassification) return ['value']; + if (!hasProbabilities) return ['label']; + return classCount === 2 ? ['label', 'probability'] : ['label', 'probabilities']; +} + +export interface SubmissionInput { + /** Identifier cells of the scored file, or null to number the rows from 1. */ + ids: readonly Cell[] | null; + idName: string; + /** Unused by `probabilities`, whose columns carry the class names. */ + predictionName: string; + form: SubmissionForm; + predictions: readonly number[]; + probabilities: readonly (readonly number[])[] | null; + classes: readonly string[]; + isClassification: boolean; +} + +export interface SubmissionFile { + csv: string; + header: string[]; + /** Data rows written: always the row count of the scored file. */ + rows: number; + /** Rows whose identifier cell is empty. They are written, and counted. */ + missingIds: number; + /** Rows whose identifier was already used by an earlier row. */ + duplicateIds: number; + /** Predicted classes and their counts, most frequent first (`label`). */ + counts?: { label: string; count: number }[]; + /** Spread of the written numbers (`probability`, `value`). */ + range?: { min: number; max: number; mean: number }; + /** First lines of the file, header included. */ + preview: string[]; +} + +/** What crosses the worker boundary: the file, and which model predicted it. */ +export interface Submission extends SubmissionFile { + fileName: string; + model: ModelKey; + form: SubmissionForm; + /** Set when the family was fitted again on every labelled row. */ + refit: { rows: number; fittedRows: number } | null; +} + +function probabilityText(value: number): string { + return String(Number(value.toFixed(PROBABILITY_DIGITS))); +} + +/** + * Throws a named error when the request cannot give a file a platform would + * read: `name-empty`, `name-clash`, `form-unavailable`, `rows-mismatch`. + */ +export function buildSubmission(input: SubmissionInput): SubmissionFile { + const { ids, form, predictions, probabilities, classes, isClassification } = input; + const idName = input.idName.trim(); + const predictionName = input.predictionName.trim(); + const classCount = classes.length; + if (!formsFor(isClassification, classCount, probabilities !== null).includes(form)) { + throw new Error('form-unavailable'); + } + if (ids !== null && ids.length !== predictions.length) throw new Error('rows-mismatch'); + if (idName === '') throw new Error('name-empty'); + + const header = form === 'probabilities' ? [idName, ...classes] : [idName, predictionName]; + if (form !== 'probabilities' && predictionName === '') throw new Error('name-empty'); + if (new Set(header).size !== header.length) throw new Error('name-clash'); + + let missingIds = 0; + let duplicateIds = 0; + const seen = new Set(); + const counts = new Map(); + let min = Infinity; + let max = -Infinity; + let sum = 0; + const note = (value: number) => { + min = Math.min(min, value); + max = Math.max(max, value); + sum += value; + }; + + const lines = [header.map(csvCell).join(',')]; + for (let row = 0; row < predictions.length; row++) { + let id: string; + if (ids === null) { + id = String(row + 1); + } else { + const cell = ids[row]; + id = isMissing(cell) ? '' : String(cell).trim(); + if (id === '') missingIds += 1; + else if (seen.has(id)) duplicateIds += 1; + else seen.add(id); + } + + const cells = [csvCell(id)]; + const predicted = predictions[row]; + if (form === 'label') { + const label = classes[predicted] ?? String(predicted); + counts.set(label, (counts.get(label) ?? 0) + 1); + cells.push(csvCell(label)); + } else if (form === 'value') { + note(predicted); + cells.push(String(predicted)); + } else if (form === 'probability') { + const value = probabilities![row][1] ?? 0; + note(value); + cells.push(probabilityText(value)); + } else { + for (const value of probabilities![row]) cells.push(probabilityText(value)); + } + lines.push(cells.join(',')); + } + + const rows = predictions.length; + return { + csv: lines.join('\n') + '\n', + header, + rows, + missingIds, + duplicateIds, + ...(form === 'label' && { + counts: [...counts] + .map(([label, count]) => ({ label, count })) + .sort((a, b) => b.count - a.count || a.label.localeCompare(b.label)), + }), + ...((form === 'value' || form === 'probability') && + rows > 0 && { range: { min, max, mean: sum / rows } }), + preview: lines.slice(0, PREVIEW_LINES + 1), + }; +} diff --git a/src/features/ml/worker-protocol.ts b/src/features/ml/worker-protocol.ts index 80206ef..2c82941 100644 --- a/src/features/ml/worker-protocol.ts +++ b/src/features/ml/worker-protocol.ts @@ -4,6 +4,7 @@ import type { ImportedManifest } from '@/features/ml/train/deserialize'; import type { ExplorationPayload } from '@/features/ml/unsupervised/explore'; import type { ForecastPayload } from '@/features/ml/timeseries/run'; import type { BatchScore } from '@/features/ml/train/score'; +import type { Submission, SubmissionForm } from '@/features/ml/train/submission'; import type { SegmentAnalysis } from '@/features/ml/train/segments'; import type { TunableKey, TuneOutcome } from '@/features/ml/train/search'; import type { LearningCurveOutcome } from '@/features/ml/train/learning-curve'; @@ -76,6 +77,19 @@ export type WorkerRequest = model: ModelKey; decisionPolicy?: MulticlassDecisionPolicy; } + /** + * V65: the last file scored by the live run, as a file a platform accepts. + * `idColumn` null numbers the rows from 1; `refit` fits the family again on + * every labelled row first. + */ + | { + kind: 'build-submission'; + idColumn: string | null; + idName: string; + predictionName: string; + form: SubmissionForm; + refit: boolean; + } | { kind: 'threshold-analysis'; model: ModelKey; focusClass?: number } | { kind: 'multiclass-decision-analysis'; model: ModelKey } | { @@ -123,6 +137,9 @@ export type WorkerResponse = | { kind: 'predictions-csv'; model: ModelKey; csv: string } | { kind: 'python-script'; name: string; content: string } | { kind: 'batch-scored'; payload: BatchScore } + | { kind: 'submission-built'; payload: Submission } + // Dedicated channel: a refused submission leaves the scored batch in place. + | { kind: 'submission-error'; message: string } // The current dataset rebuilt as CSV, for local persistence. | { kind: 'dataset-csv'; csv: string } // null = not applicable (multiclass, regression, or no probabilities). diff --git a/src/locales/en.json b/src/locales/en.json index 8e51499..f6f1adc 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -466,6 +466,47 @@ "noDecision": "no decision", "note": "A drop against the test set is a signal, not a bug: the world moved. The production check above says which columns shifted, and whether the drop can be told from the draw." }, + "submission": { + "title": "Submission file", + "hint": "For a competition: the file scored above, reduced to an identifier and a prediction under the names the platform expects. The lab knows neither the platform nor its rules, so you name everything. The file keeps every row of the scored file, in its order.", + "idColumn": "Identifier column", + "idChoose": "choose a column…", + "idRowNumber": "no column: number the rows from 1", + "idName": "Its name in the submission", + "predictionName": "Name of the prediction column", + "suggestion": "\"{{column}}\" reads as an identifier in the training file. The choice stays yours.", + "clash": "The two columns need different names.", + "formTitle": "What a prediction is", + "form": { + "label": "the predicted class, by name (accuracy, F1)", + "probability": "the probability of \"{{positive}}\" (AUC, log loss)", + "probabilities": "one probability per class, each column named after its class (multiclass log loss)", + "value": "the predicted value (RMSE, MAE)" + }, + "modelTitle": "Which model predicts", + "model": { + "measured": "{{model}} as measured: fitted on 64 % of the labelled rows, scored on the test set", + "refit": "the same family fitted again on every labelled row: more data, and no test set left to measure it" + }, + "refitNote": "Measured on the lab's demos, on rows kept away from both models: fitting again on everything gained 5 accuracy points on reviews (180 rows), 0.8 on titanic, 2 to 3 % of RMSE on mpg and energy, and nothing on fraud and iris. Even where it pays it loses one time in five to one time in three. The learning curve says which case yours is: a curve still climbing is a reason to refit, a flat one is not.", + "refitUnavailable": "The blend cannot be fitted again on every row: its members are picked on the validation rows, and a fit on everything leaves none to pick with. Inspect one family to refit it.", + "build": "Build the submission", + "building": "Building…", + "refitting": "Fitting again on every labelled row…", + "error": "The submission could not be built ({{detail}}).", + "summary_one": "{{rows}} row, columns: {{columns}}.", + "summary_other": "{{rows}} rows, the same as the scored file and in its order. Columns: {{columns}}.", + "byMeasured": "Predicted by {{model}}, the model the test set measured.", + "byRefit": "Predicted by {{model}} fitted again on the {{rows}} labelled rows. Nobody measured this model: the test score above belongs to its sibling fitted on fewer rows.", + "byRefitCapped": "Predicted by {{model}} fitted again on {{fitted}} of the {{rows}} labelled rows, its announced cap. Nobody measured this model: the test score above belongs to its sibling fitted on fewer rows.", + "missingIds_one": "{{count}} row has no identifier: it is written with an empty one, and a platform will likely refuse it.", + "missingIds_other": "{{count}} rows have no identifier: they are written with an empty one, and a platform will likely refuse them.", + "duplicateIds_one": "{{count}} row repeats an identifier already used.", + "duplicateIds_other": "{{count}} rows repeat an identifier already used.", + "range": "Written values: from {{min}} to {{max}}, mean {{mean}}.", + "download": "Download the submission (CSV)", + "note": "A submission is not a score. The lab does not know the platform's metric nor its private rows, and promises no rank. And a public leaderboard wears like a test set: each submission you compare and keep the best of is a choice made with it." + }, "threshold": { "title": "Decision threshold & imbalance", "hint": "Only {{rate}} % of rows are “{{positive}}”: accuracy flatters any model here. The precision-recall curve tells the truth, the calibration curve says whether the probabilities can be trusted — and the threshold is YOUR decision, priced by your costs.", diff --git a/src/locales/fr.json b/src/locales/fr.json index 513ec5a..9b371ba 100644 --- a/src/locales/fr.json +++ b/src/locales/fr.json @@ -466,6 +466,47 @@ "noDecision": "aucune décision", "note": "Une baisse par rapport au jeu de test est un signal, pas un bug : le monde a bougé. La vérification en production ci-dessus dit quelles colonnes ont glissé, et si la baisse se distingue du tirage." }, + "submission": { + "title": "Fichier de soumission", + "hint": "Pour une compétition : le fichier scoré ci-dessus, réduit à un identifiant et une prédiction, sous les noms que la plateforme attend. Le labo ne connaît ni la plateforme ni ses règles : c'est vous qui nommez tout. Le fichier garde toutes les lignes du fichier scoré, dans leur ordre.", + "idColumn": "Colonne d'identifiant", + "idChoose": "choisir une colonne…", + "idRowNumber": "aucune colonne : numéroter les lignes à partir de 1", + "idName": "Son nom dans la soumission", + "predictionName": "Nom de la colonne de prédiction", + "suggestion": "« {{column}} » se lit comme un identifiant dans le fichier d'entraînement. Le choix reste le vôtre.", + "clash": "Les deux colonnes doivent porter des noms différents.", + "formTitle": "Ce qu'est une prédiction", + "form": { + "label": "la classe prédite, par son nom (exactitude, F1)", + "probability": "la probabilité de « {{positive}} » (AUC, log loss)", + "probabilities": "une probabilité par classe, chaque colonne au nom de sa classe (log loss multiclasse)", + "value": "la valeur prédite (RMSE, MAE)" + }, + "modelTitle": "Quel modèle prédit", + "model": { + "measured": "{{model}} tel que mesuré : entraîné sur 64 % des lignes étiquetées, noté sur le jeu de test", + "refit": "la même famille réentraînée sur toutes les lignes étiquetées : plus de données, et plus aucun jeu de test pour la mesurer" + }, + "refitNote": "Mesuré sur les démos du labo, sur des lignes tenues à l'écart des deux modèles : réentraîner sur tout a gagné 5 points d'exactitude sur reviews (180 lignes), 0,8 sur titanic, 2 à 3 % de RMSE sur mpg et energy, et rien sur fraud et iris. Même là où il rapporte, il perd une fois sur cinq à une fois sur trois. La courbe d'apprentissage dit dans quel cas vous êtes : une courbe qui monte encore est une raison de réentraîner, une courbe plate n'en est pas une.", + "refitUnavailable": "Le mélange ne peut pas être réentraîné sur toutes les lignes : ses membres sont choisis sur les lignes de validation, et un entraînement sur tout n'en laisse aucune pour choisir. Inspectez une famille pour la réentraîner.", + "build": "Construire la soumission", + "building": "Construction…", + "refitting": "Réentraînement sur toutes les lignes étiquetées…", + "error": "La soumission n'a pas pu être construite ({{detail}}).", + "summary_one": "{{rows}} ligne, colonnes : {{columns}}.", + "summary_other": "{{rows}} lignes, les mêmes que le fichier scoré et dans son ordre. Colonnes : {{columns}}.", + "byMeasured": "Prédit par {{model}}, le modèle que le jeu de test a mesuré.", + "byRefit": "Prédit par {{model}} réentraîné sur les {{rows}} lignes étiquetées. Personne n'a mesuré ce modèle : le score de test ci-dessus est celui de son frère entraîné sur moins de lignes.", + "byRefitCapped": "Prédit par {{model}} réentraîné sur {{fitted}} des {{rows}} lignes étiquetées, son plafond annoncé. Personne n'a mesuré ce modèle : le score de test ci-dessus est celui de son frère entraîné sur moins de lignes.", + "missingIds_one": "{{count}} ligne n'a pas d'identifiant : elle est écrite avec un identifiant vide, et une plateforme la refusera sans doute.", + "missingIds_other": "{{count}} lignes n'ont pas d'identifiant : elles sont écrites avec un identifiant vide, et une plateforme les refusera sans doute.", + "duplicateIds_one": "{{count}} ligne répète un identifiant déjà utilisé.", + "duplicateIds_other": "{{count}} lignes répètent un identifiant déjà utilisé.", + "range": "Valeurs écrites : de {{min}} à {{max}}, moyenne {{mean}}.", + "download": "Télécharger la soumission (CSV)", + "note": "Une soumission n'est pas un score. Le labo ne connaît ni la métrique de la plateforme ni ses lignes privées, et ne promet aucun rang. Et un classement public s'use comme un jeu de test : chaque soumission que vous comparez pour garder la meilleure est un choix fait avec lui." + }, "threshold": { "title": "Seuil de décision & déséquilibre", "hint": "Seulement {{rate}} % des lignes sont « {{positive}} » : ici, l’accuracy flatte n’importe quel modèle. La courbe précision-rappel dit la vérité, la courbe de calibration dit si les probabilités sont fiables — et le seuil est VOTRE décision, chiffrée par vos coûts.",