diff --git a/CHANGELOG.md b/CHANGELOG.md index ef67a70..a065e60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and the reason the wave was built. +## V70 — Twin columns are flagged, not removed + +G2. + +_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. + ## V69 — A SQL result reaches the lab whole G1, the first of the ten proposals a new reading of the repository produced after V68 (the others are listed below). diff --git a/PLAN.md b/PLAN.md index 8e2bae1..7dee061 100644 --- a/PLAN.md +++ b/PLAN.md @@ -680,7 +680,7 @@ defect that produces a wrong number comes first), then extends the lab past the export: uncertainty with a guarantee, governance, and monitoring after deployment, with nothing added on a server. -V53 to V69 are delivered. Every other row is a proposal with its launch +V53 to V70 are delivered. Every other row is a proposal with its launch condition, and none starts without an explicit launch command. | Wave | Content | Why | @@ -703,24 +703,24 @@ condition, and none starts without an explicit launch command. | **V67 — delivered** | **Early stopping for the boosting: measured, and not built.** F9, opened by V66, which had shown the boosting's validation error lowest well before its 120th round on 17 runs of 30. The obvious reading is overfitting and the obvious fix is to stop at that round. This wave measured the fix before writing it, and the measure says no. **The trap seen before measuring**: the validation rows choose the family (V35, V53). A boosting that also chooses its number of rounds on them would be scored on rows it tuned itself on, against families that tuned nothing there. So three stops were compared, not one: (A) at the lowest error on the run's validation rows; (B) at the lowest error on 15 or 20 % of the training rows held out inside the boosting, keeping that smaller model; (C) the round found by B, applied to the model fitted on all the training rows. Plus a stop by patience (10 or 20 rounds without progress) against the global minimum. **Measured on six demos, ten splits each, a quarter of the rows kept away from everything, against the 120 fixed rounds, on the metric the leaderboard ranks on.** Titanic: A gains 0.4 accuracy point, better on 5 splits and worse on 4. Fraud, iris and mpg: no change. **Reviews: A loses 4.0 accuracy points** (worse 6, better 2), B and C lose 5 to 7. Energy: RMSE 3.5 % lower with A (better 9, worse 1), where the stop is often round 0, the mean of the training rows: there the boosting does no better than the baseline, and it is the family chosen on validation on none of the ten splits. B and C did no better than A on five demos of six, and B, which gives up rows, was the worst of the three on four demos. Patience and global minimum gave the same result within noise. **What does improve is the log loss**: better on 10 splits of 10 on titanic (0.02) and on reviews (0.10). Fewer rounds give better probabilities and no better predictions, and on reviews both at once: a lower log loss and 4 points less accuracy. V66's curve is a log loss in classification, which is why it turned early while the classes predicted kept improving or held. **The trap was real**: with A the boosting's validation score rose above its score on the private rows by 0.9 accuracy point on titanic, 6 points on reviews, 0.04 of RMSE on mpg and 0.18 on energy, beyond what it already did at round 120. On the choice of the family it changed little, and never for the better: the family chosen on validation scored 0.828 on the private rows with A against 0.832 without on titanic, an RMSE of 3.034 against 3.023 on mpg, the same elsewhere. **Decision, by the rule the plan had set (« ship only what the measure justifies »)**: nothing changes in training. The boosting keeps its 120 rounds, no figure of any run moves, and no golden reference is regenerated. **What ships**: the measure, written in `gbdt.ts` where the next reader will look for early stopping; the glass box's sentence, which said the later rounds « did not help » and let the reader conclude that fewer rounds would predict better, now says what was measured; the how-to page and the limits page in both languages carry the table. F9 is closed. **Defects of this wave's own making**: V66's own wording, above, was the defect: it read a log-loss curve as a verdict on the predictions. **What this wave deliberately does not do**: stop early when the user ranks on log loss, where the measure does favour fewer rounds on two demos of four: the lab ranks on accuracy or RMSE by default, and a stop tied to the ranking metric is another design, with the validation trap to solve first; tune depth or learning rate; touch the network or the forest. 1294 unit tests, 133 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | | **V68 — delivered** | **Counterfactuals: what would have to change.** F7, the last feature proposed. For a row typed in the what-if panel the lab said what the model predicts, what weighed (Shapley, V28) and what happened (the glass box, V66). It did not answer the question a person told « no » asks: what would have had to be different? The user looked for it by hand, one field at a time. **A search a reader can follow, not an optimisation**: one column at a time, every value the training rows hold tried in place of the row's own, the column keeping the nearest that flips the answer; two columns together only when no single one flips it; never three. Only observed values, no interpolation; free text is never edited; the columns the user locks are left alone, since the lab does not guess that an age cannot change. Nothing found is an answer, given with what was tried. The plan spoke of refining a numeric threshold by bisection; that would have produced values no row holds, against the rule just stated, and was dropped. **Measured on the four classification demos, three seeds, 60 test rows each, seven families.** One column is enough on 63 to 98 % of titanic's rows, 28 to 100 % of fraud's, 96 to 100 % of iris's; two are needed on up to 30 % (titanic, logistic) and 69 % (fraud, forest), so the pair search earns its place. On reviews most families find nothing (100 % of rows for the naive Bayes, 97 % for the tree): the answer is in the text. A search costs 30 ms at most (the forest), 100 to 550 predictions: no progress indicator is needed. **Two findings changed the design.** (1) **Plausibility**: 8 to 60 % of the nearest changes landed farther from every training row than 95 % of training rows are from their nearest neighbour; the model was answering where it had seen nothing. The search now keeps, per column, the nearest flipping value that stays inside that cloud, and flags the rest. It helps where an alternative exists (the tree on titanic from 8 to 4 %, the naive Bayes on fraud from 14 to 0 %) and not where none does: about half of iris's answers stay flagged, its four columns moving together. (2) **Fragility**: of the other trained models that agreed on the row, only 14 to 54 % answered the new class for the same change. A minimal change lands just across one model's boundary; each answer therefore says how many of the others follow. **What the flag does not catch**, seen on the screenshot review: on titanic the forest proposed `adult_male`: True to False while `sex` stayed `male`. Each value exists, the row does not, and a 0/1 column moving alone is not far from the cloud. The lab does not know two columns say the same thing; the panel and the limits page say so and tell the user to lock or drop the duplicate. **Defects of this wave's own making, caught before the push**: (1) the first search returned the nearest change whatever it was, above; (2) a category loaded into the form from a counterfactual could be rarer than the eight the form lists, and the field then showed another value than the one predicted on: the form now lists the loaded value; (3) the lock checkbox was first put inside the label of its field, so one label named two controls: two existing end-to-end tests that fill a field by its label failed, and the lock is now a control of its own beside the label; (4) the new page quoted a duration in milliseconds, which the documentation refuses since a test says so. **What this wave deliberately does not do**: give advice about the world, a counterfactual speaks of the model and the panel says so; search beyond two columns or edit free text; detect redundant columns; apply the chosen decision threshold (the change flips the prediction the what-if panel shows); handle a regression, where « another class » has no meaning, or a re-imported model, which carries no training rows; implement a published method (DiCE and others). A counterfactual is shown and saved nowhere. The list of proposed features is now empty. 1320 unit tests, 134 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | | **V69 — delivered** | **A SQL result reaches the lab whole.** G1, the first of the ten proposals a new reading of the repository produced after V68 (the others are listed below). The how-to page presents « filter, join or aggregate in SQL, then send to the ML Lab » as the normal path. **What the code said**: the console paints 200 rows, which is sound, and the three ways out of it were built from those painted rows. `toLab` and the CSV export sent `tableToCsv(result)`, a table already cut to 200. A note in the interface said so; it still emptied the gesture: nobody could train on a SQL result of more than 200 rows, and the demo file, 118 rows long, hid it. The cut saved no memory either, the engine having loaded the whole result before cutting. **Measured before the fix**, in the browser with the real engine, on a query returning 590 rows with a NULL in a numeric column: the lab received **200 rows**, and read the column as **categorical**, the NULL having left as `∅`, the mark the console paints for it, which the lab took for a category. **The fix**: an export is the query, not the screen. `SqlEngine.toCsv` asks DuckDB for the file, `COPY (…) TO … (FORMAT CSV, HEADER)`, as `toParquet` already did; the CSV export, the lab handoff and a Parquet button (the method existed since V40 and no button called it, though the reference announced it) all go through the engine, on the query that produced the result on screen, not on the text of the box, which may have been edited since. **Measured after**: the 590 rows arrive, the column is numeric with missing values, and a plain `SELECT *` over each of the lab's six demos gives back the same rows, the same columns and the same inferred types as the file loaded directly. Writing 118,000 rows took 0.2 s and 4.9 MB, 1,180,000 rows 6.4 s and 49 MB, on the single-threaded engine. The lab's cell budget (V54, 20 million) is applied before anything is built, so a result the lab would refuse is refused in the console, by name. **A file name of its own.** Every handoff was `query.csv`; the test-set ledger (V60) knows a file by its name and shape, and took two queries of the same shape for one file. The name now carries a fingerprint of the query, never its text; the FNV-1a function the ledger had moved to `lib/fingerprint.ts` instead of being copied. **Defects of this wave's own making**: none found by the suites; the fake engine of the panel's test had been answering `total` where the type says `totalRows` since V47, unnoticed because no test read it. **What this wave deliberately does not do**: make SQL read a file the way the rest of the app does (decimal comma, windows-1252, Excel, the cleaned or joined table): what leaves the console is complete, what enters it is still DuckDB's own reading, and the limits page says so; carry the recipe or the contract with the handoff; raise the 200 painted rows or paginate the console; change the engine or its version. 1330 unit tests, 134 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | +| **V70 — delivered** | **Twin columns are flagged, not removed.** G2. The exclusion suggestions knew an identifier, a constant, a near-empty column and a leak; the functional-dependency test ran from a column to the target only. Titanic kept `class` and `pclass`, `embarked` and `embark_town`, `who` and `adult_male` as separate features. Twins split the importance and the Shapley values between them, and V68 had proposed a passenger who cannot exist by changing one without the other; its limits row told the user to deal with it alone. **The test**: the one `leakSuggestions` runs towards the target, run between columns. Each value of A goes with one value of B, both ways: they say the same thing. One way only: B can be read off A. Two continuous numbers: the near-perfect correlation of the leak test. It is target-independent, so it is computed when the file is read. **Measured before the interface, on the ten demo files.** Titanic: `pclass`/`class`, `embarked`/`embark_town`, `adult_male` read off `who`, and `survived`/`alive`, which is the leak and is flagged as one. Exactly the known truth, and **nothing on the nine other files**: an exact test on values seen at least five times on average produced no false twin, so no tolerance was added. `sex` and `who` are not flagged, rightly: neither determines the other, a child being of either sex. **What removing a twin changes**, titanic, ten seeds, eight families: mean accuracy 0.813 with the thirteen columns, 0.814 without `class`, `embark_town` and `adult_male`. No score moves. Removing `who` as well costs 0.9 point (5.5 for the naive Bayes): a column another is read off is not itself redundant. So this is about reading a model, not about its performance, and **the lab flags and leaves both in**: a redundancy is a note on the card, not an exclusion reason, and which one to keep is the user's call. **What it changes for counterfactuals** (V68), same rows, four families: with the thirteen columns, 296 of 695 first counterfactuals (43 %) changed one column of a pair alone; with the twins excluded, none could. A counterfactual that touches one twin without the other now says so, naming both. **Defect of this wave's own making, caught by the measure**: the first version read and trimmed every cell again for every pair, 34 s on a file of 200 columns and 100,000 rows. Each column is now read once into integer codes, on a sample of 20,000 rows, at most 80 columns: 1.2 s on the same file, the same pairs on every demo. **What this wave deliberately does not do**: exclude a twin or choose which to keep; see a fact carried by three columns together (`who` from `sex` and `age`, `alone` from `sibsp` and `parch`); measure an approximate redundancy (mutual information, VIF). 1344 unit tests, 135 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | Proposed features, ranked by value over effort. Each reuses a module that exists; none is scheduled. F1 to F9 were delivered, or measured and closed, through V68. G1 to G10 come from a new -reading of the repository after V68; G1 shipped as V69. - -| Feature | Content | Reuses | -| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | -| G2 | **Flag columns that say the same thing.** `ExclusionReason` knows `id`, `constant`, `nearEmpty`, `leak`, `unsupported`; the functional-dependency check runs from a column to the target only. Titanic keeps `class`/`pclass`, `embarked`/`embark_town`, `sex`/`who`/`adult_male` as separate features, which splits importance between twins and let V68 propose a passenger who cannot exist. To measure first: true and false duplicates on the demos. The lab flags, it does not remove. | `data/suggest.ts`, `data/types.ts` | -| G3 | **Say how many cells were filled.** A missing or unreadable number becomes the median, a missing category the most frequent one, with no count anywhere. On titanic `deck` is 77 % empty, under the `nearEmpty` cut, and three quarters of it are filled silently. A count per column in the run, the report and the model card; unreadable cells apart from empty ones. | `train/pipeline.ts`, `projects/model-card.ts` | -| G4 | **Rank on the quality of probabilities.** The leaderboard ranks on accuracy, F1, recall, precision, AUC, RMSE, MAE, R². Log loss is computed and shown and cannot rank; nor can Brier. The threshold, conformal and what-if panels all consume probabilities. It is the question V67 left open. Then: the calibration curve for every family, with an ECE. | `train/ranking.ts`, `train/threshold-analysis.ts` | -| G5 | **An honest forecast band.** The band is the 10th and 90th percentiles of one-step errors, applied flat to a horizon of up to 28 steps, on the hold-out that chose the method (the trap V57 measured). Errors per horizon step, a measured coverage, and a download of the forecast. | `timeseries/backtest.ts`, `timeseries/run.ts` | -| G6 | **Decisions travel with the export.** The export keeps the multiclass decision rule, not the binary threshold chosen nor the conformal quantile: the same model decides differently live and re-imported. | `train/serialize.ts`, `train/deserialize.ts` | -| G7 | **Back up what the browser holds.** Runs, datasets and the ledger live in IndexedDB with no export or import of the whole, and `navigator.storage.persist()` is never called, so the browser may evict the history the ledger relies on. | `projects/db.ts` | -| G8 | **Dates: ISO offsets and one order per column.** The ISO pattern refuses `Z` and `+02:00`, so such columns fall back to text; day-first or month-first is decided cell by cell, so one column can mix both in silence. | `timeseries/series.ts` | -| G9 | **Tooling: close what the compiler can close.** The worker dispatch is an `if/else` chain with no exhaustiveness check, and the stale-response fence a hand-kept set: adding a request touches six files of which one is checked. Pure-logic tests pay for jsdom. The e2e job rebuilds the app; no bundle budget; the accessibility score does not block and axe runs in English only. No `CLAUDE.md`. No global `unhandledrejection` handler. | `data/parse.worker.ts`, `store/fence.ts`, `vite.config.ts`, `ci.yml` | -| G10 | **A frozen cleaning recipe, and SQL over the cleaned table.** A recipe stores strategies, not fitted values: replayed on a new batch it cleans with that batch's own statistics, and cleaning happens before the lab splits, so test rows shape the fill values. To measure first: is the leak visible on scores? With it, SQL reading the file the way the app does. | `data/clean.ts`, `data/SqlPanel.tsx` | +reading of the repository after V68; G1 shipped as V69, G2 as V70. + +| Feature | Content | Reuses | +| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| G3 | **Say how many cells were filled.** A missing or unreadable number becomes the median, a missing category the most frequent one, with no count anywhere. On titanic `deck` is 77 % empty, under the `nearEmpty` cut, and three quarters of it are filled silently. A count per column in the run, the report and the model card; unreadable cells apart from empty ones. | `train/pipeline.ts`, `projects/model-card.ts` | +| G4 | **Rank on the quality of probabilities.** The leaderboard ranks on accuracy, F1, recall, precision, AUC, RMSE, MAE, R². Log loss is computed and shown and cannot rank; nor can Brier. The threshold, conformal and what-if panels all consume probabilities. It is the question V67 left open. Then: the calibration curve for every family, with an ECE. | `train/ranking.ts`, `train/threshold-analysis.ts` | +| G5 | **An honest forecast band.** The band is the 10th and 90th percentiles of one-step errors, applied flat to a horizon of up to 28 steps, on the hold-out that chose the method (the trap V57 measured). Errors per horizon step, a measured coverage, and a download of the forecast. | `timeseries/backtest.ts`, `timeseries/run.ts` | +| G6 | **Decisions travel with the export.** The export keeps the multiclass decision rule, not the binary threshold chosen nor the conformal quantile: the same model decides differently live and re-imported. | `train/serialize.ts`, `train/deserialize.ts` | +| G7 | **Back up what the browser holds.** Runs, datasets and the ledger live in IndexedDB with no export or import of the whole, and `navigator.storage.persist()` is never called, so the browser may evict the history the ledger relies on. | `projects/db.ts` | +| G8 | **Dates: ISO offsets and one order per column.** The ISO pattern refuses `Z` and `+02:00`, so such columns fall back to text; day-first or month-first is decided cell by cell, so one column can mix both in silence. | `timeseries/series.ts` | +| G9 | **Tooling: close what the compiler can close.** The worker dispatch is an `if/else` chain with no exhaustiveness check, and the stale-response fence a hand-kept set: adding a request touches six files of which one is checked. Pure-logic tests pay for jsdom. The e2e job rebuilds the app; no bundle budget; the accessibility score does not block and axe runs in English only. No `CLAUDE.md`. No global `unhandledrejection` handler. | `data/parse.worker.ts`, `store/fence.ts`, `vite.config.ts`, `ci.yml` | +| G10 | **A frozen cleaning recipe, and SQL over the cleaned table.** A recipe stores strategies, not fitted values: replayed on a new batch it cleans with that batch's own statistics, and cleaning happens before the lab splits, so test rows shape the fill values. To measure first: is the leak visible on scores? With it, SQL reading the file the way the app does. | `data/clean.ts`, `data/SqlPanel.tsx` | Not proposed, because the project already refused them: an AutoML mode, a tenth model family, model-based imputation, tabular deep learning, telemetry. diff --git a/README.md b/README.md index 160219f..9c4a252 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,7 @@ The project follows three non-negotiable principles: | Competition path | **An unlabelled file becomes a submission.** Under a scored batch, one block writes an identifier and a prediction under the names the platform expects: a class, a probability or a value, every row kept in its order, missing and repeated identifiers counted. The lab guesses neither the identifier nor the names. The same family can be **fitted again on every labelled row**, announced as a model nobody measured: on the demos, against rows kept away from both models, it gained 5 accuracy points on the smallest text file, 0.8 on titanic, 2 to 3 % of RMSE on two regressions, nothing on two others, and lost one time in five to one in three even where it paid | | Glass box | **One row, replayed through the model.** Beside the Shapley explanation (what weighed), a replay tells what happened: the questions a decision tree asked, in the units of the file, and the training rows that share its leaf; the count of a forest's 40 trees; the five rows of the file a k-NN compared it to, with its ties flagged; the prediction a boosting builds round after round, and its error on the validation rows. The replay walks the model itself: measured on 2,560 test rows of the demos, its last step is the model's answer on every one. The replay showed the boosting's validation error turning well before its last round; early stopping was then measured on rows kept away from everything and **not built**: fewer rounds improved the probabilities, not the predictions | | Counterfactuals | **What would have to change.** For one row of a classification, the smallest change, one column or two, that makes the inspected model answer another class: only values the training rows hold, the columns the user locks left alone, free text never edited, and a plain « nothing found » with what was tried. Each change carries two readings the measure made necessary: whether the changed row still looks like a training row (8 to 60 % of the nearest changes did not), and how many of the other trained models follow it (14 to 54 % did). A counterfactual speaks of the model, not of the world | +| Twin columns | **Two columns that say the same thing are flagged, not removed.** As soon as a file is read, a column that maps one to one onto another (`class` and `pclass`), or can be read off another (`adult_male` off `who`), says so on its card. Twins split the importance between them and let a counterfactual change one without the other: on titanic 43 % of the first counterfactuals did. Removing them moves no score, so the lab leaves the choice to the user | | Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met | | Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) | | Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner | @@ -289,7 +290,7 @@ Wikimedia Commons), the portrait is NASA, public domain._ a doc page carries its whole article, in English until the app mounts); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 1330 unit tests and 134 Playwright end-to-end tests across five +- **Quality bar.** 1344 unit tests and 135 Playwright end-to-end tests across five projects — desktop, a phone viewport in English and in French, dark mode, and Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on @@ -409,9 +410,9 @@ requires of the next file what was found on a reference (V63), and a scikit-lear that redoes a run outside the browser and measures its own gap (V64), a submission file for a competition (V65), a step-by-step replay of one row through the model (V66), early stopping for the boosting, measured and not built (V67), counterfactuals (V68), and a SQL -result that reaches the lab whole instead of its first 200 rows (V69). Proposed, none -scheduled: nine items from a new reading of the repository, listed in PLAN.md (duplicate -columns, a count of filled cells, ranking on probabilities, an honest forecast band, and +result that reaches the lab whole instead of its first 200 rows (V69), and twin columns +flagged before training (V70). Proposed, none scheduled: eight items from a new reading of +the repository, listed in PLAN.md (a count of filled cells, ranking on probabilities, an honest forecast band, and others). The consented generative chat (V12) stays postponed by product decision. No wave starts without an explicit launch command. The full plan, delivery log and design decisions live in [PLAN.md](PLAN.md); [CHANGELOG.md](CHANGELOG.md) is extracted from it — one entry per diff --git a/e2e/redundancy.spec.ts b/e2e/redundancy.spec.ts new file mode 100644 index 0000000..9cb62c4 --- /dev/null +++ b/e2e/redundancy.spec.ts @@ -0,0 +1,31 @@ +import { expect, test } from '@playwright/test'; + +test.use({ locale: 'en-US' }); + +// NOTE: keep this title ASCII-only: it names the test-results directory. +test('titanic: twin columns are flagged on their cards, and excluded by the user only', async ({ + page, +}) => { + await page.goto('/ml?demo=titanic&target=survived'); + // class/pclass, embarked/embark_town, and adult_male read off who. The twin + // of the target, alive, is a leak and is flagged as one, not as a twin. + await expect(page.getByTestId('redundant-summary')).toContainText( + '5 columns say the same thing as another', + { timeout: 30000 }, + ); + await expect(page.getByTestId('redundant-pclass')).toContainText( + 'Says the same thing as "class".', + ); + await expect(page.getByTestId('redundant-embark_town')).toContainText( + 'Says the same thing as "embarked".', + ); + await expect(page.getByTestId('redundant-adult_male')).toContainText('Can be read off "who".'); + await expect(page.getByTestId('redundant-who')).toHaveCount(0); + await expect(page.getByTestId('redundant-alive')).toHaveCount(0); + + // Flagged, not removed: both are still in until the user excludes one. + const card = page.getByTestId('column-card-class'); + await card.getByRole('button', { name: 'Exclude' }).click(); + await expect(page.getByTestId('redundant-pclass')).toHaveCount(0); + await expect(page.getByTestId('redundant-summary')).toContainText('3 columns'); +}); diff --git a/package-lock.json b/package-lock.json index c03e6f8..9975dde 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "labml", - "version": "1.69.0", + "version": "1.70.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "labml", - "version": "1.69.0", + "version": "1.70.0", "license": "MIT", "dependencies": { "@duckdb/duckdb-wasm": "1.28.0", diff --git a/package.json b/package.json index 3b1b9e4..d850f31 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "labml", "private": true, - "version": "1.69.0", + "version": "1.70.0", "license": "MIT", "type": "module", "engines": { diff --git a/src/content/docs/en/find-a-counterfactual.md b/src/content/docs/en/find-a-counterfactual.md index 6c41c20..b8e04a1 100644 --- a/src/content/docs/en/find-a-counterfactual.md +++ b/src/content/docs/en/find-a-counterfactual.md @@ -65,10 +65,14 @@ the cloud of training rows, and flags the ones that do not. On iris about half the answers stay flagged whatever is tried: its four columns move together, and changing one alone leaves the cloud. -The lab also does not know that two columns say the same thing. On titanic, -`sex`, `who` and `adult_male` describe one fact; changing `adult_male` alone -describes a passenger who cannot exist. Lock the columns that duplicate another, -or better, leave them out of the features. +Two columns that say the same thing are the other way to build a row that +cannot exist: changing `pclass` while `class` stays put. Measured on titanic +with every column in, **43 % of the first answers changed one column of such a +pair alone**. Since V70 the lab flags these twins on their cards before +training, and an answer that touches one without the other says so. Exclude one +of the two and train again: on the same rows, none of the answers did it any +more. What the flag does not see is a fact spread over three columns: `sex`, +`age` and `who` together. ## Reading two: do the other models follow? diff --git a/src/content/docs/en/limits.md b/src/content/docs/en/limits.md index ec0e00b..fd61ff1 100644 --- a/src/content/docs/en/limits.md +++ b/src/content/docs/en/limits.md @@ -68,7 +68,7 @@ They are not equivalent, and mixing them would be dishonest: | Replaying a re-imported model, or drawing a whole tree | The replay needs the training rows the run holds, which an export does not carry. It shows the path of one row, not the graph of the tree | | Advice from a counterfactual | It says what would make the model answer otherwise, not what would change the outcome in the world: the model learned which values go together, not what causes what | | A counterfactual of more than two columns, or one that edits free text | A change of five columns is no longer an explanation, and a sentence has no nearest neighbour to try. When the answer sits in the text, the search finds nothing and says so | -| Knowing that two columns say the same thing | Changing one of two redundant columns alone can describe a row that cannot exist. The search flags a changed row far from every training row, which does not catch this case: lock the duplicate, or leave it out of the features | +| Removing a twin column, or seeing a fact spread over three | Two columns that map one to one, or one read off another, are flagged on their cards (V70) and left in: which one to keep is your call, and removing them moves no score. A fact carried by three columns together (`who` from `sex` and `age`) is not seen, nor is a redundancy that is only approximate | | Counterfactuals for a regression, a re-imported model, or with the decision threshold | « Another class » has no meaning for a value; an export carries no training rows to draw observed values from; and the change found flips the prediction the what-if panel shows, not the decision of a chosen threshold | | SQL reading a file the way the rest of the app does | The console hands the file to DuckDB's own reader: a decimal comma, a windows-1252 file or an Excel sheet are read by the app and not by SQL, which also always sees the original file, before the recipe and before a join. What leaves the console is complete since V69; what enters it is still DuckDB's reading | | The recipe or the contract travelling with a SQL result | The ML Lab handoff carries the rows and a file name, nothing else | diff --git a/src/content/docs/en/reference-ml.md b/src/content/docs/en/reference-ml.md index 705b942..3b66107 100644 --- a/src/content/docs/en/reference-ml.md +++ b/src/content/docs/en/reference-ml.md @@ -31,6 +31,16 @@ triggers task detection (binary, multi-class, regression) and **leak detection**: a column that almost perfectly mirrors the target is excluded automatically and named. +**Twin columns** are flagged too, as soon as the file is read: two columns that +map one to one (`class` and `pclass`), or one that can be read off another +(`adult_male` off `who`). Unlike a leak, a twin is **not excluded**: its card +says so and leaves the choice to you. Twins split the importance and the +Shapley values between them, and let a counterfactual change one without the +other. Measured on titanic over ten splits, removing them moves no score (0.813 +against 0.814 of mean accuracy): it is about reading the model, not about its +performance. The test is exact, on values seen often enough to judge: it found +the three pairs of titanic and nothing on the nine other demo files. + While a run trains, and while anything computed from it is running (hyperparameter search, learning curve, robust ranking, batch scoring), the target and the columns are **locked**. Changing them at that moment would file diff --git a/src/content/docs/fr/chercher-un-contrefactuel.md b/src/content/docs/fr/chercher-un-contrefactuel.md index 89f0a58..cbac61d 100644 --- a/src/content/docs/fr/chercher-un-contrefactuel.md +++ b/src/content/docs/fr/chercher-un-contrefactuel.md @@ -71,10 +71,14 @@ pas. Sur iris, environ la moitié des réponses restent signalées quoi qu'on essaie : ses quatre colonnes bougent ensemble, et en changer une seule fait sortir du nuage. -Le labo ne sait pas non plus que deux colonnes disent la même chose. Sur titanic, -`sex`, `who` et `adult_male` décrivent un seul fait ; changer `adult_male` seul -décrit un passager qui ne peut pas exister. Verrouillez les colonnes qui en -doublent une autre, ou mieux, retirez-les des variables. +Deux colonnes qui disent la même chose sont l'autre façon de construire une +ligne qui ne peut pas exister : changer `pclass` alors que `class` ne bouge pas. +Mesuré sur titanic avec toutes les colonnes, **43 % des premières réponses +changeaient une seule colonne d'une telle paire**. Depuis V70 le labo signale +ces jumelles sur leurs cartes avant l'entraînement, et une réponse qui touche +l'une sans l'autre le dit. Excluez l'une des deux et réentraînez : sur les mêmes +lignes, plus aucune réponse ne le faisait. Ce que le signalement ne voit pas, +c'est un fait réparti sur trois colonnes : `sex`, `age` et `who` ensemble. ## Seconde lecture : les autres modèles suivent-ils ? diff --git a/src/content/docs/fr/limites.md b/src/content/docs/fr/limites.md index 6303d8d..c7ed8e5 100644 --- a/src/content/docs/fr/limites.md +++ b/src/content/docs/fr/limites.md @@ -68,7 +68,7 @@ Elles ne se valent pas, et les mélanger serait malhonnête : | Rejouer un modèle réimporté, ou dessiner un arbre entier | Le rejeu a besoin des lignes d'entraînement que le run détient, et qu'un export ne porte pas. Il montre le chemin d'une ligne, pas le graphe de l'arbre | | Un conseil tiré d'un contrefactuel | Il dit ce qui ferait répondre autrement le modèle, pas ce qui changerait le résultat dans le monde : le modèle a appris quelles valeurs vont ensemble, pas ce qui cause quoi | | Un contrefactuel de plus de deux colonnes, ou qui modifie du texte libre | Un changement de cinq colonnes n'est plus une explication, et une phrase n'a pas de plus proche voisine à essayer. Quand la réponse est dans le texte, la recherche ne trouve rien et le dit | -| Savoir que deux colonnes disent la même chose | Changer une seule de deux colonnes redondantes peut décrire une ligne qui ne peut pas exister. La recherche signale une ligne modifiée loin de toute ligne d'entraînement, ce qui n'attrape pas ce cas : verrouillez le doublon, ou retirez-le des variables | +| Retirer une colonne jumelle, ou voir un fait réparti sur trois | Deux colonnes en correspondance un pour un, ou l'une déduite de l'autre, sont signalées sur leurs cartes (V70) et laissées en place : laquelle garder est votre choix, et les retirer ne déplace aucun score. Un fait porté par trois colonnes ensemble (`who` à partir de `sex` et `age`) n'est pas vu, pas plus qu'une redondance seulement approximative | | Des contrefactuels pour une régression, un modèle réimporté, ou avec le seuil de décision | « Une autre classe » n'a pas de sens pour une valeur ; un export ne porte pas de lignes d'entraînement où puiser des valeurs observées ; et le changement trouvé fait basculer la prédiction que montre le panneau « Et si… », pas la décision d'un seuil choisi | | Le SQL qui lit un fichier comme le reste de l'application | La console confie le fichier au lecteur de DuckDB : une virgule décimale, un fichier windows-1252 ou une feuille Excel sont lus par l'application et pas par le SQL, qui voit en plus toujours le fichier d'origine, avant la recette et avant une jointure. Ce qui sort de la console est complet depuis V69 ; ce qui y entre reste la lecture de DuckDB | | La recette ou le contrat qui voyagent avec un résultat SQL | L'envoi au ML Lab porte les lignes et un nom de fichier, rien d'autre | diff --git a/src/content/docs/fr/reference-ml.md b/src/content/docs/fr/reference-ml.md index 2cc86db..2d6900f 100644 --- a/src/content/docs/fr/reference-ml.md +++ b/src/content/docs/fr/reference-ml.md @@ -33,6 +33,17 @@ et ses extrêmes. Choisir une cible déclenche la détection de tâche une colonne qui reflète presque parfaitement la cible est exclue automatiquement et nommée. +Les **colonnes jumelles** sont signalées aussi, dès la lecture du fichier : deux +colonnes en correspondance un pour un (`class` et `pclass`), ou une colonne qui +se déduit d'une autre (`adult_male` de `who`). Contrairement à une fuite, une +jumelle n'est **pas exclue** : sa carte le dit et vous laisse le choix. Des +jumelles se partagent l'importance et les valeurs de Shapley, et laissent un +contrefactuel changer l'une sans l'autre. Mesuré sur titanic sur dix découpes, +les retirer ne déplace aucun score (0,813 contre 0,814 d'exactitude moyenne) : +c'est une affaire de lecture du modèle, pas de performance. Le test est exact, +sur des valeurs vues assez souvent pour en juger : il a trouvé les trois paires +de titanic et rien sur les neuf autres fichiers de démonstration. + Pendant un entraînement, et pendant tout calcul qui en dépend (recherche d'hyperparamètres, courbe d'apprentissage, classement robuste, scoring d'un lot), la cible et les colonnes sont **verrouillées**. Les changer à ce diff --git a/src/features/ml/components/ColumnCard.test.tsx b/src/features/ml/components/ColumnCard.test.tsx new file mode 100644 index 0000000..89afc4c --- /dev/null +++ b/src/features/ml/components/ColumnCard.test.tsx @@ -0,0 +1,76 @@ +import { render, screen } from '@testing-library/react'; +import { beforeEach, describe, expect, it } from 'vitest'; +import { ColumnCard } from '@/features/ml/components/ColumnCard'; +import { activeTwins, useLabStore } from '@/features/ml/lab-store'; +import type { ColumnProfile, ColumnRedundancy } from '@/features/ml/data/types'; +import i18n from '@/lib/i18n'; + +// V70: a twin is flagged on its card, and excluded by nobody but the user. + +const profile = (name: string): ColumnProfile => + ({ name, type: 'categorical', rowCount: 891, missingCount: 0, cardinality: 3 }) as ColumnProfile; + +const redundancies: ColumnRedundancy[] = [ + { column: 'pclass', of: 'class', relation: 'same' }, + { column: 'class', of: 'pclass', relation: 'same' }, + { column: 'adult_male', of: 'who', relation: 'determined' }, + { column: 'survived', of: 'alive', relation: 'same' }, + { column: 'alive', of: 'survived', relation: 'same' }, +]; + +describe('ColumnCard twins', () => { + beforeEach(async () => { + await i18n.changeLanguage('en'); + useLabStore.setState({ + target: 'survived', + baseline: [], + leaks: [{ column: 'alive', reason: 'leak' }], + overrides: {}, + redundancies, + trainStatus: 'idle', + }); + }); + + it('says a column says the same thing as another, and leaves it in', () => { + render(); + expect(screen.getByTestId('redundant-pclass')).toHaveTextContent( + 'Says the same thing as "class". Keeping both splits their weight between them; excluding one loses nothing.', + ); + // Flagged, not excluded: the button still offers to exclude it. + expect(screen.getByRole('button', { name: 'Exclude' })).toBeInTheDocument(); + expect(screen.getByTestId('column-card-pclass')).not.toHaveClass('opacity-70'); + }); + + it('says which way it goes when only one column can be read off the other', () => { + const { unmount } = render(); + expect(screen.getByTestId('redundant-adult_male')).toHaveTextContent('Can be read off "who".'); + unmount(); + render(); + expect(screen.queryByTestId('redundant-who')).toBeNull(); + }); + + it('drops the flag once one of the two is excluded', () => { + useLabStore.setState({ overrides: { class: 'exclude' } }); + const { unmount } = render(); + expect(screen.queryByTestId('redundant-pclass')).toBeNull(); + unmount(); + render(); + expect(screen.queryByTestId('redundant-class')).toBeNull(); + }); + + it('leaves the twin of the target to the leak flag', () => { + render(); + expect(screen.queryByTestId('redundant-alive')).toBeNull(); + const state = useLabStore.getState(); + expect(activeTwins(state, 'survived')).toEqual([]); + expect(activeTwins(state, 'alive')).toEqual([]); + }); + + it('speaks French', async () => { + await i18n.changeLanguage('fr'); + render(); + expect(screen.getByTestId('redundant-pclass')).toHaveTextContent( + 'Dit la même chose que « class ».', + ); + }); +}); diff --git a/src/features/ml/components/ColumnCard.tsx b/src/features/ml/components/ColumnCard.tsx index 52ed1b9..091d73c 100644 --- a/src/features/ml/components/ColumnCard.tsx +++ b/src/features/ml/components/ColumnCard.tsx @@ -2,7 +2,12 @@ import { Crosshair } from 'lucide-react'; import { useTranslation } from 'react-i18next'; import { Badge } from '@/components/ui/badge'; import { MiniDistribution } from '@/features/ml/components/MiniDistribution'; -import { effectiveExclusion, isTrainingBusy, useLabStore } from '@/features/ml/lab-store'; +import { + activeTwins, + effectiveExclusion, + isTrainingBusy, + useLabStore, +} from '@/features/ml/lab-store'; import type { ColumnProfile } from '@/features/ml/data/types'; import { cn } from '@/lib/utils'; @@ -16,6 +21,7 @@ export function ColumnCard({ profile }: { profile: ColumnProfile }) { const baseline = useLabStore((s) => s.baseline); const leaks = useLabStore((s) => s.leaks); const overrides = useLabStore((s) => s.overrides); + const redundancies = useLabStore((s) => s.redundancies); const setTarget = useLabStore((s) => s.setTarget); const toggleColumn = useLabStore((s) => s.toggleColumn); // V47: same rule as the target picker — no edit while a run answers for it. @@ -23,6 +29,7 @@ export function ColumnCard({ profile }: { profile: ColumnProfile }) { const isTarget = target === profile.name; const exclusion = effectiveExclusion({ baseline, leaks, overrides, target }, profile.name); + const twins = activeTwins({ baseline, leaks, overrides, target, redundancies }, profile.name); const missingPercent = profile.rowCount ? Math.round((profile.missingCount / profile.rowCount) * 100) : 0; @@ -74,6 +81,15 @@ export function ColumnCard({ profile }: { profile: ColumnProfile }) { )}

+ {twins.length > 0 && ( +

+ {twins + .map((pair) => t(`ml.lab.redundant.${pair.relation}`, { column: pair.of })) + .join(' ')}{' '} + {t('ml.lab.redundant.keepOne')} +

+ )} +
{!isTarget && (