diff --git a/CHANGELOG.md b/CHANGELOG.md index 83f9baa..ef67a70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and the reason the wave was built. +## V69 — A SQL result reaches the lab whole + +G1, the first of the ten proposals a new reading of the repository produced after V68 (the others are listed below). + +_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. + ## V68 — Counterfactuals: what would have to change F7, the last feature proposed. diff --git a/PLAN.md b/PLAN.md index 522e18b..8e2bae1 100644 --- a/PLAN.md +++ b/PLAN.md @@ -680,7 +680,7 @@ defect that produces a wrong number comes first), then extends the lab past the export: uncertainty with a guarantee, governance, and monitoring after deployment, with nothing added on a server. -V53 to V68 are delivered. Every other row is a proposal with its launch +V53 to V69 are delivered. Every other row is a proposal with its launch condition, and none starts without an explicit launch command. | Wave | Content | Why | @@ -702,12 +702,25 @@ condition, and none starts without an explicit launch command. | **V66 — delivered** | **The glass box: one row, replayed through the model.** The third and last part of F8. For a row typed in the what-if panel the lab gave a prediction and a Shapley explanation (V28), what weighed. It did not show what happened. A replay now tells it for four families: the questions a decision tree asked and the training rows that share its leaf, the count of a forest's trees, the five rows a k-NN compared the row to, the prediction a boosting builds round after round. **What the code said before anything was written**: F8 spoke of « hand-written » models, and the tree and the forest come from ml-cart and ml-random-forest. A replay stays possible by walking their exported JSON, which is also what a model rebuilt from a helper core holds (V37). No family kept what a replay needs: the boosting only summed all its trees, the k-NN search dropped the index of its neighbours, the tree exposed no path. **One rule: the replay is the model.** Nothing is approximated, and the last step of a replay must equal `predict`. **Measured before the panel, six demos, five seeds, 2,560 test rows**: zero mismatch for the tree, the k-NN and the boosting, in binary, multiclass and regression. **The forest failed on three rows of titanic**: on a 20 to 20 vote the replay kept the smallest class, and the library keeps the first class to reach the top count, in the order of its trees. The replay does the same now, flags a tied vote, and the forest, which the plan kept only « if its count is exact », is in. **Questions in the units of the file.** A threshold on a standardized number is given its unit back, a 0/1 column becomes « is it this category? ». Measured on the tree's paths: every question translates on five demos, and 6 of 1,878 on reviews, whose columns are free text: there a question is the weight of a word, shown as the model sees it. A path is 1.9 questions long on fraud and 8 on energy on average, 10 at most. **Two findings the replay made visible.** (1) **k-NN ties**: the next row is exactly as far as the last neighbour kept on 17 % of titanic's test rows and 39 % of energy's, 0 to 1 % elsewhere. V64 had met this from the outside, as a gap with scikit-learn; the panel flags it on the row concerned. (2) **The boosting's last rounds**: read on the validation rows, the error was lowest well before round 120 (more than 2 % under the final error) on 17 runs of 30. On titanic the lowest log loss came at rounds 21 to 25 on three seeds (0.43 against 0.45 to 0.48 at the end); on reviews at round 11 on one seed (0.67 against 0.79, worse than the 0.69 it started from); on energy the boosting never beat its starting point on three seeds of five. The lab runs its rounds to the end and has no early stopping. The panel shows the curve, on validation and never on the test set, and sets nothing; the fix is proposed as F9 below, to be measured on its own. **Defects of this wave's own making, caught before the push**: (1) the forest's tie, above; (2) ml-random-forest nests its parameters under `baseModel`, which the first replay did not read; (3) the three rounds told for the boosting were always rounds 1, 2 and 3, the same tree three times, seen on the screenshot review: the rounds told are now the largest movers among those asking different questions, each with the count of other rounds that asked the same. **What this wave deliberately does not do**: replay how a model was trained; replay a network, a linear model or a blend (the panel says why for each), or a forest tree by tree; stop the boosting early or change any setting from what is seen; replay a re-imported model, which carries no training rows; draw a whole tree. A replay is shown and saved nowhere: it is absent from the saved run and from the share link. A trace is not a cause, and the panel says so. F8 is complete. 1294 unit tests, 133 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | | **V67 — delivered** | **Early stopping for the boosting: measured, and not built.** F9, opened by V66, which had shown the boosting's validation error lowest well before its 120th round on 17 runs of 30. The obvious reading is overfitting and the obvious fix is to stop at that round. This wave measured the fix before writing it, and the measure says no. **The trap seen before measuring**: the validation rows choose the family (V35, V53). A boosting that also chooses its number of rounds on them would be scored on rows it tuned itself on, against families that tuned nothing there. So three stops were compared, not one: (A) at the lowest error on the run's validation rows; (B) at the lowest error on 15 or 20 % of the training rows held out inside the boosting, keeping that smaller model; (C) the round found by B, applied to the model fitted on all the training rows. Plus a stop by patience (10 or 20 rounds without progress) against the global minimum. **Measured on six demos, ten splits each, a quarter of the rows kept away from everything, against the 120 fixed rounds, on the metric the leaderboard ranks on.** Titanic: A gains 0.4 accuracy point, better on 5 splits and worse on 4. Fraud, iris and mpg: no change. **Reviews: A loses 4.0 accuracy points** (worse 6, better 2), B and C lose 5 to 7. Energy: RMSE 3.5 % lower with A (better 9, worse 1), where the stop is often round 0, the mean of the training rows: there the boosting does no better than the baseline, and it is the family chosen on validation on none of the ten splits. B and C did no better than A on five demos of six, and B, which gives up rows, was the worst of the three on four demos. Patience and global minimum gave the same result within noise. **What does improve is the log loss**: better on 10 splits of 10 on titanic (0.02) and on reviews (0.10). Fewer rounds give better probabilities and no better predictions, and on reviews both at once: a lower log loss and 4 points less accuracy. V66's curve is a log loss in classification, which is why it turned early while the classes predicted kept improving or held. **The trap was real**: with A the boosting's validation score rose above its score on the private rows by 0.9 accuracy point on titanic, 6 points on reviews, 0.04 of RMSE on mpg and 0.18 on energy, beyond what it already did at round 120. On the choice of the family it changed little, and never for the better: the family chosen on validation scored 0.828 on the private rows with A against 0.832 without on titanic, an RMSE of 3.034 against 3.023 on mpg, the same elsewhere. **Decision, by the rule the plan had set (« ship only what the measure justifies »)**: nothing changes in training. The boosting keeps its 120 rounds, no figure of any run moves, and no golden reference is regenerated. **What ships**: the measure, written in `gbdt.ts` where the next reader will look for early stopping; the glass box's sentence, which said the later rounds « did not help » and let the reader conclude that fewer rounds would predict better, now says what was measured; the how-to page and the limits page in both languages carry the table. F9 is closed. **Defects of this wave's own making**: V66's own wording, above, was the defect: it read a log-loss curve as a verdict on the predictions. **What this wave deliberately does not do**: stop early when the user ranks on log loss, where the measure does favour fewer rounds on two demos of four: the lab ranks on accuracy or RMSE by default, and a stop tied to the ranking metric is another design, with the validation trap to solve first; tune depth or learning rate; touch the network or the forest. 1294 unit tests, 133 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | | **V68 — delivered** | **Counterfactuals: what would have to change.** F7, the last feature proposed. For a row typed in the what-if panel the lab said what the model predicts, what weighed (Shapley, V28) and what happened (the glass box, V66). It did not answer the question a person told « no » asks: what would have had to be different? The user looked for it by hand, one field at a time. **A search a reader can follow, not an optimisation**: one column at a time, every value the training rows hold tried in place of the row's own, the column keeping the nearest that flips the answer; two columns together only when no single one flips it; never three. Only observed values, no interpolation; free text is never edited; the columns the user locks are left alone, since the lab does not guess that an age cannot change. Nothing found is an answer, given with what was tried. The plan spoke of refining a numeric threshold by bisection; that would have produced values no row holds, against the rule just stated, and was dropped. **Measured on the four classification demos, three seeds, 60 test rows each, seven families.** One column is enough on 63 to 98 % of titanic's rows, 28 to 100 % of fraud's, 96 to 100 % of iris's; two are needed on up to 30 % (titanic, logistic) and 69 % (fraud, forest), so the pair search earns its place. On reviews most families find nothing (100 % of rows for the naive Bayes, 97 % for the tree): the answer is in the text. A search costs 30 ms at most (the forest), 100 to 550 predictions: no progress indicator is needed. **Two findings changed the design.** (1) **Plausibility**: 8 to 60 % of the nearest changes landed farther from every training row than 95 % of training rows are from their nearest neighbour; the model was answering where it had seen nothing. The search now keeps, per column, the nearest flipping value that stays inside that cloud, and flags the rest. It helps where an alternative exists (the tree on titanic from 8 to 4 %, the naive Bayes on fraud from 14 to 0 %) and not where none does: about half of iris's answers stay flagged, its four columns moving together. (2) **Fragility**: of the other trained models that agreed on the row, only 14 to 54 % answered the new class for the same change. A minimal change lands just across one model's boundary; each answer therefore says how many of the others follow. **What the flag does not catch**, seen on the screenshot review: on titanic the forest proposed `adult_male`: True to False while `sex` stayed `male`. Each value exists, the row does not, and a 0/1 column moving alone is not far from the cloud. The lab does not know two columns say the same thing; the panel and the limits page say so and tell the user to lock or drop the duplicate. **Defects of this wave's own making, caught before the push**: (1) the first search returned the nearest change whatever it was, above; (2) a category loaded into the form from a counterfactual could be rarer than the eight the form lists, and the field then showed another value than the one predicted on: the form now lists the loaded value; (3) the lock checkbox was first put inside the label of its field, so one label named two controls: two existing end-to-end tests that fill a field by its label failed, and the lock is now a control of its own beside the label; (4) the new page quoted a duration in milliseconds, which the documentation refuses since a test says so. **What this wave deliberately does not do**: give advice about the world, a counterfactual speaks of the model and the panel says so; search beyond two columns or edit free text; detect redundant columns; apply the chosen decision threshold (the change flips the prediction the what-if panel shows); handle a regression, where « another class » has no meaning, or a re-imported model, which carries no training rows; implement a published method (DiCE and others). A counterfactual is shown and saved nowhere. The list of proposed features is now empty. 1320 unit tests, 134 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | +| **V69 — delivered** | **A SQL result reaches the lab whole.** G1, the first of the ten proposals a new reading of the repository produced after V68 (the others are listed below). The how-to page presents « filter, join or aggregate in SQL, then send to the ML Lab » as the normal path. **What the code said**: the console paints 200 rows, which is sound, and the three ways out of it were built from those painted rows. `toLab` and the CSV export sent `tableToCsv(result)`, a table already cut to 200. A note in the interface said so; it still emptied the gesture: nobody could train on a SQL result of more than 200 rows, and the demo file, 118 rows long, hid it. The cut saved no memory either, the engine having loaded the whole result before cutting. **Measured before the fix**, in the browser with the real engine, on a query returning 590 rows with a NULL in a numeric column: the lab received **200 rows**, and read the column as **categorical**, the NULL having left as `∅`, the mark the console paints for it, which the lab took for a category. **The fix**: an export is the query, not the screen. `SqlEngine.toCsv` asks DuckDB for the file, `COPY (…) TO … (FORMAT CSV, HEADER)`, as `toParquet` already did; the CSV export, the lab handoff and a Parquet button (the method existed since V40 and no button called it, though the reference announced it) all go through the engine, on the query that produced the result on screen, not on the text of the box, which may have been edited since. **Measured after**: the 590 rows arrive, the column is numeric with missing values, and a plain `SELECT *` over each of the lab's six demos gives back the same rows, the same columns and the same inferred types as the file loaded directly. Writing 118,000 rows took 0.2 s and 4.9 MB, 1,180,000 rows 6.4 s and 49 MB, on the single-threaded engine. The lab's cell budget (V54, 20 million) is applied before anything is built, so a result the lab would refuse is refused in the console, by name. **A file name of its own.** Every handoff was `query.csv`; the test-set ledger (V60) knows a file by its name and shape, and took two queries of the same shape for one file. The name now carries a fingerprint of the query, never its text; the FNV-1a function the ledger had moved to `lib/fingerprint.ts` instead of being copied. **Defects of this wave's own making**: none found by the suites; the fake engine of the panel's test had been answering `total` where the type says `totalRows` since V47, unnoticed because no test read it. **What this wave deliberately does not do**: make SQL read a file the way the rest of the app does (decimal comma, windows-1252, Excel, the cleaned or joined table): what leaves the console is complete, what enters it is still DuckDB's own reading, and the limits page says so; carry the recipe or the contract with the handoff; raise the 200 painted rows or paginate the console; change the engine or its version. 1330 unit tests, 134 e2e. | Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way. | Proposed features, ranked by value over effort. Each reuses a module that exists; none is scheduled. -None at the moment: the features proposed here were delivered, or measured and closed, -through V68. A new proposal goes here with the module it reuses and its launch condition. +F1 to F9 were delivered, or measured and closed, through V68. G1 to G10 come from a new +reading of the repository after V68; G1 shipped as V69. + +| Feature | Content | Reuses | +| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| G2 | **Flag columns that say the same thing.** `ExclusionReason` knows `id`, `constant`, `nearEmpty`, `leak`, `unsupported`; the functional-dependency check runs from a column to the target only. Titanic keeps `class`/`pclass`, `embarked`/`embark_town`, `sex`/`who`/`adult_male` as separate features, which splits importance between twins and let V68 propose a passenger who cannot exist. To measure first: true and false duplicates on the demos. The lab flags, it does not remove. | `data/suggest.ts`, `data/types.ts` | +| G3 | **Say how many cells were filled.** A missing or unreadable number becomes the median, a missing category the most frequent one, with no count anywhere. On titanic `deck` is 77 % empty, under the `nearEmpty` cut, and three quarters of it are filled silently. A count per column in the run, the report and the model card; unreadable cells apart from empty ones. | `train/pipeline.ts`, `projects/model-card.ts` | +| G4 | **Rank on the quality of probabilities.** The leaderboard ranks on accuracy, F1, recall, precision, AUC, RMSE, MAE, R². Log loss is computed and shown and cannot rank; nor can Brier. The threshold, conformal and what-if panels all consume probabilities. It is the question V67 left open. Then: the calibration curve for every family, with an ECE. | `train/ranking.ts`, `train/threshold-analysis.ts` | +| G5 | **An honest forecast band.** The band is the 10th and 90th percentiles of one-step errors, applied flat to a horizon of up to 28 steps, on the hold-out that chose the method (the trap V57 measured). Errors per horizon step, a measured coverage, and a download of the forecast. | `timeseries/backtest.ts`, `timeseries/run.ts` | +| G6 | **Decisions travel with the export.** The export keeps the multiclass decision rule, not the binary threshold chosen nor the conformal quantile: the same model decides differently live and re-imported. | `train/serialize.ts`, `train/deserialize.ts` | +| G7 | **Back up what the browser holds.** Runs, datasets and the ledger live in IndexedDB with no export or import of the whole, and `navigator.storage.persist()` is never called, so the browser may evict the history the ledger relies on. | `projects/db.ts` | +| G8 | **Dates: ISO offsets and one order per column.** The ISO pattern refuses `Z` and `+02:00`, so such columns fall back to text; day-first or month-first is decided cell by cell, so one column can mix both in silence. | `timeseries/series.ts` | +| G9 | **Tooling: close what the compiler can close.** The worker dispatch is an `if/else` chain with no exhaustiveness check, and the stale-response fence a hand-kept set: adding a request touches six files of which one is checked. Pure-logic tests pay for jsdom. The e2e job rebuilds the app; no bundle budget; the accessibility score does not block and axe runs in English only. No `CLAUDE.md`. No global `unhandledrejection` handler. | `data/parse.worker.ts`, `store/fence.ts`, `vite.config.ts`, `ci.yml` | +| G10 | **A frozen cleaning recipe, and SQL over the cleaned table.** A recipe stores strategies, not fitted values: replayed on a new batch it cleans with that batch's own statistics, and cleaning happens before the lab splits, so test rows shape the fill values. To measure first: is the leak visible on scores? With it, SQL reading the file the way the app does. | `data/clean.ts`, `data/SqlPanel.tsx` | Not proposed, because the project already refused them: an AutoML mode, a tenth model family, model-based imputation, tabular deep learning, telemetry. diff --git a/README.md b/README.md index 6cb347a..160219f 100644 --- a/README.md +++ b/README.md @@ -213,7 +213,8 @@ _The index is built at compile time and queried in the tab — typing here sends no correct answer is lost. - **Analytical SQL** (`/data`): a real OLAP engine — DuckDB-Wasm, MIT, self-hosted and single-threaded — queries the loaded file in the browser, plus any CSV, **Parquet** or - JSON attached in the session. Results export to CSV or move to the ML Lab in one click; + JSON attached in the session. A result exports to CSV or Parquet, or moves to the ML Lab, + in one click, written by the engine itself: every row of it, not the 200 the console paints; SQL errors show DuckDB's own message. The engine is pinned to 1.28.0 for a measured reason: from 1.29 its binaries exceed Cloudflare's 25 MiB per-file limit. - **Data assistant** (`/ai/chat`): plain French or English questions about a loaded @@ -288,7 +289,7 @@ Wikimedia Commons), the portrait is NASA, public domain._ a doc page carries its whole article, in English until the app mounts); Lighthouse mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX Runtime) load lazily. -- **Quality bar.** 1320 unit tests and 134 Playwright end-to-end tests across five +- **Quality bar.** 1330 unit tests and 134 Playwright end-to-end tests across five projects — desktop, a phone viewport in English and in French, dark mode, and Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on @@ -407,8 +408,11 @@ report, which had been calling small files drifted (V62), a data contract that requires of the next file what was found on a reference (V63), and a scikit-learn script that redoes a run outside the browser and measures its own gap (V64), a submission file for a competition (V65), a step-by-step replay of one row through the model (V66), early -stopping for the boosting, measured and not built (V67), and counterfactuals (V68). No -feature is proposed at the moment. The consented generative chat (V12) stays +stopping for the boosting, measured and not built (V67), counterfactuals (V68), and a SQL +result that reaches the lab whole instead of its first 200 rows (V69). Proposed, none +scheduled: nine items from a new reading of the repository, listed in PLAN.md (duplicate +columns, a count of filled cells, ranking on probabilities, an honest forecast band, and +others). The consented generative chat (V12) stays postponed by product decision. No wave starts without an explicit launch command. The full plan, delivery log and design decisions live in [PLAN.md](PLAN.md); [CHANGELOG.md](CHANGELOG.md) is extracted from it — one entry per wave, newest first — by `npm run changelog`, and a test fails when the two disagree. diff --git a/e2e/sql.spec.ts b/e2e/sql.spec.ts index 9c294e8..eba28dd 100644 --- a/e2e/sql.spec.ts +++ b/e2e/sql.spec.ts @@ -1,3 +1,4 @@ +import { readFileSync } from 'node:fs'; import { expect, test } from '@playwright/test'; test.use({ locale: 'en-US' }); @@ -6,7 +7,9 @@ test.use({ locale: 'en-US' }); // twice in parallel is the slowest thing this suite could do for no gain. test.describe.configure({ timeout: 180_000 }); -test('the SQL console answers a real query and reports the engine own errors', async ({ page }) => { +test('the SQL console answers a real query and reports the engine own errors', async ({ + page, +}, testInfo) => { await page.goto('/data'); await page.getByRole('button', { name: /cafe-sales\.csv/ }).click(); await expect(page.getByTestId('sql-panel')).toBeVisible(); @@ -37,4 +40,44 @@ test('the SQL console answers a real query and reports the engine own errors', a const error = page.getByTestId('sql-error'); await expect(error).toBeVisible({ timeout: 60_000 }); await expect(error).toContainText(/nope/i); + + // V69: an export is the query, not the screen. 590 rows come back, 200 are + // painted, and every one of them leaves the console. + await page + .getByTestId('sql-input') + .fill( + 'SELECT a.*, CASE WHEN r.range = 0 THEN NULL ELSE a.quantity END AS q2 FROM dataset a, range(5) r;', + ); + await page.getByTestId('sql-run').click(); + await expect(result).toContainText('First 200 rows shown out of 590', { timeout: 60_000 }); + await expect(result.locator('tbody tr')).toHaveCount(200); + await expect(page.getByTestId('sql-export-note')).toContainText('they carry all 590 rows'); + + const csvDownload = page.waitForEvent('download'); + await page.getByTestId('sql-export-csv').click(); + const csvFile = await csvDownload; + expect(csvFile.suggestedFilename()).toMatch(/^query-[0-9a-f]{8}\.csv$/); + const csvPath = testInfo.outputPath('query.csv'); + await csvFile.saveAs(csvPath); + const lines = readFileSync(csvPath, 'utf8').trim().split('\n'); + expect(lines).toHaveLength(591); + expect(lines[0]).toBe('order_id,date,product,city,quantity,unit_price,currency,notes,q2'); + // A NULL is an empty cell, not the mark the console paints for it. + expect(lines.join('\n')).not.toContain('∅'); + expect(lines.filter((line) => line.endsWith(','))).toHaveLength(118); + + const parquetDownload = page.waitForEvent('download'); + await page.getByTestId('sql-export-parquet').click(); + const parquetFile = await parquetDownload; + expect(parquetFile.suggestedFilename()).toMatch(/^query-[0-9a-f]{8}\.parquet$/); + const parquetPath = testInfo.outputPath('query.parquet'); + await parquetFile.saveAs(parquetPath); + expect(readFileSync(parquetPath).subarray(0, 4).toString()).toBe('PAR1'); + + // The lab receives the whole result, and reads the column with NULLs as a + // number with missing values: it used to get 200 rows and a text column. + await page.getByTestId('sql-to-lab').click(); + await expect(page).toHaveURL(/\/ml/); + await expect(page.getByText('590 rows · 9 columns')).toBeVisible({ timeout: 30_000 }); + await expect(page.getByTestId('column-card-q2')).toContainText(/numeric/i); }); diff --git a/package-lock.json b/package-lock.json index 3958488..c03e6f8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "labml", - "version": "1.68.0", + "version": "1.69.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "labml", - "version": "1.68.0", + "version": "1.69.0", "license": "MIT", "dependencies": { "@duckdb/duckdb-wasm": "1.28.0", diff --git a/package.json b/package.json index a83dd59..3b1b9e4 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "labml", "private": true, - "version": "1.68.0", + "version": "1.69.0", "license": "MIT", "type": "module", "engines": { diff --git a/src/content/docs/en/limits.md b/src/content/docs/en/limits.md index 907b457..ec0e00b 100644 --- a/src/content/docs/en/limits.md +++ b/src/content/docs/en/limits.md @@ -70,6 +70,8 @@ They are not equivalent, and mixing them would be dishonest: | A counterfactual of more than two columns, or one that edits free text | A change of five columns is no longer an explanation, and a sentence has no nearest neighbour to try. When the answer sits in the text, the search finds nothing and says so | | Knowing that two columns say the same thing | Changing one of two redundant columns alone can describe a row that cannot exist. The search flags a changed row far from every training row, which does not catch this case: lock the duplicate, or leave it out of the features | | Counterfactuals for a regression, a re-imported model, or with the decision threshold | « Another class » has no meaning for a value; an export carries no training rows to draw observed values from; and the change found flips the prediction the what-if panel shows, not the decision of a chosen threshold | +| SQL reading a file the way the rest of the app does | The console hands the file to DuckDB's own reader: a decimal comma, a windows-1252 file or an Excel sheet are read by the app and not by SQL, which also always sees the original file, before the recipe and before a join. What leaves the console is complete since V69; what enters it is still DuckDB's reading | +| The recipe or the contract travelling with a SQL result | The ML Lab handoff carries the rows and a file name, nothing else | | A generative chat backed by a remote API (V12) | Postponed by product decision (confirmed 09/10/2026). Today's chat runs entirely in the browser; a remote API would send out the question and the column schema, never the data, and would require a consent screen | ## Dropped after measurement diff --git a/src/content/docs/en/reference-data.md b/src/content/docs/en/reference-data.md index cfbce48..d131105 100644 --- a/src/content/docs/en/reference-data.md +++ b/src/content/docs/en/reference-data.md @@ -155,8 +155,10 @@ DuckDB-Wasm, MIT, self-hosted, single-threaded (no COOP/COEP headers). Pinned to 25 MiB per-file limit. Queries the active dataset plus any CSV, Parquet or JSON attached to the -session. Results export to CSV or move to the ML Lab in one click. **Parquet** -export through `COPY … TO`. +session. A result exports to CSV or **Parquet**, or moves to the ML Lab, in one +click. All three are written by the engine through `COPY … TO` and carry every +row of the result, not the 200 the console paints; a NULL leaves as an empty +cell. See [hand a SQL result to the ML Lab](/docs/sql-vers-le-lab). ## Where to go next diff --git a/src/content/docs/en/sql-to-the-lab.md b/src/content/docs/en/sql-to-the-lab.md index b6c583a..5606f9c 100644 --- a/src/content/docs/en/sql-to-the-lab.md +++ b/src/content/docs/en/sql-to-the-lab.md @@ -26,6 +26,30 @@ way. - **Aggregate**: go from one row per event to one row per customer, which is what most real ML problems need. +## What leaves the console + +The console paints the first 200 rows so it stays responsive, and always states +how many rows the query returned. **The exports are the query, not the screen**: +the CSV, the Parquet file and the ML Lab handoff are written by the engine +itself, and carry every row. + +- **A NULL stays a missing value.** It leaves as an empty cell, which the lab + reads as missing. A numeric column with NULLs arrives as a number with gaps. +- **Each query has a file name of its own**, such as `query-3f2a91c4.csv`: a + fingerprint of the query, never its text. The lab's test-set ledger knows a + file by its name and shape, and two queries of the same shape must not share + one count. +- **The lab's cell budget applies.** A result past 20 million cells (rows times + columns) is refused before it is built, with the advice to aggregate or + filter. It is the same budget a file is read under. + +Until V69 the exports were built from the painted rows: a query returning 590 +rows handed 200 to the lab, and a NULL arrived as the mark the console shows +for it, which turned a numeric column into text. Measured after the fix: the +590 rows arrive, and a plain `SELECT *` over each of the lab's six demos gives +back the same rows, the same columns and the same inferred types as the file +itself. + ## The cost, announced The SQL engine is an 18–22 MB download, **optional**. Until you open it, it is diff --git a/src/content/docs/fr/limites.md b/src/content/docs/fr/limites.md index 1546dd8..6303d8d 100644 --- a/src/content/docs/fr/limites.md +++ b/src/content/docs/fr/limites.md @@ -70,6 +70,8 @@ Elles ne se valent pas, et les mélanger serait malhonnête : | Un contrefactuel de plus de deux colonnes, ou qui modifie du texte libre | Un changement de cinq colonnes n'est plus une explication, et une phrase n'a pas de plus proche voisine à essayer. Quand la réponse est dans le texte, la recherche ne trouve rien et le dit | | Savoir que deux colonnes disent la même chose | Changer une seule de deux colonnes redondantes peut décrire une ligne qui ne peut pas exister. La recherche signale une ligne modifiée loin de toute ligne d'entraînement, ce qui n'attrape pas ce cas : verrouillez le doublon, ou retirez-le des variables | | Des contrefactuels pour une régression, un modèle réimporté, ou avec le seuil de décision | « Une autre classe » n'a pas de sens pour une valeur ; un export ne porte pas de lignes d'entraînement où puiser des valeurs observées ; et le changement trouvé fait basculer la prédiction que montre le panneau « Et si… », pas la décision d'un seuil choisi | +| Le SQL qui lit un fichier comme le reste de l'application | La console confie le fichier au lecteur de DuckDB : une virgule décimale, un fichier windows-1252 ou une feuille Excel sont lus par l'application et pas par le SQL, qui voit en plus toujours le fichier d'origine, avant la recette et avant une jointure. Ce qui sort de la console est complet depuis V69 ; ce qui y entre reste la lecture de DuckDB | +| La recette ou le contrat qui voyagent avec un résultat SQL | L'envoi au ML Lab porte les lignes et un nom de fichier, rien d'autre | | Un chat génératif branché sur une API distante (V12) | Reporté par décision produit (confirmé le 09/10/2026). Le chat actuel tourne entièrement dans le navigateur ; une API distante ferait sortir la question et le schéma des colonnes, jamais les données, et exigerait un écran de consentement | ## Ce qui a été abandonné après mesure diff --git a/src/content/docs/fr/reference-data.md b/src/content/docs/fr/reference-data.md index 8ad3140..5402793 100644 --- a/src/content/docs/fr/reference-data.md +++ b/src/content/docs/fr/reference-data.md @@ -167,8 +167,10 @@ DuckDB-Wasm, MIT, auto-hébergé, mono-thread (pas d'en-têtes COOP/COEP). Épin la limite Cloudflare de 25 Mio par fichier. Interroge le dataset actif plus tout CSV, Parquet ou JSON attaché à la session. -Résultats exportables en CSV ou transférables au ML Lab en un clic. Export -**Parquet** par `COPY … TO`. +Un résultat s'exporte en CSV ou en **Parquet**, ou part au ML Lab, en un clic. +Les trois sont écrits par le moteur via `COPY … TO` et portent toutes les lignes +du résultat, pas les 200 que la console peint ; un NULL part comme une cellule +vide. Voir [envoyer un résultat SQL au ML Lab](/docs/sql-vers-le-lab). ## Et ensuite ? diff --git a/src/content/docs/fr/sql-vers-le-lab.md b/src/content/docs/fr/sql-vers-le-lab.md index 7cebe3e..52db25b 100644 --- a/src/content/docs/fr/sql-vers-le-lab.md +++ b/src/content/docs/fr/sql-vers-le-lab.md @@ -28,6 +28,32 @@ disque au passage. - **Agréger** : passer d'une ligne par événement à une ligne par client, ce que la plupart des problèmes de ML réels demandent. +## Ce qui sort de la console + +La console peint les 200 premières lignes pour rester fluide, et dit toujours +combien de lignes la requête a rendues. **Les exports sont la requête, pas +l'écran** : le CSV, le fichier Parquet et l'envoi au ML Lab sont écrits par le +moteur lui-même, et portent toutes les lignes. + +- **Un NULL reste une valeur manquante.** Il part comme une cellule vide, que le + labo lit comme manquante. Une colonne numérique avec des NULL arrive comme un + nombre avec des trous. +- **Chaque requête a son propre nom de fichier**, par exemple + `query-3f2a91c4.csv` : une empreinte de la requête, jamais son texte. Le + registre du jeu de test du labo reconnaît un fichier à son nom et à sa forme, + et deux requêtes de même forme ne doivent pas partager un seul compte. +- **Le budget de cellules du labo s'applique.** Un résultat de plus de + 20 millions de cellules (lignes fois colonnes) est refusé avant d'être + construit, avec le conseil d'agréger ou de filtrer. C'est le même budget que + pour la lecture d'un fichier. + +Jusqu'à V69, les exports étaient construits à partir des lignes peintes : une +requête de 590 lignes en donnait 200 au labo, et un NULL arrivait sous la forme +de la marque que la console affiche pour lui, ce qui transformait une colonne +numérique en texte. Mesuré après la correction : les 590 lignes arrivent, et un +simple `SELECT *` sur chacune des six démos du labo redonne les mêmes lignes, +les mêmes colonnes et les mêmes types inférés que le fichier lui-même. + ## Le coût, annoncé Le moteur SQL est un téléchargement de 18 à 22 Mo, **optionnel**. Tant que vous diff --git a/src/features/data/SqlPanel.test.tsx b/src/features/data/SqlPanel.test.tsx index 9c0fbe6..a9b5202 100644 --- a/src/features/data/SqlPanel.test.tsx +++ b/src/features/data/SqlPanel.test.tsx @@ -3,6 +3,7 @@ import userEvent from '@testing-library/user-event'; import { MemoryRouter } from 'react-router'; import { beforeEach, describe, expect, it, vi } from 'vitest'; import type { SqlEngine } from '@/features/data/sql/engine'; +import { useDataStore } from '@/features/data/data-store'; import i18n from '@/lib/i18n'; /** @@ -12,6 +13,14 @@ import i18n from '@/lib/i18n'; */ const engines: Array = []; let release: (() => void) | null = null; +/** V69: what the fake engine answers, and what it was asked to export. */ +const fake = { + table: { columns: ['a', 'b'], rows: [['1', '2']], totalRows: 1, truncated: false }, + csv: 'a,b\n1,2\n', + csvAsked: [] as string[], + parquetAsked: [] as string[], + failExport: false, +}; vi.mock('@/features/data/sql/engine', () => ({ BUNDLE_BYTES: { eh: 18_157_568, mvp: 22_167_552 }, @@ -21,8 +30,16 @@ vi.mock('@/features/data/sql/engine', () => ({ flavour: 'eh' as const, closed: false, register: async () => undefined, - run: async () => ({ columns: [], rows: [], total: 0, truncated: false }), - toParquet: async () => new Uint8Array(), + run: async () => fake.table, + toParquet: async (sql: string) => { + fake.parquetAsked.push(sql); + return new Uint8Array([1, 2, 3]); + }, + toCsv: async (sql: string) => { + fake.csvAsked.push(sql); + if (fake.failExport) throw new Error('Parser Error: syntax error'); + return fake.csv; + }, close: async () => { engine.closed = true; }, @@ -32,6 +49,11 @@ vi.mock('@/features/data/sql/engine', () => ({ }), })); +const downloads: { name: string; content: unknown }[] = []; +vi.mock('@/lib/download', () => ({ + downloadFile: (name: string, content: unknown) => void downloads.push({ name, content }), +})); + const { SqlPanel } = await import('@/features/data/SqlPanel'); function renderPanel() { @@ -45,6 +67,12 @@ function renderPanel() { beforeEach(async () => { engines.length = 0; release = null; + downloads.length = 0; + fake.csvAsked.length = 0; + fake.parquetAsked.length = 0; + fake.failExport = false; + fake.table = { columns: ['a', 'b'], rows: [['1', '2']], totalRows: 1, truncated: false }; + useDataStore.setState({ labHandoff: null }); await i18n.changeLanguage('en'); }); @@ -75,3 +103,87 @@ describe('SqlPanel engine lifecycle', () => { await waitFor(() => expect(engines[0].closed).toBe(true)); }); }); + +/** Opens the console and runs `sql`, leaving a result on screen. */ +async function runQuery(sql: string) { + const user = userEvent.setup(); + renderPanel(); + await user.click(screen.getByTestId('sql-open')); + await waitFor(() => expect(release).not.toBeNull()); + await act(async () => release?.()); + const input = await screen.findByTestId('sql-input'); + await user.clear(input); + await user.type(input, sql); + await user.click(screen.getByTestId('sql-run')); + await screen.findByTestId('sql-result'); + return user; +} + +describe('SqlPanel exports (V69)', () => { + it('hands the lab the whole result written by the engine, not the painted rows', async () => { + // The console paints 200 rows of 590; the engine writes all of them. + fake.table = { + columns: ['a', 'b'], + rows: Array.from({ length: 200 }, (_, i) => [String(i), 'x']), + totalRows: 590, + truncated: true, + }; + fake.csv = ['a,b', ...Array.from({ length: 590 }, (_, i) => `${i},`)].join('\n') + '\n'; + const user = await runQuery('SELECT * FROM dataset;'); + expect(screen.getByTestId('sql-export-note')).toHaveTextContent( + 'they carry all 590 rows, written by the engine itself', + ); + await user.click(screen.getByTestId('sql-to-lab')); + await waitFor(() => expect(useDataStore.getState().labHandoff).not.toBeNull()); + const handoff = useDataStore.getState().labHandoff!; + // The trailing semicolon is gone: COPY (…) would refuse it. + expect(fake.csvAsked).toEqual(['SELECT * FROM dataset']); + expect(handoff.content.trim().split('\n')).toHaveLength(591); + expect(handoff.content).not.toContain('∅'); + expect(handoff.name).toMatch(/^query-[0-9a-f]{8}\.csv$/); + }); + + it('exports the query behind the result on screen, even after the box was edited', async () => { + const user = await runQuery('SELECT 1'); + await user.type(screen.getByTestId('sql-input'), ' FROM nowhere'); + await user.click(screen.getByTestId('sql-export-csv')); + await waitFor(() => expect(downloads).toHaveLength(1)); + expect(fake.csvAsked).toEqual(['SELECT 1']); + expect(downloads[0].name).toMatch(/^query-[0-9a-f]{8}\.csv$/); + expect(downloads[0].content).toBe(fake.csv); + }); + + it('offers the Parquet export the engine always had', async () => { + const user = await runQuery('SELECT 1'); + await user.click(screen.getByTestId('sql-export-parquet')); + await waitFor(() => expect(downloads).toHaveLength(1)); + expect(fake.parquetAsked).toEqual(['SELECT 1']); + expect(downloads[0].name).toMatch(/^query-[0-9a-f]{8}\.parquet$/); + }); + + it('refuses a result past the lab cell budget before building it', async () => { + fake.table = { + columns: ['a', 'b', 'c'], + rows: [['1', '2', '3']], + totalRows: 7_000_000, + truncated: true, + }; + const user = await runQuery('SELECT 1'); + await user.click(screen.getByTestId('sql-to-lab')); + expect(await screen.findByTestId('sql-export-error')).toHaveTextContent( + 'This result has 7,000,000 rows and 3 columns: more cells than the 20,000,000 the lab reads', + ); + expect(fake.csvAsked).toEqual([]); + expect(useDataStore.getState().labHandoff).toBeNull(); + }); + + it('reports an export the engine refuses, in the engine own words', async () => { + fake.failExport = true; + const user = await runQuery('SELECT 1'); + await user.click(screen.getByTestId('sql-export-csv')); + expect(await screen.findByTestId('sql-export-error')).toHaveTextContent( + 'The result could not be exported (Parser Error: syntax error).', + ); + expect(downloads).toHaveLength(0); + }); +}); diff --git a/src/features/data/SqlPanel.tsx b/src/features/data/SqlPanel.tsx index edcc957..1f9a5a1 100644 --- a/src/features/data/SqlPanel.tsx +++ b/src/features/data/SqlPanel.tsx @@ -7,7 +7,14 @@ import { Card } from '@/components/ui/card'; import { Eyebrow } from '@/components/ui/eyebrow'; import { useDataStore } from '@/features/data/data-store'; import { BUNDLE_BYTES, type SqlEngine } from '@/features/data/sql/engine'; -import { readerFor, tableNameFor, tableToCsv, type SqlTable } from '@/features/data/sql/table'; +import { + exportableQuery, + queryFileName, + readerFor, + tableNameFor, + type SqlTable, +} from '@/features/data/sql/table'; +import { MAX_CELLS } from '@/features/ml/data/limits'; import { ScrollRegion } from '@/components/ui/scroll-region'; import { downloadFile } from '@/lib/download'; @@ -36,6 +43,11 @@ export function SqlPanel() { const [tables, setTables] = useState([]); const [sql, setSql] = useState(`SELECT * FROM ${MAIN_VIEW} LIMIT 20`); const [result, setResult] = useState(null); + // V69: the query behind the result on screen. An export runs THIS, not the + // text of the box, which may have been edited since. + const [resultSql, setResultSql] = useState(''); + const [exporting, setExporting] = useState<'csv' | 'parquet' | 'lab' | null>(null); + const [exportError, setExportError] = useState(null); const [queryError, setQueryError] = useState(null); const [running, setRunning] = useState(false); @@ -129,7 +141,9 @@ export function SqlPanel() { setRunning(true); setQueryError(null); try { + setExportError(null); setResult(await engine.run(sql, DISPLAY_CAP)); + setResultSql(sql); } catch (error) { // DuckDB's own message names the line and the token — far more useful // than anything we could paraphrase. @@ -140,16 +154,60 @@ export function SqlPanel() { } }, [sql]); - const download = useCallback(() => { - if (!result) return; - downloadFile('query.csv', tableToCsv(result), 'text/csv'); - }, [result]); - - const toLab = useCallback(() => { - if (!result) return; - sendSqlToLab('query.csv', tableToCsv(result)); - void navigate('/ml'); - }, [navigate, result, sendSqlToLab]); + /** + * V69: an export is the query, not the screen. The console paints 200 rows; + * the engine writes every row the query returns, under the lab's own cell + * budget (V54) so a result the lab would refuse is not built at all. + */ + const exportResult = useCallback( + async (kind: 'csv' | 'parquet' | 'lab') => { + const engine = engineRef.current; + if (!engine || !result || exporting) return; + if (result.totalRows * result.columns.length > MAX_CELLS) { + setExportError( + t('data.sql.tooLarge', { + rows: result.totalRows.toLocaleString(lang), + columns: result.columns.length, + budget: MAX_CELLS.toLocaleString(lang), + }), + ); + return; + } + setExporting(kind); + setExportError(null); + try { + const query = exportableQuery(resultSql); + if (kind === 'parquet') { + const bytes = await engine.toParquet(query); + downloadFile( + queryFileName(resultSql, 'parquet'), + bytes as BlobPart, + 'application/vnd.apache.parquet', + ); + return; + } + const csv = await engine.toCsv(query); + const name = queryFileName(resultSql, 'csv'); + if (kind === 'csv') { + downloadFile(name, csv, 'text/csv'); + } else if (mountedRef.current) { + sendSqlToLab(name, csv); + void navigate('/ml'); + } + } catch (error) { + if (mountedRef.current) { + setExportError( + t('data.sql.exportFailed', { + reason: error instanceof Error ? error.message : String(error), + }), + ); + } + } finally { + if (mountedRef.current) setExporting(null); + } + }, + [exporting, lang, navigate, result, resultSql, sendSqlToLab, t], + ); return ( @@ -261,15 +319,50 @@ export function SqlPanel() { -
- - + + {exporting !== null && ( + + {t('data.sql.exporting', { total: result.totalRows.toLocaleString(lang) })} + + )}
- {result.truncated &&

{t('data.sql.capNote')}

} + {exportError && ( +

+ {exportError} +

+ )} +

+ {t(result.truncated ? 'data.sql.capNote' : 'data.sql.exportNote', { + total: result.totalRows.toLocaleString(lang), + })} +

)} diff --git a/src/features/data/sql/engine.ts b/src/features/data/sql/engine.ts index abce1a1..0557a7e 100644 --- a/src/features/data/sql/engine.ts +++ b/src/features/data/sql/engine.ts @@ -33,6 +33,13 @@ export interface SqlEngine { * leave the tab: the caller turns them into a download. */ toParquet(sql: string): Promise; + /** + * V69: the WHOLE result of a query as CSV text, written by DuckDB itself. + * The table the console paints holds 200 rows; an export built from it + * carried 200 rows, whatever the query returned. A NULL is an empty cell + * here, as in any file, and not the mark the console shows for it. + */ + toCsv(sql: string): Promise; close(): Promise; } @@ -96,6 +103,16 @@ export async function openEngine(): Promise { await db.dropFile(name).catch(() => undefined); } }, + async toCsv(sql) { + exports += 1; + const name = `export-${exports}.csv`; + try { + await connection.query(`COPY (${sql}) TO '${name}' (FORMAT CSV, HEADER)`); + return new TextDecoder().decode(await db.copyFileToBuffer(name)); + } finally { + await db.dropFile(name).catch(() => undefined); + } + }, async close() { await connection.close(); await db.terminate(); diff --git a/src/features/data/sql/table.test.ts b/src/features/data/sql/table.test.ts index 1c821b5..00039d8 100644 --- a/src/features/data/sql/table.test.ts +++ b/src/features/data/sql/table.test.ts @@ -1,7 +1,9 @@ import { describe, expect, it } from 'vitest'; import { + exportableQuery, formatCell, NULL_CELL, + queryFileName, readerFor, tableNameFor, tableToCsv, @@ -82,3 +84,26 @@ describe('tableNameFor', () => { expect(tableNameFor('.csv')).toBe('dropped'); }); }); + +describe('exportableQuery', () => { + it('removes a trailing semicolon and nothing else', () => { + expect(exportableQuery(' SELECT 1 ;; \n')).toBe('SELECT 1'); + expect(exportableQuery("SELECT ';' AS s")).toBe("SELECT ';' AS s"); + expect(exportableQuery('SELECT 1; SELECT 2')).toBe('SELECT 1; SELECT 2'); + }); +}); + +describe('queryFileName', () => { + it('gives each query a name of its own, stable, and free of the query text', () => { + const a = queryFileName("SELECT * FROM dataset WHERE city = 'Lyon'", 'csv'); + const b = queryFileName("SELECT * FROM dataset WHERE city = 'Paris'", 'csv'); + expect(a).toMatch(/^query-[0-9a-f]{8}\.csv$/); + expect(a).not.toBe(b); + expect(a).not.toContain('Lyon'); + expect(queryFileName('SELECT 1', 'parquet')).toMatch(/\.parquet$/); + }); + + it('names the same query the same way with or without its semicolon', () => { + expect(queryFileName('SELECT 1;', 'csv')).toBe(queryFileName('SELECT 1', 'csv')); + }); +}); diff --git a/src/features/data/sql/table.ts b/src/features/data/sql/table.ts index d61665e..9a66adf 100644 --- a/src/features/data/sql/table.ts +++ b/src/features/data/sql/table.ts @@ -9,6 +9,7 @@ * Pure functions, so the formatting is unit-tested rather than eyeballed in a * browser. */ +import { fingerprint } from '@/lib/fingerprint'; import { toCsv } from '@/lib/csv'; export interface SqlTable { @@ -60,7 +61,31 @@ export function toSqlTable( }; } -/** RFC-4180-ish escaping, matching what the Data Studio exports elsewhere. */ +/** + * V69: the query as `COPY (…)` can take it. A trailing semicolon is the only + * thing removed: anything else DuckDB refuses is refused in DuckDB's words. + */ +export function exportableQuery(sql: string): string { + return sql + .trim() + .replace(/;+\s*$/, '') + .trim(); +} + +/** + * V69: a name of its own for the file a query produces. Every handoff used to + * be `query.csv`, and the test-set ledger (V60), which knows a file by its + * name and shape, took two queries of the same shape for one file. The name + * carries a fingerprint of the query, never its text. + */ +export function queryFileName(sql: string, extension: 'csv' | 'parquet'): string { + return `query-${fingerprint(exportableQuery(sql))}.${extension}`; +} + +/** + * The painted rows as CSV. Not an export of the result: it holds what the + * console shows, NULL marks included. Exports go through the engine (V69). + */ export function tableToCsv(table: SqlTable): string { return toCsv(table.columns, table.rows); } diff --git a/src/features/ml/train/ledger.ts b/src/features/ml/train/ledger.ts index a8b0612..8073489 100644 --- a/src/features/ml/train/ledger.ts +++ b/src/features/ml/train/ledger.ts @@ -42,6 +42,7 @@ * Pure functions only. The table lives in `projects/ledger-db.ts`. */ import type { SplitChoice } from '@/features/ml/train/types'; +import { fingerprint } from '@/lib/fingerprint'; /** * From this many distinct runs, the best test score seen carries a visible @@ -114,16 +115,6 @@ export function testSetKey( return parts.join('\u001f'); } -/** FNV-1a, 32 bits: enough to tell configurations apart, and it names no column. */ -function fingerprint(text: string): string { - let hash = 0x811c9dc5; - for (let i = 0; i < text.length; i++) { - hash ^= text.charCodeAt(i); - hash = Math.imul(hash, 0x01000193); - } - return (hash >>> 0).toString(16).padStart(8, '0'); -} - /** What makes two runs on the same test set different: the columns and the weighting. */ export function configKey(features: readonly string[], classWeighting: boolean): string { const columns = [...features].sort().join('\u001f'); diff --git a/src/lib/fingerprint.test.ts b/src/lib/fingerprint.test.ts new file mode 100644 index 0000000..9d9d4ac --- /dev/null +++ b/src/lib/fingerprint.test.ts @@ -0,0 +1,14 @@ +import { describe, expect, it } from 'vitest'; +import { fingerprint } from '@/lib/fingerprint'; + +describe('fingerprint', () => { + it('is eight hexadecimal digits, the same for the same text', () => { + expect(fingerprint('SELECT 1')).toMatch(/^[0-9a-f]{8}$/); + expect(fingerprint('SELECT 1')).toBe(fingerprint('SELECT 1')); + expect(fingerprint('')).toBe('811c9dc5'); + }); + + it('tells two texts apart', () => { + expect(fingerprint('SELECT 1')).not.toBe(fingerprint('SELECT 2')); + }); +}); diff --git a/src/lib/fingerprint.ts b/src/lib/fingerprint.ts new file mode 100644 index 0000000..4ac4f8a --- /dev/null +++ b/src/lib/fingerprint.ts @@ -0,0 +1,15 @@ +/** + * FNV-1a, 32 bits, as eight hexadecimal digits: enough to tell two short + * texts apart, and it carries nothing of what it was computed from. + * + * V60 used it to name a run's configuration without naming its columns; V69 + * uses it to name the file a SQL query produces without quoting the query. + */ +export function fingerprint(text: string): string { + let hash = 0x811c9dc5; + for (let i = 0; i < text.length; i++) { + hash ^= text.charCodeAt(i); + hash = Math.imul(hash, 0x01000193); + } + return (hash >>> 0).toString(16).padStart(8, '0'); +} diff --git a/src/locales/en.json b/src/locales/en.json index a800852..11ea4eb 100644 --- a/src/locales/en.json +++ b/src/locales/en.json @@ -1386,9 +1386,14 @@ "addFile": "Add a file (CSV, Parquet, JSON)", "rows": "{{total}} rows.", "rowsTruncated": "First {{shown}} rows shown out of {{total}}.", - "capNote": "The display is capped to stay responsive; the CSV export and the ML Lab handoff carry the rows shown. Add a LIMIT or a filter to frame the result.", + "capNote": "The display is capped to stay responsive. The exports and the ML Lab handoff are not: they carry all {{total}} rows, written by the engine itself.", "exportCsv": "Export as CSV", - "toLab": "Open in the ML Lab" + "toLab": "Open in the ML Lab", + "exportNote": "The exports and the ML Lab handoff are written by the engine itself: an empty value stays an empty cell.", + "exportParquet": "Export as Parquet", + "exporting": "Writing {{total}} rows…", + "exportFailed": "The result could not be exported ({{reason}}).", + "tooLarge": "This result has {{rows}} rows and {{columns}} columns: more cells than the {{budget}} the lab reads. Aggregate or filter the query, then export." } }, "ai": { diff --git a/src/locales/fr.json b/src/locales/fr.json index 1599e04..5966648 100644 --- a/src/locales/fr.json +++ b/src/locales/fr.json @@ -1386,9 +1386,14 @@ "addFile": "Ajouter un fichier (CSV, Parquet, JSON)", "rows": "{{total}} lignes.", "rowsTruncated": "{{shown}} premières lignes affichées sur {{total}} au total.", - "capNote": "L'affichage est plafonné pour rester fluide ; l'export CSV et l'envoi au ML Lab portent sur les lignes affichées. Ajoutez un LIMIT ou un filtre pour cadrer le résultat.", + "capNote": "L'affichage est plafonné pour rester fluide. Les exports et l'envoi au ML Lab ne le sont pas : ils portent les {{total}} lignes, écrites par le moteur lui-même.", "exportCsv": "Exporter en CSV", - "toLab": "Ouvrir dans le ML Lab" + "toLab": "Ouvrir dans le ML Lab", + "exportNote": "Les exports et l'envoi au ML Lab sont écrits par le moteur lui-même : une valeur vide reste une cellule vide.", + "exportParquet": "Exporter en Parquet", + "exporting": "Écriture de {{total}} lignes…", + "exportFailed": "Le résultat n'a pas pu être exporté ({{reason}}).", + "tooLarge": "Ce résultat a {{rows}} lignes et {{columns}} colonnes : plus de cellules que les {{budget}} que le labo sait lire. Agrégez ou filtrez la requête, puis exportez." } }, "ai": {