Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
the reason the wave was built.

## V69 — A SQL result reaches the lab whole

G1, the first of the ten proposals a new reading of the repository produced after V68 (the others are listed below).

_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way.

## V68 — Counterfactuals: what would have to change

F7, the last feature proposed.
Expand Down
19 changes: 16 additions & 3 deletions PLAN.md

Large diffs are not rendered by default.

12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,8 @@ _The index is built at compile time and queried in the tab — typing here sends
no correct answer is lost.
- **Analytical SQL** (`/data`): a real OLAP engine — DuckDB-Wasm, MIT, self-hosted and
single-threaded — queries the loaded file in the browser, plus any CSV, **Parquet** or
JSON attached in the session. Results export to CSV or move to the ML Lab in one click;
JSON attached in the session. A result exports to CSV or Parquet, or moves to the ML Lab,
in one click, written by the engine itself: every row of it, not the 200 the console paints;
SQL errors show DuckDB's own message. The engine is pinned to 1.28.0 for a measured
reason: from 1.29 its binaries exceed Cloudflare's 25 MiB per-file limit.
- **Data assistant** (`/ai/chat`): plain French or English questions about a loaded
Expand Down Expand Up @@ -288,7 +289,7 @@ Wikimedia Commons), the portrait is NASA, public domain._
a doc page carries its whole article, in English until the app mounts); Lighthouse
mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
Runtime) load lazily.
- **Quality bar.** 1320 unit tests and 134 Playwright end-to-end tests across five
- **Quality bar.** 1330 unit tests and 134 Playwright end-to-end tests across five
projects — desktop, a phone viewport in English and in French, dark mode, and
Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on
Expand Down Expand Up @@ -407,8 +408,11 @@ report, which had been calling small files drifted (V62), a data contract that
requires of the next file what was found on a reference (V63), and a scikit-learn script
that redoes a run outside the browser and measures its own gap (V64), a submission file
for a competition (V65), a step-by-step replay of one row through the model (V66), early
stopping for the boosting, measured and not built (V67), and counterfactuals (V68). No
feature is proposed at the moment. The consented generative chat (V12) stays
stopping for the boosting, measured and not built (V67), counterfactuals (V68), and a SQL
result that reaches the lab whole instead of its first 200 rows (V69). Proposed, none
scheduled: nine items from a new reading of the repository, listed in PLAN.md (duplicate
columns, a count of filled cells, ranking on probabilities, an honest forecast band, and
others). The consented generative chat (V12) stays
postponed by product decision. No wave starts without an explicit launch command. The full plan, delivery log and design decisions live in
[PLAN.md](PLAN.md); [CHANGELOG.md](CHANGELOG.md) is extracted from it — one entry per
wave, newest first — by `npm run changelog`, and a test fails when the two disagree.
Expand Down
45 changes: 44 additions & 1 deletion e2e/sql.spec.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { readFileSync } from 'node:fs';
import { expect, test } from '@playwright/test';

test.use({ locale: 'en-US' });
Expand All @@ -6,7 +7,9 @@ test.use({ locale: 'en-US' });
// twice in parallel is the slowest thing this suite could do for no gain.
test.describe.configure({ timeout: 180_000 });

test('the SQL console answers a real query and reports the engine own errors', async ({ page }) => {
test('the SQL console answers a real query and reports the engine own errors', async ({
page,
}, testInfo) => {
await page.goto('/data');
await page.getByRole('button', { name: /cafe-sales\.csv/ }).click();
await expect(page.getByTestId('sql-panel')).toBeVisible();
Expand Down Expand Up @@ -37,4 +40,44 @@ test('the SQL console answers a real query and reports the engine own errors', a
const error = page.getByTestId('sql-error');
await expect(error).toBeVisible({ timeout: 60_000 });
await expect(error).toContainText(/nope/i);

// V69: an export is the query, not the screen. 590 rows come back, 200 are
// painted, and every one of them leaves the console.
await page
.getByTestId('sql-input')
.fill(
'SELECT a.*, CASE WHEN r.range = 0 THEN NULL ELSE a.quantity END AS q2 FROM dataset a, range(5) r;',
);
await page.getByTestId('sql-run').click();
await expect(result).toContainText('First 200 rows shown out of 590', { timeout: 60_000 });
await expect(result.locator('tbody tr')).toHaveCount(200);
await expect(page.getByTestId('sql-export-note')).toContainText('they carry all 590 rows');

const csvDownload = page.waitForEvent('download');
await page.getByTestId('sql-export-csv').click();
const csvFile = await csvDownload;
expect(csvFile.suggestedFilename()).toMatch(/^query-[0-9a-f]{8}\.csv$/);
const csvPath = testInfo.outputPath('query.csv');
await csvFile.saveAs(csvPath);
const lines = readFileSync(csvPath, 'utf8').trim().split('\n');
expect(lines).toHaveLength(591);
expect(lines[0]).toBe('order_id,date,product,city,quantity,unit_price,currency,notes,q2');
// A NULL is an empty cell, not the mark the console paints for it.
expect(lines.join('\n')).not.toContain('∅');
expect(lines.filter((line) => line.endsWith(','))).toHaveLength(118);

const parquetDownload = page.waitForEvent('download');
await page.getByTestId('sql-export-parquet').click();
const parquetFile = await parquetDownload;
expect(parquetFile.suggestedFilename()).toMatch(/^query-[0-9a-f]{8}\.parquet$/);
const parquetPath = testInfo.outputPath('query.parquet');
await parquetFile.saveAs(parquetPath);
expect(readFileSync(parquetPath).subarray(0, 4).toString()).toBe('PAR1');

// The lab receives the whole result, and reads the column with NULLs as a
// number with missing values: it used to get 200 rows and a text column.
await page.getByTestId('sql-to-lab').click();
await expect(page).toHaveURL(/\/ml/);
await expect(page.getByText('590 rows · 9 columns')).toBeVisible({ timeout: 30_000 });
await expect(page.getByTestId('column-card-q2')).toContainText(/numeric/i);
});
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "labml",
"private": true,
"version": "1.68.0",
"version": "1.69.0",
"license": "MIT",
"type": "module",
"engines": {
Expand Down
2 changes: 2 additions & 0 deletions src/content/docs/en/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ They are not equivalent, and mixing them would be dishonest:
| A counterfactual of more than two columns, or one that edits free text | A change of five columns is no longer an explanation, and a sentence has no nearest neighbour to try. When the answer sits in the text, the search finds nothing and says so |
| Knowing that two columns say the same thing | Changing one of two redundant columns alone can describe a row that cannot exist. The search flags a changed row far from every training row, which does not catch this case: lock the duplicate, or leave it out of the features |
| Counterfactuals for a regression, a re-imported model, or with the decision threshold | « Another class » has no meaning for a value; an export carries no training rows to draw observed values from; and the change found flips the prediction the what-if panel shows, not the decision of a chosen threshold |
| SQL reading a file the way the rest of the app does | The console hands the file to DuckDB's own reader: a decimal comma, a windows-1252 file or an Excel sheet are read by the app and not by SQL, which also always sees the original file, before the recipe and before a join. What leaves the console is complete since V69; what enters it is still DuckDB's reading |
| The recipe or the contract travelling with a SQL result | The ML Lab handoff carries the rows and a file name, nothing else |
| A generative chat backed by a remote API (V12) | Postponed by product decision (confirmed 09/10/2026). Today's chat runs entirely in the browser; a remote API would send out the question and the column schema, never the data, and would require a consent screen |

## Dropped after measurement
Expand Down
6 changes: 4 additions & 2 deletions src/content/docs/en/reference-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,10 @@ DuckDB-Wasm, MIT, self-hosted, single-threaded (no COOP/COEP headers). Pinned to
25 MiB per-file limit.

Queries the active dataset plus any CSV, Parquet or JSON attached to the
session. Results export to CSV or move to the ML Lab in one click. **Parquet**
export through `COPY … TO`.
session. A result exports to CSV or **Parquet**, or moves to the ML Lab, in one
click. All three are written by the engine through `COPY … TO` and carry every
row of the result, not the 200 the console paints; a NULL leaves as an empty
cell. See [hand a SQL result to the ML Lab](/docs/sql-vers-le-lab).

## Where to go next

Expand Down
24 changes: 24 additions & 0 deletions src/content/docs/en/sql-to-the-lab.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,30 @@ way.
- **Aggregate**: go from one row per event to one row per customer, which is
what most real ML problems need.

## What leaves the console

The console paints the first 200 rows so it stays responsive, and always states
how many rows the query returned. **The exports are the query, not the screen**:
the CSV, the Parquet file and the ML Lab handoff are written by the engine
itself, and carry every row.

- **A NULL stays a missing value.** It leaves as an empty cell, which the lab
reads as missing. A numeric column with NULLs arrives as a number with gaps.
- **Each query has a file name of its own**, such as `query-3f2a91c4.csv`: a
fingerprint of the query, never its text. The lab's test-set ledger knows a
file by its name and shape, and two queries of the same shape must not share
one count.
- **The lab's cell budget applies.** A result past 20 million cells (rows times
columns) is refused before it is built, with the advice to aggregate or
filter. It is the same budget a file is read under.

Until V69 the exports were built from the painted rows: a query returning 590
rows handed 200 to the lab, and a NULL arrived as the mark the console shows
for it, which turned a numeric column into text. Measured after the fix: the
590 rows arrive, and a plain `SELECT *` over each of the lab's six demos gives
back the same rows, the same columns and the same inferred types as the file
itself.

## The cost, announced

The SQL engine is an 18–22 MB download, **optional**. Until you open it, it is
Expand Down
2 changes: 2 additions & 0 deletions src/content/docs/fr/limites.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ Elles ne se valent pas, et les mélanger serait malhonnête :
| Un contrefactuel de plus de deux colonnes, ou qui modifie du texte libre | Un changement de cinq colonnes n'est plus une explication, et une phrase n'a pas de plus proche voisine à essayer. Quand la réponse est dans le texte, la recherche ne trouve rien et le dit |
| Savoir que deux colonnes disent la même chose | Changer une seule de deux colonnes redondantes peut décrire une ligne qui ne peut pas exister. La recherche signale une ligne modifiée loin de toute ligne d'entraînement, ce qui n'attrape pas ce cas : verrouillez le doublon, ou retirez-le des variables |
| Des contrefactuels pour une régression, un modèle réimporté, ou avec le seuil de décision | « Une autre classe » n'a pas de sens pour une valeur ; un export ne porte pas de lignes d'entraînement où puiser des valeurs observées ; et le changement trouvé fait basculer la prédiction que montre le panneau « Et si… », pas la décision d'un seuil choisi |
| Le SQL qui lit un fichier comme le reste de l'application | La console confie le fichier au lecteur de DuckDB : une virgule décimale, un fichier windows-1252 ou une feuille Excel sont lus par l'application et pas par le SQL, qui voit en plus toujours le fichier d'origine, avant la recette et avant une jointure. Ce qui sort de la console est complet depuis V69 ; ce qui y entre reste la lecture de DuckDB |
| La recette ou le contrat qui voyagent avec un résultat SQL | L'envoi au ML Lab porte les lignes et un nom de fichier, rien d'autre |
| Un chat génératif branché sur une API distante (V12) | Reporté par décision produit (confirmé le 09/10/2026). Le chat actuel tourne entièrement dans le navigateur ; une API distante ferait sortir la question et le schéma des colonnes, jamais les données, et exigerait un écran de consentement |

## Ce qui a été abandonné après mesure
Expand Down
6 changes: 4 additions & 2 deletions src/content/docs/fr/reference-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,8 +167,10 @@ DuckDB-Wasm, MIT, auto-hébergé, mono-thread (pas d'en-têtes COOP/COEP). Épin
la limite Cloudflare de 25 Mio par fichier.

Interroge le dataset actif plus tout CSV, Parquet ou JSON attaché à la session.
Résultats exportables en CSV ou transférables au ML Lab en un clic. Export
**Parquet** par `COPY … TO`.
Un résultat s'exporte en CSV ou en **Parquet**, ou part au ML Lab, en un clic.
Les trois sont écrits par le moteur via `COPY … TO` et portent toutes les lignes
du résultat, pas les 200 que la console peint ; un NULL part comme une cellule
vide. Voir [envoyer un résultat SQL au ML Lab](/docs/sql-vers-le-lab).

## Et ensuite ?

Expand Down
26 changes: 26 additions & 0 deletions src/content/docs/fr/sql-vers-le-lab.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,32 @@ disque au passage.
- **Agréger** : passer d'une ligne par événement à une ligne par client, ce que
la plupart des problèmes de ML réels demandent.

## Ce qui sort de la console

La console peint les 200 premières lignes pour rester fluide, et dit toujours
combien de lignes la requête a rendues. **Les exports sont la requête, pas
l'écran** : le CSV, le fichier Parquet et l'envoi au ML Lab sont écrits par le
moteur lui-même, et portent toutes les lignes.

- **Un NULL reste une valeur manquante.** Il part comme une cellule vide, que le
labo lit comme manquante. Une colonne numérique avec des NULL arrive comme un
nombre avec des trous.
- **Chaque requête a son propre nom de fichier**, par exemple
`query-3f2a91c4.csv` : une empreinte de la requête, jamais son texte. Le
registre du jeu de test du labo reconnaît un fichier à son nom et à sa forme,
et deux requêtes de même forme ne doivent pas partager un seul compte.
- **Le budget de cellules du labo s'applique.** Un résultat de plus de
20 millions de cellules (lignes fois colonnes) est refusé avant d'être
construit, avec le conseil d'agréger ou de filtrer. C'est le même budget que
pour la lecture d'un fichier.

Jusqu'à V69, les exports étaient construits à partir des lignes peintes : une
requête de 590 lignes en donnait 200 au labo, et un NULL arrivait sous la forme
de la marque que la console affiche pour lui, ce qui transformait une colonne
numérique en texte. Mesuré après la correction : les 590 lignes arrivent, et un
simple `SELECT *` sur chacune des six démos du labo redonne les mêmes lignes,
les mêmes colonnes et les mêmes types inférés que le fichier lui-même.

## Le coût, annoncé

Le moteur SQL est un téléchargement de 18 à 22 Mo, **optionnel**. Tant que vous
Expand Down
Loading
Loading