Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ entry per wave, newest first; the first six waves (V1–V6, the MVP) predate the
are recorded in PLAN.md §B and §J. Each entry carries the opening sentence of its row and
the reason the wave was built.

## V65 — An unlabelled file becomes a submission

The second part of F8; the glass box stays proposed below.

_Why:_ Approved 13/11/2026. A learner who trains here should be able to carry the run to a notebook and see, in figures, what changed on the way.

## V64 — A run, redone in Python

The first of the three parts of F8, the two others staying proposed below.
Expand Down
11 changes: 6 additions & 5 deletions PLAN.md

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ The project follows three non-negotiable principles:
| No target? | Seeded k-means (k chosen by silhouette) + power-iteration PCA projection, groups described in plain language; date column? **Holt-Winters forecasting** validated by rolling-origin backtest |
| MLOps loop | Score a **new batch** with honest test-vs-batch metrics; **compare two runs** side by side with cross-run uncertainty verdicts, or **up to six at once** read against the oldest of the selection; **export a model as JSON and re-import it later** — the exact predictor is rebuilt (byte-identical predictions) and scores any CSV without retraining |
| Python bridge | **A run, redone with scikit-learn.** One readable script carries the row numbers of each split and the preprocessing, and translates each model to its nearest estimator. It does not promise the lab's figures: it prints them next to its own, with the gap. Measured on 30 runs: the encoded matrix is equal to within 2.9e-14, the baseline, the linear regression and the naive Bayes agree to the last digit, k-NN agrees until two neighbours tie, and the hand-written families differ by up to several test rows on one split. A column name is never code: the run travels as one JSON block. |
| Competition path | **An unlabelled file becomes a submission.** Under a scored batch, one block writes an identifier and a prediction under the names the platform expects: a class, a probability or a value, every row kept in its order, missing and repeated identifiers counted. The lab guesses neither the identifier nor the names. The same family can be **fitted again on every labelled row**, announced as a model nobody measured: on the demos, against rows kept away from both models, it gained 5 accuracy points on the smallest text file, 0.8 on titanic, 2 to 3 % of RMSE on two regressions, nothing on two others, and lost one time in five to one in three even where it paid |
| Speed | Heavy families train on **helper cores** (announced on the leaderboard, split by measured cost, never silent), and a model crosses back as JSON so it is rebuilt through the same path an imported model uses. Measured on a 60 000-row run: **74 s → 9.5 s**, with every leaderboard number identical. Every best-effort step that gives up is recorded in a **run journal** shown with the leaderboard and in the report: which step, which family, how many times, and the error it met |
| Persistence | Local run history with attached artifacts, opted-in dataset storage (compressed, explicit 50 MB budget), self-contained HTML reports, data-free share links (a damaged link is refused, never half-rendered) |
| Model card | A page for whoever must **decide to use** the model, in HTML or Markdown: intended use, data, performance with its interval, the selection gap, prediction-set coverage, the decision rule, where it fails, the run journal, and limits. The intended use is typed by the user; left empty the card says « not stated » and calls itself incomplete rather than guessing. Every analysis the run does not carry is listed under « Not assessed », and each section names the model it describes when that is not the winner |
Expand Down Expand Up @@ -285,7 +286,7 @@ Wikimedia Commons), the portrait is NASA, public domain._
a doc page carries its whole article, in English until the app mounts); Lighthouse
mobile ≈ 0.99 on `/ml` under real throttling. Heavy dependencies (Dexie, SheetJS, ONNX
Runtime) load lazily.
- **Quality bar.** 1224 unit tests and 131 Playwright end-to-end tests across five
- **Quality bar.** 1251 unit tests and 132 Playwright end-to-end tests across five
projects — desktop, a phone viewport in English and in French, dark mode, and
Cloudflare Pages' own routing emulated by `wrangler pages dev` (a real 404, the security
headers as served) — covering offline PWA, a fake webcam, a horizontal-overflow guard on
Expand Down
85 changes: 85 additions & 0 deletions e2e/submission.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
import { readFileSync, writeFileSync } from 'node:fs';
import { expect, test } from '@playwright/test';

test.use({ locale: 'en-US' });
test.setTimeout(120_000);

// NOTE: keep this title ASCII-only: it names the test-results directory, and
// Chromium's setInputFiles silently drops files whose path is non-ASCII.
test('submission: an unlabelled file becomes an identifier and a prediction', async ({
page,
}, testInfo) => {
await page.goto('/ml?demo=fraud&target=status');
await page.getByTestId('train-button').click();
await expect(page.getByTestId('train-again')).toBeVisible({ timeout: 60000 });
// A family with probabilities, and one that can be fitted again.
await page.getByRole('cell', { name: /Logistic regression/ }).click();
await expect(page.getByRole('button', { name: 'Model (JSON)' })).toBeEnabled();

// What a competition hands out: an identifier, the features, no label. The
// identifiers are not in order, so a sorted file would be caught.
const fraud = (await (await page.request.get('/datasets/fraud.csv')).text()).trim().split('\n');
const ids = Array.from({ length: 40 }, (_, i) => `tx-${(i * 17) % 40}`);
const unlabelled = [
`tx_id,${fraud[0].split(',').slice(0, 5).join(',')}`,
...fraud.slice(1, 41).map((line, i) => `${ids[i]},${line.split(',').slice(0, 5).join(',')}`),
].join('\n');
const batchPath = testInfo.outputPath('holdout.csv');
writeFileSync(batchPath, unlabelled);
await page.getByTestId('batch-score').locator('input[type="file"]').setInputFiles(batchPath);
const panel = page.getByTestId('submission-panel');
await expect(panel).toBeVisible({ timeout: 30000 });

// Nothing is chosen for the user: no identifier, no file.
await expect(panel.getByTestId('submission-build')).toBeDisabled();
await panel.getByTestId('submission-id-column').selectOption('tx_id');
await expect(panel.getByTestId('submission-id-name')).toHaveValue('tx_id');
await panel.getByTestId('submission-id-name').fill('TransactionId');
await panel.getByTestId('submission-prediction-name').fill('is_fraud');
await panel.getByTestId('submission-build').click();

const result = panel.getByTestId('submission-result');
await expect(result).toHaveAttribute('data-model', 'measured', { timeout: 30000 });
await expect(result.getByTestId('submission-summary')).toContainText(
'40 rows, the same as the scored file and in its order. Columns: TransactionId, is_fraud.',
);
await expect(result).toContainText('A submission is not a score');

const labelDownload = page.waitForEvent('download');
await result.getByTestId('submission-download').click();
const labelFile = await labelDownload;
expect(labelFile.suggestedFilename()).toBe('holdout-submission.csv');
const labelPath = testInfo.outputPath('labels.csv');
await labelFile.saveAs(labelPath);
const labels = readFileSync(labelPath, 'utf8').trim().split('\n');
expect(labels[0]).toBe('TransactionId,is_fraud');
expect(labels).toHaveLength(41);
expect(labels.slice(1).map((line) => line.split(',')[0])).toEqual(ids);
for (const line of labels.slice(1)) expect(['clean', 'fraud']).toContain(line.split(',')[1]);

// The same file as probabilities, from the family fitted again on every
// labelled row: announced as a model nobody measured.
await panel.getByTestId('submission-form-probability').check();
await panel.getByTestId('submission-model-refit').check();
await panel.getByTestId('submission-build').click();
await expect(result).toHaveAttribute('data-model', 'refit', { timeout: 60000 });
await expect(result.getByTestId('submission-model')).toContainText(
'fitted again on the 640 labelled rows. Nobody measured this model',
);
const probaDownload = page.waitForEvent('download');
await result.getByTestId('submission-download').click();
const probaPath = testInfo.outputPath('probabilities.csv');
await (await probaDownload).saveAs(probaPath);
const probabilities = readFileSync(probaPath, 'utf8').trim().split('\n');
expect(probabilities).toHaveLength(41);
expect(probabilities.slice(1).map((line) => line.split(',')[0])).toEqual(ids);
for (const line of probabilities.slice(1)) {
const value = Number(line.split(',')[1]);
expect(value).toBeGreaterThanOrEqual(0);
expect(value).toBeLessThanOrEqual(1);
}

// Another model inspected: the submission of the previous one is gone.
await page.getByRole('cell', { name: /Naive Bayes/ }).click();
await expect(page.getByTestId('submission-panel')).toHaveCount(0);
});
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "labml",
"private": true,
"version": "1.64.0",
"version": "1.65.0",
"license": "MIT",
"type": "module",
"engines": {
Expand Down
Loading
Loading