Skip to content

V69: a SQL result reaches the lab whole - #169

Merged
dapiced merged 1 commit into
mainfrom
dom/modest-darwin-4oe9q0
Nov 13, 2026
Merged

dapiced merged 1 commit into
mainfrom
dom/modest-darwin-4oe9q0

Conversation

@dapiced

@dapiced dapiced commented Nov 13, 2026

Copy link
Copy Markdown
Owner

What

G1, the first of ten proposals from a new reading of the repository after V68. The how-to page presents « filter, join or aggregate in SQL, then send to the ML Lab » as the normal path. In the code that path was cut short.

The console paints 200 rows, which is sound. Its three ways out were built from those painted rows: toLab and the CSV export sent tableToCsv(result), a table already cut to 200. A note in the interface said so, but nobody could train on a SQL result of more than 200 rows, and the 118-row demo hid it.

Measured before the fix

In the browser, with the real engine, on a query returning 590 rows with a NULL in a numeric column:

  • the lab received 200 rows;
  • it read the column as categorical: the NULL had left as ∅, the mark the console paints for it, and the lab took it for a category.

The fix

An export is the query, not the screen.

  • SqlEngine.toCsv asks DuckDB for the file with COPY (…) TO … (FORMAT CSV, HEADER), as toParquet already did.
  • The CSV export, the lab handoff and a new Parquet button all go through the engine (the Parquet method existed since V40; no button called it, though the reference announced it).
  • They run the query behind the result on screen, not the text of the box, which may have been edited since.
  • The lab's cell budget (20 million) is applied before anything is built.
  • Each query gets a file name of its own (query-3f2a91c4.csv, a fingerprint, never the text). Every handoff used to be query.csv, and the test-set ledger took two queries of the same shape for one file. The FNV-1a function moved from the ledger to lib/fingerprint.ts instead of being copied.

Measured after

  • The 590 rows arrive, and the column is numeric with missing values.
  • A plain SELECT * over each of the lab's six demos gives back the same rows, the same columns and the same inferred types as the file loaded directly.
  • Writing 118,000 rows took 0.2 s and 4.9 MB; 1,180,000 rows took 6.4 s and 49 MB, on the single-threaded engine.

Not in this wave

Making SQL read a file the way the rest of the app does (decimal comma, windows-1252, Excel, the cleaned or joined table): what leaves the console is complete, what enters it is still DuckDB's own reading, and the limits page says so. Carrying the recipe or the contract with the handoff. Raising the 200 painted rows.

Docs

« Hand a SQL result to the ML Lab » (new section on what leaves the console), Data Studio reference, limits, in both languages; README; PLAN row V69; changelog; version 1.69.0. The nine remaining proposals (G2 to G10) are now written in PLAN.md's « Proposed features » section.

Checks

  • Lint, typecheck (apart from the xlsx types unreachable in my sandbox), format: clean.
  • Unit: 1325 passed locally, 5 failed, all in spreadsheet.test.ts (the xlsx tarball cannot be fetched in my sandbox; CI is the reference). 10 new tests; the panel's test now checks that the lab receives what the engine wrote, not the painted rows.
  • E2E: 133 passed locally after a fresh build, 1 failed (lab.spec.ts:31, the Excel upload, same cause). e2e/sql.spec.ts now runs the 590-row query with the real engine: CSV row count, no ∅, a valid Parquet file, and the lab showing 590 rows with the NULL column read as numeric.
  • Screenshot review in French, light desktop and dark mobile, no sideways overflow.

Generated by Claude Code

The SQL console paints 200 rows, and its three ways out were built from
those painted rows: a query returning 590 rows handed 200 to the ML Lab,
and a NULL left as the mark the console shows for it, which turned a
numeric column into text. The how-to page presents this handoff as the
normal path.

An export is now the query, not the screen. The engine writes the CSV
itself with COPY, as it already did for Parquet; the CSV export, the lab
handoff and a new Parquet button all go through it, on the query behind
the result on screen. The lab's cell budget is applied before anything
is built, and each query gets a file name of its own so the test-set
ledger no longer takes two queries of the same shape for one file.

Measured with the real engine: the 590 rows arrive, the column is
numeric with missing values, and SELECT * over each of the six demos
gives back the same rows, columns and inferred types as the file.

Docs in both languages, README, PLAN row V69 and the nine remaining
proposals (G2 to G10), version 1.69.0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AyBsUet8WybTG986ubroar
@dapiced
dapiced marked this pull request as ready for review November 13, 2026 20:43
@dapiced
dapiced merged commit b3a4d24 into main Nov 13, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants