Next

.changeset/prefetch-run-gate.md

@@ -0,0 +1,6 @@
+---
+'@noormdev/cli': patch
+'@noormdev/sdk': patch
+---
+
+Decide which files to skip with one or two tracking queries per batch instead of one or two per file. Skipped files no longer write execution rows, and a build where nothing changed writes no operation record. The SDK's `run.file(path, { preview: true })` now renders without executing. The TUI's re-run confirmation now finds earlier runs.

docs/design/prefetch-run-gate.md

@@ -0,0 +1,43 @@
+# Prefetch run gate
+
+
+## Problem
+
+`executeFiles` decides skip-or-run one file at a time. Each file pays a `needsRun` SELECT and an `updateFileExecution` UPDATE, skip or not, so a 500-file build against a remote database spends about 1,000 round trips before any real work. The change executor has the same shape: `needsRunFile` runs a boundary SELECT and a history SELECT per file.
+
+
+## Concepts
+
+| Term | Meaning |
+|------|---------|
+| Gate | The pass before any tracking write: hash every file, fetch tracking state once, decide every file in memory. |
+| Run set | Files the gate decided must run. Only these get a pending row. |
+| Skip set | Files the gate decided are unchanged. They produce a `FileResult` and a `file:skip` event, and no database write. |
+| Decision function | A pure function from (latest record, current checksum, force) to `NeedsRunResult`. The per-file lookup and the gate share it. |
+
+
+## Business rules
+
+1. A skipped file is never recorded. No pending row, no skipped row, no UPDATE.
+2. A build or run whose run set is empty writes no operation row.
+3. Templates are rendered, then hashed, then decided. The rendered SQL is kept and executed, so each template renders once.
+4. The skip rules stay what they are today: force, new, failed, pending, a skip that is not `unchanged`, stale parent, changed checksum, and lookup error all run. Rows already in existing databases (`skipped/unchanged`, `pending`) still decide correctly.
+5. Execution stays sequential, in discovery order.
+6. A failure cascade still marks the run-set files that were never reached as `skipped`. These rows are run-set files that never executed, not unchanged-skips, and `needsRun` already reads them as must-run.
+7. On the change path the operation row is still written when every file skips. It records the change's combined checksum, which the change-level `needsRun` compares against.
+
+
+## Approach
+
+The prefetch is one SELECT per batch: the newest execution row per filepath for the config, chosen by a `MAX(id) ... GROUP BY filepath` subquery joined back to the executions and change tables. This form runs unchanged on Postgres, MySQL, MSSQL and SQLite, and has no IN list, so it never reaches MSSQL's 2,100-parameter limit. The result includes filepaths outside the batch, and the runner ignores those in memory.
+
+The change path fetches the opposite-direction boundary once per change rather than once per file. It then fetches the newest qualifying row per filepath for that change in one query.
+
+`runFile` goes through `executeFiles` like the other entry points. This removes the legacy insert-per-file path (`executeSingleFile`, `recordExecution`), which recorded skips.
+
+
+## Rejected approaches
+
+- **`ROW_NUMBER() OVER (PARTITION BY filepath)`.** Needs SQLite 3.25+ and MySQL 8, and the subquery form is equally fast at this scale.
+- **`WHERE filepath IN (...)` and reduce in JS.** Returns the whole history and is capped by MSSQL's parameter limit.
+- **Raw-hash gate for templates.** A template's output depends on config and secrets, so the raw bytes cannot prove the output is unchanged.

docs/dev/change.md

@@ -70,10 +70,26 @@ Changes use checksum-based detection at the change level. Each condition has an
 | `force` | `--force` flag provided | Run |
 | — | Successfully applied, unchanged | Skip |
 
-Unlike the runner which tracks individual files, changes track the combined checksum of all files in `change/` or `revert/`.
-
 The `stale` status is set by teardown operations that wipe the database. When you run `db:teardown`, all applied changes are marked as stale so they'll re-run on the next `ff` operation.
 
+This check uses the combined checksum of all files in `change/` or `revert/`.
+
+When the change runs, each file gets its own decision, so a retry after a fix re-runs only the files that need it. `ChangeHistory.latestFileExecutions(name, direction)` reads the newest qualifying execution row for every file of the change in two SELECTs, whatever the file count. `decideNeedsRunFile` then judges each file:
+
+| Reason | Condition | Action |
+|--------|-----------|--------|
+| `force` | `--force` flag provided | Run |
+| `new` | No qualifying row, or the lookup failed | Run |
+| `failed` | Last attempt failed | Run |
+| `changed` | File checksum differs | Run |
+| — | Last success has a matching checksum | Skip, `'already applied'` |
+
+A row qualifies only while its success still stands. Rows from a `reverted` or `stale` operation do not count, and neither do rows older than the newest operation in the opposite direction, so apply, revert, apply re-runs every file.
+
+A filepath listed more than once through `.txt` manifests runs at its first position and skips as `'already applied'` at the rest.
+
+A skipped file pushes its `skipped` result and makes no database call. Pending rows are written for the files that will run. The operation row is written even when every file skips, because it records the change checksum.
+
 
 ## Basic Usage
 
@@ -275,6 +291,8 @@ Change execution is recorded in the same tables as the runner. Names below are t
 | `status` | `'pending'`, `'success'`, `'failed'`, `'skipped'` |
 | `duration_ms` | Execution time (integer, not float) |
 
+`skipped` marks a file a failure left unreached. See [Change Detection](#change-detection) for files that get no row.
+
 **Note:** The `duration_ms` column is an integer. `performance.now()` returns floats, so all writes use `Math.round(durationMs)`. This is important for PostgreSQL compatibility.
 
 
@@ -496,9 +514,10 @@ const marked = await tracker.markAllAsStale()
 ```
 
 The base `Tracker` class (from `core/runner`) provides:
-- `needsRun()` / `needsRunByName()` - Change detection by filepath or name
+- `latestExecutions()` - Newest execution row per filepath, for the batch skip decision
+- `needsRunByName()` - Change detection by change name
 - `createOperation()` - Create operation records with direction
-- `createFileRecords()` - Create pending file records upfront
+- `createFileRecords()` - Create pending records for the files about to run
 - `updateFileExecution()` - Update individual file status
 - `skipRemainingFiles()` - Mark remaining files as skipped
 - `finalizeOperation()` - Complete operation with final status

docs/dev/datamodel.md

@@ -731,16 +731,16 @@ Used in `__noorm_executions__` and file results.
 
 ### Skip Reasons
 
-Free-form text, not an enum. The values emitted today:
+Free-form text, not an enum. A skip row marks only a file a failure left unreached. The values written today:
 
 | Reason | Meaning |
 |--------|---------|
-| unchanged | File checksum matches previous run |
-| already-run | File was already executed successfully |
-| already applied | The change as a whole was already applied |
+| `Skipped: failure in {file}` | A runner batch aborted at a named file; remaining files skipped |
 | change failed | Parent change failed with no single culprit file |
 | `{file} failed: {error}` | Parent change failed at a named file; remaining files skipped |
 
+A file skipped because a prior run covers it gets no row. Rows written by older versions may still hold `unchanged` (runner) or `already applied` (change) for that case.
+
 
 ### Lock
 

docs/dev/runner.md

@@ -10,7 +10,7 @@ You have SQL files that need to run against a database. But running them manuall
 - What happens if execution fails halfway through?
 - How do you preview what will run before committing?
 
-noorm's runner solves this with checksum-based change detection and execution tracking. Files that haven't changed are skipped. Failed files are automatically retried. Everything is logged to the database for auditability.
+noorm's runner solves this with checksum-based change detection and execution tracking. Files that haven't changed are skipped. Failed files are automatically retried. Every execution is logged to the database for auditability.
 
 
 ## How It Works
@@ -24,14 +24,15 @@ The runner provides four execution modes:
 | **Dir** | Execute all files in a directory | Directory path |
 | **Files** | Execute specific files selectively | Array of file paths |
 
-When you run a file:
+When you run a batch (a single file is a batch of one):
 
-1. Compute SHA-256 checksum of file contents
-2. Check tracking database for previous execution
-3. Skip if unchanged, run if new/changed/failed
-4. Render template if `.sql.tmpl` file
-5. Execute SQL against database
-6. Record result in tracking tables
+1. Render every `.sql.tmpl` file, then compute the SHA-256 checksum of the SQL that will execute
+2. Read the newest execution row per file from the tracking database, in one query for the whole batch
+3. Skip if unchanged, run if new/changed/failed. A skipped file writes no tracking row
+4. If no file is left to run, return here without writing an operation row
+5. Create the operation and a `pending` row for each file about to run
+6. Execute SQL against database, in discovery order
+7. Record result in tracking tables
 
 
 ## Change Detection
@@ -143,9 +144,9 @@ Every execution is recorded in two tables. Names below are the MySQL/SQLite pref
 |-------|-------------|
 | `change_id` | FK to parent operation |
 | `filepath` | File that was executed |
-| `checksum` | SHA-256 of file contents |
-| `status` | `'success'`, `'failed'`, `'skipped'` |
-| `skip_reason` | `'unchanged'` if skipped |
+| `checksum` | SHA-256 of the executed SQL (the rendered output, for a template) |
+| `status` | `'pending'`, `'success'`, `'failed'`, `'skipped'` (a file the batch never reached after an earlier failure) |
+| `skip_reason` | Which failure stopped the batch, if skipped |
 | `duration_ms` | Execution time |
 
 
@@ -420,15 +421,14 @@ This utility is used internally by build operations when applying settings rules
 
 ## Unified executeFiles
 
-The `executeFiles` function provides low-level file execution with full tracking. `runBuild`/`runDir`/`runFiles` funnel through it. The change module does **not** — `core/change/executor.ts` has its own private `executeFiles` so it can emit `change:file` events and honour manifest resolution; it shares only `computeChecksum`, `computeCombinedChecksum`, and the `Tracker` base class with the runner.
+The `executeFiles` function provides low-level file execution with full tracking. `runBuild`/`runDir`/`runFiles`/`runFile` funnel through it. The change module does **not** — `core/change/executor.ts` has its own private `executeFiles` so it can emit `change:file` events and honour manifest resolution; it shares only `computeChecksum`, `computeCombinedChecksum`, and the `Tracker` base class with the runner.
 
 ```typescript
 import { executeFiles, type FileInput } from './core/runner'
 
-// Prepare file inputs with pre-computed checksums
 const files: FileInput[] = [
-    { path: '/project/sql/001_users.sql', type: 'sql', checksum: 'abc123' },
-    { path: '/project/sql/002_posts.sql', type: 'sql', checksum: 'def456' },
+    { path: '/project/sql/001_users.sql', type: 'sql' },
+    { path: '/project/sql/002_posts.sql', type: 'sql' },
 ]
 
 // Four arguments: run options and execution options are separate objects
@@ -452,37 +452,34 @@ The `FileInput` type:
 interface FileInput {
     path: string
     type: 'sql' | 'txt'
-    checksum?: string     // Pre-computed SHA-256; computed if omitted
 }
 ```
 
 `ExecuteFilesOptions` creates the operation record itself (or reuses a `tracker` you pass in) — there is no `operationId` parameter.
 
 This design separates file discovery (external) from execution (internal), enabling:
 - Pre-validation before database operations begin
-- Checksum computation for all files upfront
-- Full batch visibility via `createFileRecords()`
+- Rendering and hashing every file before any tracking write
+- One tracking query to decide skip-or-run for the whole batch
 
 
 ## Tracker Class
 
 The `Tracker` class handles execution history and change detection. It serves as the base for both runner and change operations.
 
 ```typescript
-import { Tracker } from './core/runner'
+import { Tracker, decideNeedsRun } from './core/runner'
 
 // (db, configName, dialect) — dialect defaults to 'sqlite', which selects the
 // prefixed table names. Pass the real dialect or pg/mssql queries hit the
 // wrong identifiers.
 const tracker = new Tracker(db, configName, 'postgres')
 
-// Check if file needs to run (by filepath). Returns NeedsRunResult:
+// Check if files need to run: the newest row per filepath in one query,
+// decided in memory. Returns NeedsRunResult:
 // { needsRun, reason?, skipReason?, previousChecksum? }
-const { needsRun, reason, skipReason } = await tracker.needsRun(filepath, checksum, force)
-
-// Inside a running operation, exclude that operation's own pending rows or
-// every file reads as 'new' forever
-const check = await tracker.needsRun(filepath, checksum, force, operationId)
+const [latest, latestErr] = await tracker.latestExecutions()
+const { needsRun, reason, skipReason } = decideNeedsRun(latestErr ?? latest.get(filepath), checksum, force)
 
 // Check if file needs to run (by name only - for changes)
 const byName = await tracker.needsRunByName(name, checksum, force)
@@ -496,29 +493,19 @@ const operationId = await tracker.createOperation({
     executedBy: 'alice@example.com',
 })
 
-// Create file records upfront (for full batch visibility) — fileType required
+// Create file records for the files about to run — fileType required
 await tracker.createFileRecords(operationId, [
     { filepath: '/path/to/file1.sql', fileType: 'sql', checksum: 'abc123' },
     { filepath: '/path/to/file2.sql', fileType: 'sql', checksum: 'def456' },
 ])
 
 // Update individual file after execution. Positional, not an options object:
-// (operationId, filepath, status, durationMs, errorMessage?, skipReason?, checksum?)
-await tracker.updateFileExecution(operationId, filepath, 'success', 45, undefined, undefined, 'abc123')
+// (operationId, filepath, status, durationMs, errorMessage?, skipReason?)
+await tracker.updateFileExecution(operationId, filepath, 'success', 45)
 
 // Skip remaining files (e.g., on error with abortOnError)
 await tracker.skipRemainingFiles(operationId, 'aborted due to previous error')
 
-// Legacy: Record a file execution (still supported). Note the field is
-// `changeId`, not `operationId`.
-await tracker.recordExecution({
-    changeId: operationId,
-    filepath,
-    checksum,
-    status: 'success',
-    durationMs: 45,
-})
-
 // Finalize operation with optional checksum and error message
 await tracker.finalizeOperation(operationId, 'success', 1234, checksum, errorMessage)
 ```
@@ -542,14 +529,14 @@ The API uses `'commit'` | `'revert'` for clarity, but the database stores `'chan
 
 ### Batch Visibility
 
-Creating file records upfront provides complete audit trails:
+Every file in the run set appears as `pending` before the first one executes. Skipped files leave no row.
 
 ```typescript
-// All files are visible immediately as 'pending'
-await tracker.createFileRecords(operationId, files)
+// Every file in the run set is visible immediately as 'pending'
+await tracker.createFileRecords(operationId, runSet)
 
 // As each executes, status updates to 'success' or 'failed'
-await tracker.updateFileExecution(operationId, filepath, { status: 'success', ... })
+await tracker.updateFileExecution(operationId, filepath, 'success', 45)
 
 // If aborted, remaining files marked as 'skipped'
 await tracker.skipRemainingFiles(operationId, 'aborted due to error')

docs/guide/changes/history.md

@@ -3,6 +3,8 @@
 
 Every database operation noorm performs is recorded. When a change runs, a build executes, or files get deployed, that execution goes into the history log with who ran it, when, how long it took, and whether it succeeded.
 
+A build, dir, files or file run where every file is unchanged writes no operation, so it appears neither in history nor in Recent Activity.
+
 This gives you an audit trail for compliance, a debugging tool for failures, and a reference for understanding what happened to your database.
 
 
@@ -248,12 +250,12 @@ The detail view breaks down each file:
 |--------|------|---------|
 | Success | `[OK]` | File executed without errors |
 | Failed | `[ERR]` | Execution threw an error |
-| Skipped | `[-]` | Either a prior run already applied this exact file, or an earlier file in the change failed and this one was never reached |
+| Skipped | `[-]` | An earlier file in the change failed and this one was never reached |
 
-For skipped files, the skip reason explains why:
+A file a prior run already applied leaves no row, so it does not appear here. For skipped files, the skip reason names the failure:
 
 ```
-[-]  setup.sql     - already applied
+[-]  setup.sql     - 001_tables.sql failed: table users already exists
 ```
 
 

docs/guide/sql-files/execution.md

@@ -145,10 +145,10 @@ noorm maintains two tables in your database to track execution history. On Postg
 | `filepath` | File that was executed, relative to the project root |
 | `checksum` | SHA-256 of the SQL that actually ran (the rendered output for a `.sql.tmpl`) |
 | `status` | `'pending'`, `'success'`, `'failed'`, `'skipped'` |
-| `skip_reason` | `'unchanged'` when change detection skipped it, or `Skipped: failure in <file>` when an earlier file in the batch failed |
+| `skip_reason` | `Skipped: failure in <file>` when an earlier file in the batch failed |
 | `duration_ms` | Execution time |
 
-Every file in a batch gets a `'pending'` row before the first one executes, so an interrupted build still shows you what it was going to do. Rows move to their final status as the run proceeds.
+Every file that will run gets a `'pending'` row before the first one executes, so an interrupted build still shows you what it was going to do. Rows move to their final status as the run proceeds. A file that change detection skips gets no row.
 
 These tables let noorm answer: "Has this exact file content been executed before, and did it succeed?"
 

docs/reference/sdk.md

@@ -575,7 +575,9 @@ Execute a single SQL file.
 await ctx.noorm.run.file('seeds/test-data.sql');
 ```
 
-**Returns:** `Promise<FileResult>`. On failure `result.error` carries the SQL/load error message; on skip `result.skipReason` carries `'unchanged'` or `'already-run'`.
+**Options (`RunOptions`):** `{ force?: boolean; abortOnError?: boolean; dryRun?: boolean; preview?: boolean; output?: string | null }`. `preview` renders the file without executing it or writing history.
+
+**Returns:** `Promise<FileResult>`. On failure `result.error` carries the SQL/load error message; on skip `result.skipReason` carries `'unchanged'`. A skipped file writes no execution history.
 
 
 #### run.files(filepaths, options?)