Simplify README and organize reference documentation

docs/development.md

@@ -0,0 +1,92 @@
+# Build and test
+
+[Back to README](../README.md)
+
+The native extension requires Rust, Bun (for smoke tests), CMake, Ninja, Git, and
+a C/C++ toolchain. The build scripts fetch pinned dependencies. No UI source
+checkout or frontend build is required.
+
+Run the commands below from the repository root.
+
+## Rust tests
+
+```sh
+mbx test --manifest-path http/Cargo.toml --locked -- --test-threads=1
+```
+
+`mbx` is the Boxington compiler cache. If it is unavailable, use `cargo` in its
+place. The release build also runs the ignored deployed-UI check, which requires
+network access.
+
+## DuckDB 2.0 on macOS
+
+Install Rust 1.99.0 through rustup, have Curl, Perl, and Make available, and supply
+the matching official DuckDB 2.0 CLI:
+
+```sh
+scripts/build-release.sh --duckdb /absolute/path/to/v2.0.0/duckdb
+```
+
+This builds Quack and SQLNB, runs Rust and native smoke tests, and writes artifacts
+to `.artifacts/v2.0.0/<platform>/`. See [release builds](releases.md) for exact
+versions, prerequisites, and signing procedures.
+
+For local testing on Apple Silicon, start that same CLI with `-unsigned`, then:
+
+```sql
+load '/absolute/path/to/sqlnb-extension/.artifacts/v2.0.0/osx_arm64/quack.duckdb_extension';
+load '/absolute/path/to/sqlnb-extension/.artifacts/v2.0.0/osx_arm64/sqlnb.duckdb_extension';
+select sqlnb_start();
+```
+
+Use `osx_amd64` for Intel macOS. Unsigned loading is only needed for local build
+artifacts; installed releases use the signed repository.
+
+## Windows qualification
+
+The [Windows workflow](../.github/workflows/windows.yml) runs the Rust contracts
+and filesystem security tests, then builds DuckDB 2.0, Quack, and SQLNB and exercises
+the native HTTP bridge. From a Visual Studio developer PowerShell with Rust 1.99.0,
+Bun, CMake, Ninja, and Git available:
+
+```powershell
+./scripts/qualify-windows.ps1
+```
+
+The script uses a temporary build directory. The Rust security tests also require
+permission to create symbolic links, as configured by the workflow.
+
+## Legacy DuckDB 1.5.4 build
+
+`scripts/build-and-test.sh` targets **DuckDB 1.5.4**, independently of the 2.0
+release profile. It supports macOS and Linux, requires Python 3 in addition to the
+tools above, and retains a matching CLI and extensions:
+
+```sh
+bash scripts/build-and-test.sh
+.artifacts/osx_arm64/duckdb -unsigned
+```
+
+Use the corresponding platform directory on Linux or Intel macOS. In that CLI:
+
+```sql
+load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/httpfs.duckdb_extension';
+load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/quack.duckdb_extension';
+load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/sqlnb.duckdb_extension';
+select sqlnb_start();
+```
+
+Do not mix artifacts between DuckDB versions. The legacy `sidequery_ui_start`,
+`sidequery_ui_status`, and `sidequery_ui_stop` compatibility aliases remain available.
+
+## Working on the UI
+
+UI source and build tooling live separately from this repository. On DuckDB 2.0,
+point the native server at a built frontend directory:
+
+```sql
+select sqlnb_start(assets := '/absolute/path/to/assets');
+```
+
+Local `assets` take precedence over the remote `ui` URL. See the
+[asset contract](ui-assets.md) for remote hosting and cache behavior.

docs/mcp.md

@@ -0,0 +1,107 @@
+# Local MCP server
+
+[Back to README](../README.md)
+
+MCP is disabled by default. On DuckDB 2.0, enable it explicitly:
+
+```sql
+select sqlnb_start(mcp := true, notebook_root := '~/Documents/notebooks');
+select sqlnb_status();
+```
+
+The returned status contains `mcp.enabled`, `mcp.url` (the loopback `/mcp`
+Streamable HTTP endpoint), and `mcp.token`. Configure an MCP client with that URL
+and an `Authorization: Bearer <mcp.token>` header. The token is separate from the
+browser capability, changes at every start, and is available only from the SQL
+connection that started the server. Do not put it in a URL. The server rejects
+foreign browser origins and Host headers. `sqlnb_stop()` stops both surfaces;
+closing the owning SQL connection also stops a server started with MCP enabled.
+
+On the legacy DuckDB 1.5.4 compatibility interface, the sixth positional argument enables MCP:
+
+```sql
+select sidequery_ui_start(
+  'https://app.sidequery.dev/sqlnb/', '', 0, true,
+  '~/Documents/notebooks', true
+);
+```
+
+The server uses Sidequery's canonical `sidequery.*` tool names and contract 2.0
+metadata. This is an intentional wire break: the earlier generic tool names are
+removed. Every result includes `ok`; failures set MCP `isError` and include
+`error.code` and `error.message`, with both structured content and text fallback.
+
+| Tool | Arguments and behavior |
+| --- | --- |
+| `sidequery.sql.query` | `sql`, optional `limit` (default 100, 1–1000). Executes one read-only SELECT. |
+| `sidequery.sql.execute` | Same arguments; permits one statement including DDL and DML. |
+| `sidequery.sql.schema.inspect` | Lists schema columns visible to the agent connection. |
+| `sidequery.notebook.sql.list` | Returns saved notebook paths and `sessions` with browser `session_id`, `notebook_path`, and `revision`. |
+| `sidequery.notebook.sql.get` | Inspects an open notebook; optional `cell_limit` (default 3, maximum 3), `limit`, `run`, and `bust_cache`. |
+| `sidequery.notebook.sql.cell.list` | Lists all cells in the open notebook, including SQL and stable current document IDs. |
+| `sidequery.notebook.sql.run` | Runs the open notebook and returns per-cell results. |
+| `sidequery.notebook.sql.cell.run` | Runs one cell addressed by `cell_id` or zero-based `cell_index`; optional `limit`, `bust_cache`. |
+| `sidequery.notebook.sql.cell.insert` | `sql`, optional zero-based `index`, display `title`, `run`, `limit`. |
+| `sidequery.notebook.sql.cell.update` | Cell address, optional `sql`, display `title`, `run`, `limit`. |
+| `sidequery.notebook.sql.cell.delete` | Deletes the addressed live cell. |
+| `sidequery.notebook.sql.source.read` | Embedded saved-file operation: `notebook_path`; returns source and revision. |
+| `sidequery.notebook.sql.source.write` | Embedded saved-file operation: `notebook_path`, `source`, `expected_revision`; empty revision creates, existing files require their current revision. Returns `mutation.saved=true`. |
+
+SQL tools use one persistent agent connection to the existing DuckDB database,
+shared by MCP clients for this server's lifetime. It is separate from the browser's
+Quack connection: database objects are shared, but temporary tables, transactions,
+and connection settings are not. Results have a 1 MiB limit and a 30-second deadline.
+`max_rows` is accepted as the hosted spelling of `limit`; conflicting values fail.
+Rows are JSON objects, with BIGINT, DECIMAL, date/time and nested values serialized
+as strings to preserve precision. Native cancellation interrupts only its matching
+query and finishes cleanup before reusing the connection.
+
+The read-only tool checks DuckDB's parsed statement, prepared-statement mutation
+properties, and bound plan. It rejects volatile functions and table functions it
+cannot establish are read-only. User-defined views and macros are rejected before
+binding because they can hide bind-time side effects; use the explicitly mutating
+`sidequery.sql.execute` tool when these are needed.
+
+Live tools accept `notebook_path` to select a unique open notebook. Omit it only
+when exactly one browser session exists, or provide `session_id` to select a window.
+Every edit and explicit run requires `expected_revision` from listing or a previous
+result. Reads may omit it; they still validate the selected revision in the browser.
+Cell `title` changes the display title, preserving the named SQL result used by
+dependent cells. An empty title clears it. Live edits return `mutation.saved=false`
+and stay unsaved until the user saves in the browser.
+
+For example, after listing sessions, call `sidequery.notebook.sql.cell.update`:
+
+```json
+{
+  "session_id": "<session_id>",
+  "expected_revision": "<revision>",
+  "cell_index": 0,
+  "sql": "select 42 as answer",
+  "title": "Answer",
+  "run": "none"
+}
+```
+
+Then call `sidequery.notebook.sql.cell.run` with the returned revision and cell
+address. Notebook results use the shared `kind: notebook` envelope with operation,
+per-cell statuses/results, and explicit mutation state.
+
+Unlike native/hosted Sidequery, embedded reads and edits default to `run=none`.
+Quack cannot enforce read-only automatic execution on its browser connection, so
+`get` with `run=missing` can reuse cached results, but any requested automatic SQL
+execution reports `POLICY_BLOCKED`. `run=all`, `bust_cache=true` with an automatic
+run mode, and edit `run=cell|affected` likewise cannot execute SQL. The explicit
+notebook/cell run tools execute on the browser connection and permit mutations.
+
+The browser bridge rejects stale revisions and does not replay delivered commands.
+A timeout or cancellation stops subsequent cells, but Quack cannot interrupt a
+browser query already running. Inspect the document before retrying a mutation.
+Saved-file writes reject paths currently reported open by connected browsers.
+Browser polling leases expire after five seconds without a heartbeat, and live
+calls time out after 30 seconds.
+
+The browser/native MCP bridge negotiates `contract_version: 2`. Old cached UI
+builds and older native servers cannot apply commands from this contract: the
+bridge refuses incompatible versions before editing. Publish the matching UI and
+native extension together; restart the owning SQLNB server to load updated assets.

docs/notebooks.md

@@ -0,0 +1,119 @@
+# Notebook storage and integrations
+
+[Back to README](../README.md)
+
+## Recent notebook
+
+The authenticated `GET /api/v1/recent-notebook` endpoint returns `{ "path": "relative.sqlnb" }`, or `{ "path": null }` when no readable recent notebook is available. The UI records a successful open or save using `PUT` to the same endpoint with `{ "path": "relative.sqlnb" }`. The path must identify an existing notebook within the configured root. This preference survives server restarts and port changes and is isolated by canonical notebook root under the OS cache's `sidequery/sqlnb-extension/notebooks` directory. It contains only the relative path; unavailable or corrupt preferences do not affect normal notebook file operations.
+
+## DuckDB UI notebooks
+
+The local DuckDB UI launched by `duckdb -ui` stores its notebooks in a DuckDB
+database at `~/.duckdb/extension_data/ui/ui.db`. SQLNB exposes existing notebooks
+as a second live source alongside local SQLNB files. Opening a notebook does not
+execute it; saving edits writes a new version to that same database.
+
+The browser uses three native SQL functions, each returning a JSON string:
+
+```sql
+select sqlnb_duckdb_ui_notebooks();
+select sqlnb_duckdb_ui_notebook('notebook-uuid');
+select sqlnb_duckdb_ui_save('notebook-uuid', expected_entity_version, 'Title', 'edited notebook JSON');
+```
+
+Reads return the official ID, entity version, title and stored notebook JSON.
+Listing omits the JSON bodies. Saves require serialization format 3 and both the
+expected entity version and unchanged content version. An accepted save advances
+both versions and expires the preceding current revision in seven days, matching
+the official UI. Conflicts and malformed edits roll back. Notebook identity,
+creation time, internal name, current-notebook preference, unknown JSON fields
+and historical revisions remain intact. Browsing never creates or migrates a
+database, and `enable_external_access` must be enabled.
+
+An official UI catalog already attached to the host instance takes precedence;
+otherwise each operation opens the existing default database for its duration.
+Normal DuckDB locks apply. A separate process holding a write lock prevents
+access; stop that process or run both UIs in the same DuckDB instance. No lock
+bypass or database copy is used. The current adapter supports editing existing
+notebooks, not creating or deleting notebook records.
+
+The frontend owns conversion through the canonical Rust SQLNB parser and
+serializer. Cell IDs, database choices, grouping, comments and SQLNB directives
+survive saves. Features the official UI does not implement remain SQLNB source
+comments, with a compatibility notice in SQLNB. The official UI must reopen or
+refresh a notebook to see saved edits. SQLNB refreshes clean documents and retains
+dirty drafts when another editor changes the source.
+
+The native source smoke uses an isolated home directory and database:
+
+```sh
+bun scripts/duckdb-ui-smoke.mjs /absolute/path/to/duckdb /absolute/path/to/sqlnb.duckdb_extension
+```
+
+## Notebook history and project Git
+
+The native HTTP server records notebook source snapshots synchronously on accepted
+saves, including the previous on-disk source before replacement. Identical saves
+are deduplicated; an explicit restore always appends a revision. A → B → A keeps
+all three states. Restores require the current file revision, just like saves.
+Renaming through SQLNB keeps the notebook's earlier history accessible at its new
+path. Each snapshot records the current project branch and HEAD when available.
+
+History is a private, independent bare Git repository, without alternates or links
+to the project's object store. It neither stages files nor changes the user's
+branches or index. Embedded, vendored libgit2 provides history even when the Git
+executable is absent. State lives beneath
+`~/Library/Application Support/sidequery/sqlnb-history/<root-hash>` on macOS and
+`$XDG_STATE_HOME/sidequery/sqlnb-history/<root-hash>` on Linux (falling back to
+`~/.local/state`). The canonical notebook-root path scopes the hash, so separate
+worktrees have separate histories. This is durable application state, not the UI
+asset cache. The directory is private to the current user.
+
+Each `refs/snapshots/<commit>` names an independent Git commit containing the exact
+source bytes at `source`. Standard Git tools can inspect/export snapshots with
+`git --git-dir=<scope>/snapshots.git show <commit>:source`. `catalog.json` supplies
+ordered revision metadata, rename associations and retention settings. It is not
+SQLite. Snapshot objects and a durable intent journal precede the atomic notebook
+replacement; successful requests also finalize the catalog. Startup recovers
+unfinished intents as **interrupted-write**, which may represent an intended write
+that never reached disk. Such a revision does not claim a successful save. Disk
+reconciliation records the actual current source separately when different.
+
+The server reconciles the secured notebook tree on startup and every two seconds,
+and joins this task during shutdown. Polling deliberately avoids OS watcher limits
+and overflow gaps. External edits are best effort: transient states changed between
+polls cannot be recovered, and external renames appear as independent paths. A durable
+last-observed source hash prevents unchanged files from being recaptured after
+retention removes their snapshots; each scan prunes only once. All
+internal saves are captured synchronously, independently of polling. Directory and
+file access retains the descriptor-based no-symlink boundary. As with ordinary file
+saves, a noncooperating external writer can race the final compare/rename; portable
+filesystems do not offer cross-process compare-and-swap replacement.
+
+Authenticated endpoints are `GET /api/v1/history?path=...`,
+`GET /api/v1/history/revision?path=...&id=...`,
+`POST /api/v1/history/restore {path,id,expectedRevision}` and
+`PUT /api/v1/history/settings {maxAgeDays,maxRevisions,maxBytes}`. Defaults are
+30 days, 1,000 revisions and 256 MiB per notebook root. The byte budget counts
+retained source bytes before Git compression, not filesystem allocation or Git
+metadata; valid budgets are 16 MiB through 1 TiB. Retention removes the oldest
+revisions, truncates snapshot refs, explicitly packs retained objects and removes
+obsolete packs/loose objects. Reducing limits deletes history. The newest eligible
+snapshot is retained even if it alone exceeds a byte budget. Count/age limits still
+apply. A list/read is scoped to its notebook lineage; an arbitrary object ID cannot
+read another notebook's history.
+
+Project Git operations are separate, explicit actions using the installed Git
+executable and normal configured authentication, hooks and signing. Authenticated
+`GET /api/v1/git` reports repository state, HEAD and an `indexRevision` token;
+`GET /api/v1/git/diff?path=...&staged=...` previews a notebook. Stage, unstage and
+commit requests include `expectedHead` and `expectedIndex` to reject stale reviewed
+state. Commits start from the explicitly selected staged notebook paths, preserving
+unrelated staging and unstaged bytes. Configured hooks can change the commit: if a
+hook includes unselected paths, the API reports that the commit was created with
+additional paths instead of reporting normal success. Newly committed index entries
+are reconciled, while pre-existing unrelated staging is preserved; an overlap with
+that staging is explicitly reported for inspection. The commit and hook edits are
+never rolled back. Push uses only the configured upstream, never force. Repository
+initialization is explicit. Git paths stay within the configured notebook root,
+even when its repository is an ancestor directory.