diff --git a/README.md b/README.md index d488cd8..a4a9f95 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,25 @@ -# SQLNB extension for DuckDB +# SQLNB for DuckDB -A native DuckDB shell for Sidequery's remotely deployed notebook UI. SQL executes through Quack; an in-process Axum server owns local notebook files. This repository contains native extension code and native tests. UI source and build tooling are maintained separately. +SQL notebooks in your browser, backed by your local DuckDB session. Open and save +notebook files, run queries, and work with tables and charts. -## Remote UI and updates +This repository contains the native extension: a local HTTP server, notebook file +and history management, and an optional MCP server. SQL runs through Quack. The +Sidequery notebook UI is developed separately and downloaded on first launch; +you don't need to build the frontend to use the extension. -`sqlnb_start()` uses the production SQLNB UI by default. Each start checks its `manifest.json`, downloads a complete new generation, verifies file sizes and SHA-256 hashes, and activates it in the local disk cache. A failed download or unavailable server falls back to a verified cached generation, including an older release if the active cache is damaged. The first launch needs a reachable compatible UI release. +## Install and start -The running server keeps its chosen generation, so an update cannot mix assets or reload a notebook with unsaved edits. Stop and start the UI to pick up a newly deployed release. Normal UI updates do not require rebuilding or reinstalling the extension. +Use DuckDB 2.0 with a matching published SQLNB build for your platform. Register +Sidequery's signed extension repository once: -Sidequery's UI publishing path is `https://app.sidequery.dev/sqlnb/`. Its publishing integration is being delivered separately; this repository does not deploy it. +```sql +set allow_extension_repositories = 'allowed'; +create extension repository sidequery + with prefix 'https://extensions.sidequery.dev'; +``` -With the Sidequery extension repository configured, the installed SQLNB flow is: +Then install and open the notebook: ```sql install sqlnb from sidequery; @@ -18,342 +27,32 @@ load sqlnb from sidequery; select sqlnb_start(); ``` -On DuckDB 2.0, SQLNB installs and loads Quack from the same trusted Sidequery repository when it starts. It also loads the matching core `httpfs` and `icu` extensions through DuckDB's normal core autoloading policy, so those dependencies do not need separate `load` statements. - -## Build and run - -```sh -bash scripts/build-and-test.sh -.artifacts/osx_arm64/duckdb -unsigned -``` - -The native build requires Rust, Bun (for native smoke tests), CMake, Ninja, Git, and a C/C++ toolchain. It fetches pinned DuckDB, Quack, httpfs, and vcpkg sources and includes DuckDB's ICU extension for date profiles. Use the corresponding platform artifact directory on Linux or Intel macOS. - -Load the three matching artifacts in the DuckDB session, using absolute paths: +Keep DuckDB running while you use the browser. SQLNB loads its Quack, `httpfs`, +and `icu` dependencies automatically. The first launch needs an internet +connection to download the UI; subsequent launches can use a verified cached copy. -```sql -load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/httpfs.duckdb_extension'; -load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/quack.duckdb_extension'; -load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/sqlnb.duckdb_extension'; -select sqlnb_start(); -``` +Use `select sqlnb_status();` to see the local URL and `select sqlnb_stop();` to +stop the server. Restart the server to pick up UI updates. -The database process must remain running. `sqlnb_status()` reports its local URL; `sqlnb_stop()` stops the UI server. Notebook listing, saving, folders, and moves use the extension's local file API without browser filesystem permissions. +## Configuration -On DuckDB 2.0, startup options are named and optional. Override only what you need: +Choose a notebook directory or port, and control whether startup opens a browser: ```sql select sqlnb_start( - ui := 'https://example.com/sqlnb/', - port := 4567, - open_browser := false, - notebook_root := '~/Documents/notebooks' -); -``` - -For local UI development, use `assets := '/absolute/path/to/assets'`; local assets take precedence over `ui`. Existing `sidequery_ui_start(...)`, `sidequery_ui_status()`, and `sidequery_ui_stop()` calls remain available as compatibility aliases. - -The authenticated `GET /api/v1/recent-notebook` endpoint returns `{ "path": "relative.sqlnb" }`, or `{ "path": null }` when no readable recent notebook is available. The UI records a successful open or save using `PUT` to the same endpoint with `{ "path": "relative.sqlnb" }`. The path must identify an existing notebook within the configured root. This preference survives server restarts and port changes and is isolated by canonical notebook root under the OS cache's `sidequery/sqlnb-extension/notebooks` directory. It contains only the relative path; unavailable or corrupt preferences do not affect normal notebook file operations. - -## DuckDB UI notebooks - -The local DuckDB UI launched by `duckdb -ui` stores its notebooks in a DuckDB -database at `~/.duckdb/extension_data/ui/ui.db`. SQLNB exposes existing notebooks -as a second live source alongside local SQLNB files. Opening a notebook does not -execute it; saving edits writes a new version to that same database. - -The browser uses three native SQL functions, each returning a JSON string: - -```sql -select sqlnb_duckdb_ui_notebooks(); -select sqlnb_duckdb_ui_notebook('notebook-uuid'); -select sqlnb_duckdb_ui_save('notebook-uuid', expected_entity_version, 'Title', 'edited notebook JSON'); -``` - -Reads return the official ID, entity version, title and stored notebook JSON. -Listing omits the JSON bodies. Saves require serialization format 3 and both the -expected entity version and unchanged content version. An accepted save advances -both versions and expires the preceding current revision in seven days, matching -the official UI. Conflicts and malformed edits roll back. Notebook identity, -creation time, internal name, current-notebook preference, unknown JSON fields -and historical revisions remain intact. Browsing never creates or migrates a -database, and `enable_external_access` must be enabled. - -An official UI catalog already attached to the host instance takes precedence; -otherwise each operation opens the existing default database for its duration. -Normal DuckDB locks apply. A separate process holding a write lock prevents -access; stop that process or run both UIs in the same DuckDB instance. No lock -bypass or database copy is used. The current adapter supports editing existing -notebooks, not creating or deleting notebook records. - -The frontend owns conversion through the canonical Rust SQLNB parser and -serializer. Cell IDs, database choices, grouping, comments and SQLNB directives -survive saves. Features the official UI does not implement remain SQLNB source -comments, with a compatibility notice in SQLNB. The official UI must reopen or -refresh a notebook to see saved edits. SQLNB refreshes clean documents and retains -dirty drafts when another editor changes the source. - -The native source smoke uses an isolated home directory and database: - -```sh -bun scripts/duckdb-ui-smoke.mjs /absolute/path/to/duckdb /absolute/path/to/sqlnb.duckdb_extension -``` - -## Deployment contract - -The stable base URL serves `manifest.json` with `version` (7–64 lowercase hexadecimal characters), `bridgeVersion: 1`, an optional relative `assetBase` directory, and a sorted `files` array of `{path, sha256, size}`. File paths are relative to the asset base and include `index.html`. Publish the manifest together with its complete immutable generation. Incompatible bridge versions are rejected; capabilities beyond the native bridge require an extension update. - -The cache is scoped by the configured URL under `~/Library/Caches/sidequery/sqlnb-extension/ui` on macOS and `${XDG_CACHE_HOME:-~/.cache}/sidequery/sqlnb-extension/ui` on Linux. - -On Linux, an absolute `XDG_CACHE_HOME` works even when `HOME` is unset. Empty or relative XDG paths fall back to `$HOME/.cache`. - -New releases live in a separate `managed-v1` directory. Cache cleanup retains the current release, one verified fallback, and releases leased by running extension instances. Obsolete managed releases are removed after their leases are released. Older cache directories are kept as read-only offline fallbacks, since older extension processes do not support leases and may still be using them. - -No UI source checkout or frontend build is required by the installed extension. - -## Private repository release build - -For official DuckDB `v2.0.0` (source `d54cbbe3df`), use the matching official CLI: - -```sh -scripts/build-release.sh --duckdb /absolute/path/to/v2.0.0/duckdb -``` - -The macOS build pins DuckDB's final source and its tested Quack revision -`f96420ed46aee9c42064bdb99ddfd1230d6c0086`. It builds static OpenSSL 3.6.3, -runs the Rust tests (including the deployed UI check) and native smoke suite, -then retains Quack and SQLNB in `.artifacts/v2.0.0//` with source -provenance and SHA-256 checksums. All native code targets macOS 11.0. Build -concurrency defaults to two; set `CMAKE_BUILD_PARALLEL_LEVEL` to change it. -Install Rust 1.99.0 through rustup before building; this profile selects its -distributable standard library so Homebrew's host deployment target cannot leak -into the extension. - -Sign both artifacts using the monorepo's `tools/extension-repository` operator -tool. Run its signed native smoke and browser qualification before publishing -`quack`, then `sqlnb`, to `https://extensions.sidequery.dev`. Core `httpfs` and -`icu` remain official DuckDB dependencies and are not republished by Sidequery. -After publication, verify a fresh signed installation: - -```sh -bun scripts/runtime-smoke.mjs --duckdb /absolute/path/to/v2.0.0/duckdb \ - --repository https://extensions.sidequery.dev -``` - -The stable 1.5.4 and alpha profiles below remain available for their exact -runtimes; their C++ artifacts cannot be relabeled as final 2.0 builds. - -## Private repository alpha build - -The stable build above remains pinned to DuckDB 1.5.4. For the private repository, -use the official `v2.0.0-alpha41489` CLI (source `10de957379`): - -```sh -scripts/build-alpha.sh --duckdb /absolute/path/to/alpha41489/duckdb -``` - -This macOS build compiles only Quack and `sqlnb` against that exact source, -applies its upstream Quack compatibility patches, and runs the Rust and native -smoke suites, including the live deployed UI check. It fetches and verifies pinned -OpenSSL 3.6.3, then builds its static libraries for the same macOS 11.0 deployment -target as the extensions and Rust server. Curl, Perl, and Make are required in -addition to the normal build tools. Build concurrency defaults to two; set -`CMAKE_BUILD_PARALLEL_LEVEL` to change it. Outputs are retained -in `.artifacts/v2.0.0-alpha41489//`. The matching official `httpfs` is -installed from `core`; it is not republished by Sidequery. The output includes -source/dependency provenance and SHA-256 checksums. - -Starting the notebook also requires ICU for timezone conversion in date profiles. -The server loads it before accepting queries, using DuckDB's normal extension -autoloading policy. If ICU is not bundled with your DuckDB runtime, install the -matching extension with `install icu from core; load icu;` before starting the -notebook when automatic installation is disabled. Installation requires a -writable extension directory. ICU remains an official DuckDB dependency and is -not republished by Sidequery. - -After signing and preparing the repository with the Sidequery monorepo's -`tools/extension-repository` tool, validate the signed distribution: - -```sh -bun scripts/runtime-smoke.mjs --duckdb /absolute/path/to/alpha41489/duckdb \ - --repository https://extensions.sidequery.dev -``` - -A local repository directory is also accepted. Each client uses fresh extension -and trust directories, allows only signed extensions, and proves that starting -`sqlnb` bootstraps Quack from Sidequery. The suite checks queries, notebook files, caller -isolation, authentication, server cleanup, and remote UI generation updates. - -### Notebook history and project Git - -The native HTTP server records notebook source snapshots synchronously on accepted -saves, including the previous on-disk source before replacement. Identical saves -are deduplicated; an explicit restore always appends a revision. A → B → A keeps -all three states. Restores require the current file revision, just like saves. -Renaming through SQLNB keeps the notebook's earlier history accessible at its new -path. Each snapshot records the current project branch and HEAD when available. - -History is a private, independent bare Git repository, without alternates or links -to the project's object store. It neither stages files nor changes the user's -branches or index. Embedded, vendored libgit2 provides history even when the Git -executable is absent. State lives beneath -`~/Library/Application Support/sidequery/sqlnb-history/` on macOS and -`$XDG_STATE_HOME/sidequery/sqlnb-history/` on Linux (falling back to -`~/.local/state`). The canonical notebook-root path scopes the hash, so separate -worktrees have separate histories. This is durable application state, not the UI -asset cache. The directory is private to the current user. - -Each `refs/snapshots/` names an independent Git commit containing the exact -source bytes at `source`. Standard Git tools can inspect/export snapshots with -`git --git-dir=/snapshots.git show :source`. `catalog.json` supplies -ordered revision metadata, rename associations and retention settings. It is not -SQLite. Snapshot objects and a durable intent journal precede the atomic notebook -replacement; successful requests also finalize the catalog. Startup recovers -unfinished intents as **interrupted-write**, which may represent an intended write -that never reached disk. Such a revision does not claim a successful save. Disk -reconciliation records the actual current source separately when different. - -The server reconciles the secured notebook tree on startup and every two seconds, -and joins this task during shutdown. Polling deliberately avoids OS watcher limits -and overflow gaps. External edits are best effort: transient states changed between -polls cannot be recovered, and external renames appear as independent paths. A durable -last-observed source hash prevents unchanged files from being recaptured after -retention removes their snapshots; each scan prunes only once. All -internal saves are captured synchronously, independently of polling. Directory and -file access retains the descriptor-based no-symlink boundary. As with ordinary file -saves, a noncooperating external writer can race the final compare/rename; portable -filesystems do not offer cross-process compare-and-swap replacement. - -Authenticated endpoints are `GET /api/v1/history?path=...`, -`GET /api/v1/history/revision?path=...&id=...`, -`POST /api/v1/history/restore {path,id,expectedRevision}` and -`PUT /api/v1/history/settings {maxAgeDays,maxRevisions,maxBytes}`. Defaults are -30 days, 1,000 revisions and 256 MiB per notebook root. The byte budget counts -retained source bytes before Git compression, not filesystem allocation or Git -metadata; valid budgets are 16 MiB through 1 TiB. Retention removes the oldest -revisions, truncates snapshot refs, explicitly packs retained objects and removes -obsolete packs/loose objects. Reducing limits deletes history. The newest eligible -snapshot is retained even if it alone exceeds a byte budget. Count/age limits still -apply. A list/read is scoped to its notebook lineage; an arbitrary object ID cannot -read another notebook's history. - -Project Git operations are separate, explicit actions using the installed Git -executable and normal configured authentication, hooks and signing. Authenticated -`GET /api/v1/git` reports repository state, HEAD and an `indexRevision` token; -`GET /api/v1/git/diff?path=...&staged=...` previews a notebook. Stage, unstage and -commit requests include `expectedHead` and `expectedIndex` to reject stale reviewed -state. Commits start from the explicitly selected staged notebook paths, preserving -unrelated staging and unstaged bytes. Configured hooks can change the commit: if a -hook includes unselected paths, the API reports that the commit was created with -additional paths instead of reporting normal success. Newly committed index entries -are reconciled, while pre-existing unrelated staging is preserved; an overlap with -that staging is explicitly reported for inspection. The commit and hook edits are -never rolled back. Push uses only the configured upstream, never force. Repository -initialization is explicit. Git paths stay within the configured notebook root, -even when its repository is an ancestor directory. - -## Local MCP server - -MCP is disabled by default. On DuckDB 2.0, enable it explicitly: - -```sql -select sqlnb_start(mcp := true, notebook_root := '~/Documents/notebooks'); -select sqlnb_status(); -``` - -The returned status contains `mcp.enabled`, `mcp.url` (the loopback `/mcp` -Streamable HTTP endpoint), and `mcp.token`. Configure an MCP client with that URL -and an `Authorization: Bearer ` header. The token is separate from the -browser capability, changes at every start, and is available only from the SQL -connection that started the server. Do not put it in a URL. The server rejects -foreign browser origins and Host headers. `sqlnb_stop()` stops both surfaces; -closing the owning SQL connection also stops a server started with MCP enabled. - -On the stable compatibility interface, the sixth positional argument enables MCP: - -```sql -select sidequery_ui_start( - 'https://app.sidequery.dev/sqlnb/', '', 0, true, - '~/Documents/notebooks', true + notebook_root := '~/Documents/notebooks', + port := 4567, + open_browser := false ); ``` -The server uses Sidequery's canonical `sidequery.*` tool names and contract 2.0 -metadata. This is an intentional wire break: the earlier generic tool names are -removed. Every result includes `ok`; failures set MCP `isError` and include -`error.code` and `error.message`, with both structured content and text fallback. - -| Tool | Arguments and behavior | -| --- | --- | -| `sidequery.sql.query` | `sql`, optional `limit` (default 100, 1–1000). Executes one read-only SELECT. | -| `sidequery.sql.execute` | Same arguments; permits one statement including DDL and DML. | -| `sidequery.sql.schema.inspect` | Lists schema columns visible to the agent connection. | -| `sidequery.notebook.sql.list` | Returns saved notebook paths and `sessions` with browser `session_id`, `notebook_path`, and `revision`. | -| `sidequery.notebook.sql.get` | Inspects an open notebook; optional `cell_limit` (default 3, maximum 3), `limit`, `run`, and `bust_cache`. | -| `sidequery.notebook.sql.cell.list` | Lists all cells in the open notebook, including SQL and stable current document IDs. | -| `sidequery.notebook.sql.run` | Runs the open notebook and returns per-cell results. | -| `sidequery.notebook.sql.cell.run` | Runs one cell addressed by `cell_id` or zero-based `cell_index`; optional `limit`, `bust_cache`. | -| `sidequery.notebook.sql.cell.insert` | `sql`, optional zero-based `index`, display `title`, `run`, `limit`. | -| `sidequery.notebook.sql.cell.update` | Cell address, optional `sql`, display `title`, `run`, `limit`. | -| `sidequery.notebook.sql.cell.delete` | Deletes the addressed live cell. | -| `sidequery.notebook.sql.source.read` | Embedded saved-file operation: `notebook_path`; returns source and revision. | -| `sidequery.notebook.sql.source.write` | Embedded saved-file operation: `notebook_path`, `source`, `expected_revision`; empty revision creates, existing files require their current revision. Returns `mutation.saved=true`. | - -SQL tools use one persistent agent connection to the existing DuckDB database, -shared by MCP clients for this server's lifetime. It is separate from the browser's -Quack connection: database objects are shared, but temporary tables, transactions, -and connection settings are not. Results have a 1 MiB limit and a 30-second deadline. -`max_rows` is accepted as the hosted spelling of `limit`; conflicting values fail. -Rows are JSON objects, with BIGINT, DECIMAL, date/time and nested values serialized -as strings to preserve precision. Native cancellation interrupts only its matching -query and finishes cleanup before reusing the connection. - -The read-only tool checks DuckDB's parsed statement, prepared-statement mutation -properties, and bound plan. It rejects volatile functions and table functions it -cannot establish are read-only. User-defined views and macros are rejected before -binding because they can hide bind-time side effects; use the explicitly mutating -`sidequery.sql.execute` tool when these are needed. - -Live tools accept `notebook_path` to select a unique open notebook. Omit it only -when exactly one browser session exists, or provide `session_id` to select a window. -Every edit and explicit run requires `expected_revision` from listing or a previous -result. Reads may omit it; they still validate the selected revision in the browser. -Cell `title` changes the display title, preserving the named SQL result used by -dependent cells. An empty title clears it. Live edits return `mutation.saved=false` -and stay unsaved until the user saves in the browser. - -For example, after listing sessions, call `sidequery.notebook.sql.cell.update`: - -```json -{ - "session_id": "", - "expected_revision": "", - "cell_index": 0, - "sql": "select 42 as answer", - "title": "Answer", - "run": "none" -} -``` - -Then call `sidequery.notebook.sql.cell.run` with the returned revision and cell -address. Notebook results use the shared `kind: notebook` envelope with operation, -per-cell statuses/results, and explicit mutation state. - -Unlike native/hosted Sidequery, embedded reads and edits default to `run=none`. -Quack cannot enforce read-only automatic execution on its browser connection, so -`get` with `run=missing` can reuse cached results, but any requested automatic SQL -execution reports `POLICY_BLOCKED`. `run=all`, `bust_cache=true` with an automatic -run mode, and edit `run=cell|affected` likewise cannot execute SQL. The explicit -notebook/cell run tools execute on the browser connection and permit mutations. +To connect an MCP client, start with `mcp := true` and follow the +[MCP setup guide](docs/mcp.md). MCP is disabled by default. -The browser bridge rejects stale revisions and does not replay delivered commands. -A timeout or cancellation stops subsequent cells, but Quack cannot interrupt a -browser query already running. Inspect the document before retrying a mutation. -Saved-file writes reject paths currently reported open by connected browsers. -Browser polling leases expire after five seconds without a heartbeat, and live -calls time out after 30 seconds. +## Development and reference -The browser/native MCP bridge negotiates `contract_version: 2`. Old cached UI -builds and older native servers cannot apply commands from this contract: the -bridge refuses incompatible versions before editing. Publish the matching UI and -native extension together; restart the owning SQLNB server to load updated assets. +- [Build and test](docs/development.md) — native prerequisites and version-specific builds. +- [Notebook storage](docs/notebooks.md) — files, DuckDB UI notebooks, history, and project Git. +- [MCP reference](docs/mcp.md) — authentication, tools, and execution behavior. +- [UI assets](docs/ui-assets.md) — custom frontends, updates, and the deployment contract. +- [Release procedures](docs/releases.md) — maintainer builds, signing, and publication. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..a611765 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,92 @@ +# Build and test + +[Back to README](../README.md) + +The native extension requires Rust, Bun (for smoke tests), CMake, Ninja, Git, and +a C/C++ toolchain. The build scripts fetch pinned dependencies. No UI source +checkout or frontend build is required. + +Run the commands below from the repository root. + +## Rust tests + +```sh +mbx test --manifest-path http/Cargo.toml --locked -- --test-threads=1 +``` + +`mbx` is the Boxington compiler cache. If it is unavailable, use `cargo` in its +place. The release build also runs the ignored deployed-UI check, which requires +network access. + +## DuckDB 2.0 on macOS + +Install Rust 1.99.0 through rustup, have Curl, Perl, and Make available, and supply +the matching official DuckDB 2.0 CLI: + +```sh +scripts/build-release.sh --duckdb /absolute/path/to/v2.0.0/duckdb +``` + +This builds Quack and SQLNB, runs Rust and native smoke tests, and writes artifacts +to `.artifacts/v2.0.0//`. See [release builds](releases.md) for exact +versions, prerequisites, and signing procedures. + +For local testing on Apple Silicon, start that same CLI with `-unsigned`, then: + +```sql +load '/absolute/path/to/sqlnb-extension/.artifacts/v2.0.0/osx_arm64/quack.duckdb_extension'; +load '/absolute/path/to/sqlnb-extension/.artifacts/v2.0.0/osx_arm64/sqlnb.duckdb_extension'; +select sqlnb_start(); +``` + +Use `osx_amd64` for Intel macOS. Unsigned loading is only needed for local build +artifacts; installed releases use the signed repository. + +## Windows qualification + +The [Windows workflow](../.github/workflows/windows.yml) runs the Rust contracts +and filesystem security tests, then builds DuckDB 2.0, Quack, and SQLNB and exercises +the native HTTP bridge. From a Visual Studio developer PowerShell with Rust 1.99.0, +Bun, CMake, Ninja, and Git available: + +```powershell +./scripts/qualify-windows.ps1 +``` + +The script uses a temporary build directory. The Rust security tests also require +permission to create symbolic links, as configured by the workflow. + +## Legacy DuckDB 1.5.4 build + +`scripts/build-and-test.sh` targets **DuckDB 1.5.4**, independently of the 2.0 +release profile. It supports macOS and Linux, requires Python 3 in addition to the +tools above, and retains a matching CLI and extensions: + +```sh +bash scripts/build-and-test.sh +.artifacts/osx_arm64/duckdb -unsigned +``` + +Use the corresponding platform directory on Linux or Intel macOS. In that CLI: + +```sql +load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/httpfs.duckdb_extension'; +load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/quack.duckdb_extension'; +load '/absolute/path/to/sqlnb-extension/.artifacts/osx_arm64/sqlnb.duckdb_extension'; +select sqlnb_start(); +``` + +Do not mix artifacts between DuckDB versions. The legacy `sidequery_ui_start`, +`sidequery_ui_status`, and `sidequery_ui_stop` compatibility aliases remain available. + +## Working on the UI + +UI source and build tooling live separately from this repository. On DuckDB 2.0, +point the native server at a built frontend directory: + +```sql +select sqlnb_start(assets := '/absolute/path/to/assets'); +``` + +Local `assets` take precedence over the remote `ui` URL. See the +[asset contract](ui-assets.md) for remote hosting and cache behavior. diff --git a/docs/mcp.md b/docs/mcp.md new file mode 100644 index 0000000..9ef5ade --- /dev/null +++ b/docs/mcp.md @@ -0,0 +1,107 @@ +# Local MCP server + +[Back to README](../README.md) + +MCP is disabled by default. On DuckDB 2.0, enable it explicitly: + +```sql +select sqlnb_start(mcp := true, notebook_root := '~/Documents/notebooks'); +select sqlnb_status(); +``` + +The returned status contains `mcp.enabled`, `mcp.url` (the loopback `/mcp` +Streamable HTTP endpoint), and `mcp.token`. Configure an MCP client with that URL +and an `Authorization: Bearer ` header. The token is separate from the +browser capability, changes at every start, and is available only from the SQL +connection that started the server. Do not put it in a URL. The server rejects +foreign browser origins and Host headers. `sqlnb_stop()` stops both surfaces; +closing the owning SQL connection also stops a server started with MCP enabled. + +On the legacy DuckDB 1.5.4 compatibility interface, the sixth positional argument enables MCP: + +```sql +select sidequery_ui_start( + 'https://app.sidequery.dev/sqlnb/', '', 0, true, + '~/Documents/notebooks', true +); +``` + +The server uses Sidequery's canonical `sidequery.*` tool names and contract 2.0 +metadata. This is an intentional wire break: the earlier generic tool names are +removed. Every result includes `ok`; failures set MCP `isError` and include +`error.code` and `error.message`, with both structured content and text fallback. + +| Tool | Arguments and behavior | +| --- | --- | +| `sidequery.sql.query` | `sql`, optional `limit` (default 100, 1–1000). Executes one read-only SELECT. | +| `sidequery.sql.execute` | Same arguments; permits one statement including DDL and DML. | +| `sidequery.sql.schema.inspect` | Lists schema columns visible to the agent connection. | +| `sidequery.notebook.sql.list` | Returns saved notebook paths and `sessions` with browser `session_id`, `notebook_path`, and `revision`. | +| `sidequery.notebook.sql.get` | Inspects an open notebook; optional `cell_limit` (default 3, maximum 3), `limit`, `run`, and `bust_cache`. | +| `sidequery.notebook.sql.cell.list` | Lists all cells in the open notebook, including SQL and stable current document IDs. | +| `sidequery.notebook.sql.run` | Runs the open notebook and returns per-cell results. | +| `sidequery.notebook.sql.cell.run` | Runs one cell addressed by `cell_id` or zero-based `cell_index`; optional `limit`, `bust_cache`. | +| `sidequery.notebook.sql.cell.insert` | `sql`, optional zero-based `index`, display `title`, `run`, `limit`. | +| `sidequery.notebook.sql.cell.update` | Cell address, optional `sql`, display `title`, `run`, `limit`. | +| `sidequery.notebook.sql.cell.delete` | Deletes the addressed live cell. | +| `sidequery.notebook.sql.source.read` | Embedded saved-file operation: `notebook_path`; returns source and revision. | +| `sidequery.notebook.sql.source.write` | Embedded saved-file operation: `notebook_path`, `source`, `expected_revision`; empty revision creates, existing files require their current revision. Returns `mutation.saved=true`. | + +SQL tools use one persistent agent connection to the existing DuckDB database, +shared by MCP clients for this server's lifetime. It is separate from the browser's +Quack connection: database objects are shared, but temporary tables, transactions, +and connection settings are not. Results have a 1 MiB limit and a 30-second deadline. +`max_rows` is accepted as the hosted spelling of `limit`; conflicting values fail. +Rows are JSON objects, with BIGINT, DECIMAL, date/time and nested values serialized +as strings to preserve precision. Native cancellation interrupts only its matching +query and finishes cleanup before reusing the connection. + +The read-only tool checks DuckDB's parsed statement, prepared-statement mutation +properties, and bound plan. It rejects volatile functions and table functions it +cannot establish are read-only. User-defined views and macros are rejected before +binding because they can hide bind-time side effects; use the explicitly mutating +`sidequery.sql.execute` tool when these are needed. + +Live tools accept `notebook_path` to select a unique open notebook. Omit it only +when exactly one browser session exists, or provide `session_id` to select a window. +Every edit and explicit run requires `expected_revision` from listing or a previous +result. Reads may omit it; they still validate the selected revision in the browser. +Cell `title` changes the display title, preserving the named SQL result used by +dependent cells. An empty title clears it. Live edits return `mutation.saved=false` +and stay unsaved until the user saves in the browser. + +For example, after listing sessions, call `sidequery.notebook.sql.cell.update`: + +```json +{ + "session_id": "", + "expected_revision": "", + "cell_index": 0, + "sql": "select 42 as answer", + "title": "Answer", + "run": "none" +} +``` + +Then call `sidequery.notebook.sql.cell.run` with the returned revision and cell +address. Notebook results use the shared `kind: notebook` envelope with operation, +per-cell statuses/results, and explicit mutation state. + +Unlike native/hosted Sidequery, embedded reads and edits default to `run=none`. +Quack cannot enforce read-only automatic execution on its browser connection, so +`get` with `run=missing` can reuse cached results, but any requested automatic SQL +execution reports `POLICY_BLOCKED`. `run=all`, `bust_cache=true` with an automatic +run mode, and edit `run=cell|affected` likewise cannot execute SQL. The explicit +notebook/cell run tools execute on the browser connection and permit mutations. + +The browser bridge rejects stale revisions and does not replay delivered commands. +A timeout or cancellation stops subsequent cells, but Quack cannot interrupt a +browser query already running. Inspect the document before retrying a mutation. +Saved-file writes reject paths currently reported open by connected browsers. +Browser polling leases expire after five seconds without a heartbeat, and live +calls time out after 30 seconds. + +The browser/native MCP bridge negotiates `contract_version: 2`. Old cached UI +builds and older native servers cannot apply commands from this contract: the +bridge refuses incompatible versions before editing. Publish the matching UI and +native extension together; restart the owning SQLNB server to load updated assets. diff --git a/docs/notebooks.md b/docs/notebooks.md new file mode 100644 index 0000000..29b7542 --- /dev/null +++ b/docs/notebooks.md @@ -0,0 +1,119 @@ +# Notebook storage and integrations + +[Back to README](../README.md) + +## Recent notebook + +The authenticated `GET /api/v1/recent-notebook` endpoint returns `{ "path": "relative.sqlnb" }`, or `{ "path": null }` when no readable recent notebook is available. The UI records a successful open or save using `PUT` to the same endpoint with `{ "path": "relative.sqlnb" }`. The path must identify an existing notebook within the configured root. This preference survives server restarts and port changes and is isolated by canonical notebook root under the OS cache's `sidequery/sqlnb-extension/notebooks` directory. It contains only the relative path; unavailable or corrupt preferences do not affect normal notebook file operations. + +## DuckDB UI notebooks + +The local DuckDB UI launched by `duckdb -ui` stores its notebooks in a DuckDB +database at `~/.duckdb/extension_data/ui/ui.db`. SQLNB exposes existing notebooks +as a second live source alongside local SQLNB files. Opening a notebook does not +execute it; saving edits writes a new version to that same database. + +The browser uses three native SQL functions, each returning a JSON string: + +```sql +select sqlnb_duckdb_ui_notebooks(); +select sqlnb_duckdb_ui_notebook('notebook-uuid'); +select sqlnb_duckdb_ui_save('notebook-uuid', expected_entity_version, 'Title', 'edited notebook JSON'); +``` + +Reads return the official ID, entity version, title and stored notebook JSON. +Listing omits the JSON bodies. Saves require serialization format 3 and both the +expected entity version and unchanged content version. An accepted save advances +both versions and expires the preceding current revision in seven days, matching +the official UI. Conflicts and malformed edits roll back. Notebook identity, +creation time, internal name, current-notebook preference, unknown JSON fields +and historical revisions remain intact. Browsing never creates or migrates a +database, and `enable_external_access` must be enabled. + +An official UI catalog already attached to the host instance takes precedence; +otherwise each operation opens the existing default database for its duration. +Normal DuckDB locks apply. A separate process holding a write lock prevents +access; stop that process or run both UIs in the same DuckDB instance. No lock +bypass or database copy is used. The current adapter supports editing existing +notebooks, not creating or deleting notebook records. + +The frontend owns conversion through the canonical Rust SQLNB parser and +serializer. Cell IDs, database choices, grouping, comments and SQLNB directives +survive saves. Features the official UI does not implement remain SQLNB source +comments, with a compatibility notice in SQLNB. The official UI must reopen or +refresh a notebook to see saved edits. SQLNB refreshes clean documents and retains +dirty drafts when another editor changes the source. + +The native source smoke uses an isolated home directory and database: + +```sh +bun scripts/duckdb-ui-smoke.mjs /absolute/path/to/duckdb /absolute/path/to/sqlnb.duckdb_extension +``` + +## Notebook history and project Git + +The native HTTP server records notebook source snapshots synchronously on accepted +saves, including the previous on-disk source before replacement. Identical saves +are deduplicated; an explicit restore always appends a revision. A → B → A keeps +all three states. Restores require the current file revision, just like saves. +Renaming through SQLNB keeps the notebook's earlier history accessible at its new +path. Each snapshot records the current project branch and HEAD when available. + +History is a private, independent bare Git repository, without alternates or links +to the project's object store. It neither stages files nor changes the user's +branches or index. Embedded, vendored libgit2 provides history even when the Git +executable is absent. State lives beneath +`~/Library/Application Support/sidequery/sqlnb-history/` on macOS and +`$XDG_STATE_HOME/sidequery/sqlnb-history/` on Linux (falling back to +`~/.local/state`). The canonical notebook-root path scopes the hash, so separate +worktrees have separate histories. This is durable application state, not the UI +asset cache. The directory is private to the current user. + +Each `refs/snapshots/` names an independent Git commit containing the exact +source bytes at `source`. Standard Git tools can inspect/export snapshots with +`git --git-dir=/snapshots.git show :source`. `catalog.json` supplies +ordered revision metadata, rename associations and retention settings. It is not +SQLite. Snapshot objects and a durable intent journal precede the atomic notebook +replacement; successful requests also finalize the catalog. Startup recovers +unfinished intents as **interrupted-write**, which may represent an intended write +that never reached disk. Such a revision does not claim a successful save. Disk +reconciliation records the actual current source separately when different. + +The server reconciles the secured notebook tree on startup and every two seconds, +and joins this task during shutdown. Polling deliberately avoids OS watcher limits +and overflow gaps. External edits are best effort: transient states changed between +polls cannot be recovered, and external renames appear as independent paths. A durable +last-observed source hash prevents unchanged files from being recaptured after +retention removes their snapshots; each scan prunes only once. All +internal saves are captured synchronously, independently of polling. Directory and +file access retains the descriptor-based no-symlink boundary. As with ordinary file +saves, a noncooperating external writer can race the final compare/rename; portable +filesystems do not offer cross-process compare-and-swap replacement. + +Authenticated endpoints are `GET /api/v1/history?path=...`, +`GET /api/v1/history/revision?path=...&id=...`, +`POST /api/v1/history/restore {path,id,expectedRevision}` and +`PUT /api/v1/history/settings {maxAgeDays,maxRevisions,maxBytes}`. Defaults are +30 days, 1,000 revisions and 256 MiB per notebook root. The byte budget counts +retained source bytes before Git compression, not filesystem allocation or Git +metadata; valid budgets are 16 MiB through 1 TiB. Retention removes the oldest +revisions, truncates snapshot refs, explicitly packs retained objects and removes +obsolete packs/loose objects. Reducing limits deletes history. The newest eligible +snapshot is retained even if it alone exceeds a byte budget. Count/age limits still +apply. A list/read is scoped to its notebook lineage; an arbitrary object ID cannot +read another notebook's history. + +Project Git operations are separate, explicit actions using the installed Git +executable and normal configured authentication, hooks and signing. Authenticated +`GET /api/v1/git` reports repository state, HEAD and an `indexRevision` token; +`GET /api/v1/git/diff?path=...&staged=...` previews a notebook. Stage, unstage and +commit requests include `expectedHead` and `expectedIndex` to reject stale reviewed +state. Commits start from the explicitly selected staged notebook paths, preserving +unrelated staging and unstaged bytes. Configured hooks can change the commit: if a +hook includes unselected paths, the API reports that the commit was created with +additional paths instead of reporting normal success. Newly committed index entries +are reconciled, while pre-existing unrelated staging is preserved; an overlap with +that staging is explicitly reported for inspection. The commit and hook edits are +never rolled back. Push uses only the configured upstream, never force. Repository +initialization is explicit. Git paths stay within the configured notebook root, +even when its repository is an ancestor directory. diff --git a/docs/releases.md b/docs/releases.md new file mode 100644 index 0000000..c964098 --- /dev/null +++ b/docs/releases.md @@ -0,0 +1,83 @@ +# Release builds and publication + +[Back to README](../README.md) + +These procedures are for maintainers. Signing and publication use the operator +tooling in the separate Sidequery monorepo and require maintainer access. +Building this repository does not require that access. + +Run the commands below from the repository root. + +## DuckDB 2.0 + +For official DuckDB `v2.0.0` (source `d54cbbe3df`), use the matching official CLI: + +```sh +scripts/build-release.sh --duckdb /absolute/path/to/v2.0.0/duckdb +``` + +The macOS build pins DuckDB's final source and its tested Quack revision +`f96420ed46aee9c42064bdb99ddfd1230d6c0086`. It builds static OpenSSL 3.6.3, +runs the Rust tests (including the deployed UI check) and native smoke suite, +then retains Quack and SQLNB in `.artifacts/v2.0.0//` with source +provenance and SHA-256 checksums. All native code targets macOS 11.0. Build +concurrency defaults to two; set `CMAKE_BUILD_PARALLEL_LEVEL` to change it. +Install Rust 1.99.0 through rustup before building; this profile selects its +distributable standard library so Homebrew's host deployment target cannot leak +into the extension. +The OpenSSL build also requires Curl, Perl, and Make. + +Sign both artifacts using the monorepo's `tools/extension-repository` operator +tool. Run its signed native smoke and browser qualification before publishing +`quack`, then `sqlnb`, to `https://extensions.sidequery.dev`. Core `httpfs` and +`icu` remain official DuckDB dependencies and are not republished by Sidequery. +After publication, verify a fresh signed installation: + +```sh +bun scripts/runtime-smoke.mjs --duckdb /absolute/path/to/v2.0.0/duckdb \ + --repository https://extensions.sidequery.dev +``` + +The legacy 1.5.4 build in [development.md](development.md) and the alpha +profile below remain available for their exact runtimes; their C++ artifacts +cannot be relabeled as final 2.0 builds. + +## Legacy DuckDB 2.0 alpha + +This compatibility profile targets the official `v2.0.0-alpha41489` CLI (source `10de957379`): + +```sh +scripts/build-alpha.sh --duckdb /absolute/path/to/alpha41489/duckdb +``` + +This macOS build compiles only Quack and `sqlnb` against that exact source, +applies its upstream Quack compatibility patches, and runs the Rust and native +smoke suites, including the live deployed UI check. It fetches and verifies pinned +OpenSSL 3.6.3, then builds its static libraries for the same macOS 11.0 deployment +target as the extensions and Rust server. Curl, Perl, and Make are required in +addition to the normal build tools. Build concurrency defaults to two; set +`CMAKE_BUILD_PARALLEL_LEVEL` to change it. Outputs are retained +in `.artifacts/v2.0.0-alpha41489//`. The matching official `httpfs` is +installed from `core`; it is not republished by Sidequery. The output includes +source/dependency provenance and SHA-256 checksums. + +Starting the notebook also requires ICU for timezone conversion in date profiles. +The server loads it before accepting queries, using DuckDB's normal extension +autoloading policy. If ICU is not bundled with your DuckDB runtime, install the +matching extension with `install icu from core; load icu;` before starting the +notebook when automatic installation is disabled. Installation requires a +writable extension directory. ICU remains an official DuckDB dependency and is +not republished by Sidequery. + +After signing and preparing the repository with the Sidequery monorepo's +`tools/extension-repository` tool, validate the signed distribution: + +```sh +bun scripts/runtime-smoke.mjs --duckdb /absolute/path/to/alpha41489/duckdb \ + --repository https://extensions.sidequery.dev +``` + +A local repository directory is also accepted. Each client uses fresh extension +and trust directories, allows only signed extensions, and proves that starting +`sqlnb` bootstraps Quack from Sidequery. The suite checks queries, notebook files, caller +isolation, authentication, server cleanup, and remote UI generation updates. diff --git a/docs/ui-assets.md b/docs/ui-assets.md new file mode 100644 index 0000000..2cdb080 --- /dev/null +++ b/docs/ui-assets.md @@ -0,0 +1,32 @@ +# UI assets and updates + +[Back to README](../README.md) + +`sqlnb_start()` uses the production SQLNB UI by default. Each start checks its `manifest.json`, downloads a complete new generation, verifies file sizes and SHA-256 hashes, and activates it in the local disk cache. A failed download or unavailable server falls back to a verified cached generation, including an older release if the active cache is damaged. The first launch needs a reachable compatible UI release. + +The running server keeps its chosen generation, so an update cannot mix assets or reload a notebook with unsaved edits. Stop and start the UI to pick up a newly deployed release. Normal UI updates do not require rebuilding or reinstalling the extension. + +Sidequery's UI publishing path is `https://app.sidequery.dev/sqlnb/`. UI publication is managed separately from this repository. + +On DuckDB 2.0, use another compatible deployment or a local frontend build: + +```sql +select sqlnb_start(ui := 'https://example.com/sqlnb/'); +-- Or serve a local build instead: +select sqlnb_start(assets := '/absolute/path/to/assets'); +``` + +Choose one startup command. Local `assets` take precedence over `ui` when both +are supplied. + +## Manifest and cache + +The stable base URL serves `manifest.json` with `version` (7–64 lowercase hexadecimal characters), `bridgeVersion: 1`, an optional relative `assetBase` directory, and a sorted `files` array of `{path, sha256, size}`. File paths are relative to the asset base and include `index.html`. Publish the manifest together with its complete immutable generation. Incompatible bridge versions are rejected; capabilities beyond the native bridge require an extension update. + +The cache is scoped by the configured URL under `~/Library/Caches/sidequery/sqlnb-extension/ui` on macOS and `${XDG_CACHE_HOME:-~/.cache}/sidequery/sqlnb-extension/ui` on Linux. + +On Linux, an absolute `XDG_CACHE_HOME` works even when `HOME` is unset. Empty or relative XDG paths fall back to `$HOME/.cache`. + +New releases live in a separate `managed-v1` directory. Cache cleanup retains the current release, one verified fallback, and releases leased by running extension instances. Obsolete managed releases are removed after their leases are released. Older cache directories are kept as read-only offline fallbacks, since older extension processes do not support leases and may still be using them. + +No UI source checkout or frontend build is required by the installed extension.